This is the problem, right here. How do we come to any agreement upon it? Problem is, you have too much noise coming from the far extremes, that it drowns out any sensible discussion.
This is mostly why there is never any sensible discussion on the subject. Attention is given to the wrong things, and since the media needs to grab views/readers attention, it often distorts and gives undue attention to things that don't deserve it. When media amplifies the signal, things that may seem important are most likely unimportant.
Many psychological tests have been administer to demonstrate this. For example, following a large incident in which a bomb went off in Germany, media reports stated that the suspect was a right-wing extremist. When polling asked what citizens thought was the most important issue of nation, right-wing extremist shot from less that 0% to 35%. Likewise, following 9/11 in the US, terrorism and national security became the most important issues facing the nation by an outstanding 75%. As time went on, and the issue faded from out attention-span, those numbers declined.
Media companies know this human attention flaw, and they often exploit it for their own gain.