38

I'm wondering if there is any study or other serious attempt which compares Fox News to MSNBC to see which one was more 'biased' towards their respective sides of the political spectrum.

I am not looking for opinion here, I'm looking for serious analysis that presents a criteria that can be (at least reasonably) objectively measured, has measured the two respective media outlets, and have presented results as to which outlet best met or failed those criteria.

I'm open to any criteria that can be used to measure bias, from number of factual inaccuracies in reporting, to how often they call the other side immoral and evil, to any other criteria that could be imagined as being a measure of, or result of, bias for their preferred political view and/or a sign of lack of trustworthiness in their reporting due to such a bias. All I really care is that it can be objectively measured and concrete results presented.

CDJB
  • 106,388
  • 31
  • 455
  • 516
dsollen
  • 9,440
  • 6
  • 40
  • 61
  • 25
    a lot of those measures are going to be subjective and hard if not impossible to quantify. However, there are viewer surveys and polling available that asks about a respondent's sources. Those could potentially be used to collectively judge the informedness (or mis) of viewers. For example, i recall in 2004 a report that fox news viewers were more likely than pbs viewers to believe that Iraq had weapons of mass destruction (false). It was about a 12% difference, iirc. Those kind of stats are actually attainable, "who lies more" isn't. – dandavis Jan 28 '20 at 19:21
  • @dandavis judging the degree of understanding of viewers could be a valid appraoch in my mind. It's not perfect, but as you said no answer will be perfect. so long as an attempt was scientifically done without obvious intent to skew the results. – dsollen Jan 28 '20 at 19:32
  • 5
    To measure bias scientifically (e.g. statistically) a study would need to limit itself questions where a true underlying quantitative parameter being measured. Categorical measures, such as true/false questions aren't really measurable as to bias, as I recall the definition of the scientific terms. – Burt_Harris Jan 28 '20 at 19:33
  • 1
    Right, we can calculate which viewers are better-informed, but throwing in 'more biased' requires that we find a centerpoint, which is a much bigger headache. – Carduus Jan 28 '20 at 19:50
  • This isn't using a random sample, but perhaps worth mentioning https://www.politifact.com/punditfact/article/2014/sep/16/fact-checking-fox-msnbc-and-cnn-punditfacts-networ/ – the gods from engineering Jan 28 '20 at 20:08
  • Also, it seems to me that a more grounded comparison would be with CNN given https://www.aim.org/don-irvine-blog/pew-fox-news-most-trusted-news-source-of-republicans-cnn-most-trusted-by-democrats/ – the gods from engineering Jan 28 '20 at 20:16
  • Humans do not have a process to systematically determine the truth. Any such study would not be able to produce a ground truth. A strong example: We can't even agree on whether climate change is man-made or not. Reasonable and informed people disagree about this. "97% of scientists say" is not a reliable way to determine the truth. Group opinions are subject to social effects and incentive issues. There is no objective process to determine what is true. – usr Jan 29 '20 at 09:47
  • 1
    I think a better question might be which organizations present more "fact" statements vs "opinion" statements. Opinions and Facts have two concrete definitions and determining which is which is not difficult. From here, one can apply their own subjective definitions of "Left" vs. "Right" thought and maybe assuage which side is more "opinionated" for their side of the aisle. This is still a subjective exercise but it is a way of looking for scientific evidence to make a specific point. – RLH Jan 29 '20 at 14:45
  • 1
    This is like two astronauts floating in deep space trying to determine who is moving and who is still. Without a universally agreed upon reference point you simply can't tell. Any investigation of bias will necessarily be biased by those doing the investigating. – eps Jan 29 '20 at 17:23
  • 1
    @RLH you better tell the philosophers, because they have been debating how to tell the difference between fact and opinion for literal millennia. It is absolutely not the case that it is easy to tell what facts are (or even if some determination is even possible), but it is rather easy to tell what groups of people think the facts are. For example you might say "the earth is not flat" to be a fact but from the perspective of a neutrino whizzing through the earth it appears to be almost nearly flat. – eps Jan 29 '20 at 17:31
  • Another complication such efforts may overlook, or one may overlook when misconstruing something as such an effort, is that lack of bias is often mistaken with being centrist. A prejudice against any idea either side of centrism is just as much a bias as is a prejudice against the ideas on one side only of, say, a right-wing position. – J.G. Jan 29 '20 at 20:37
  • 2
    @usr Humans do have a process for systematically and objectively determining truth - it's called science. The fact that some people don't agree on the causes of climate change doesn't indicate we can't objectively determine truth. Some think the earth is flat, but you can objectively prove that's false. You're correct that the popularity of a belief is not a measure of the truth, but popularity is certainly not the only way to measure truth. It scares me that "there's no such thing as truth" (or at least it's impossible to identify) is becoming such a popular notion these days. – Nuclear Hoagie Jan 30 '20 at 14:43
  • 1
    @NuclearWang Unfortunately a lot of questions politics has to deal with are not scientific questions, but ethical questions. You can not use the scientific method to prove or disprove ethics. – Philipp Jan 30 '20 at 16:45
  • 2
    @Philipp Agree, but those questions should not be framed as questions of truth in the first place. There is not an objective measure of everything, but to suggest there's not an objective measure of anything is, to me, bonkers. – Nuclear Hoagie Jan 30 '20 at 17:00
  • 1
    @NuclearWang the truth certainly exists and can be discovered. But society has no way of systematically doing that. Science is subject to group think, incompetence, corruption etc.. The scientific method works. But how can you know, that the reason for "scientists say..." is the scientific method and not some social effect? In climate change research it is known to be a career killer to produce non-mainstream results. This is no longer the scientific method at work! – usr Jan 31 '20 at 07:43
  • @usr science has given us the computer you typed this on and the internet and electricity that facilitate it. It's given us warm homes, fast transportation, instantaneous communication, and all but eradicated infectious disease (The fact that people freak out over a virus that has only infected 9,700 people and and is only fatal in 2% of the cases shows how much better were doing now of days!). I'd say that science has proven it's ability to determine facts. Denying science is just saying you'd rather believe whatever fits your biases or makes you feel better then accept the actual truth. – dsollen Jan 31 '20 at 14:31
  • 1
    @dsollen Scientists told us that smoking was healthy, that red meat was detrimental to health and a number of other things. How come 50% of papers cannot be reproduced? There are systematic issues in contemporary science. – usr Jan 31 '20 at 22:29
  • 1
    @usr It's not that bad. You read the wrong publications. There is bias in science and occasionally scientific reports are really low quality but on average the factualness and quality of scientific papers is very high. They mostly agree on e.g. smoking is bad, global warming is happening and the Earth is round. Just don't believe in a single given publication and you're fine. – NoDataDumpNoContribution Feb 01 '20 at 08:17

9 Answers9

46

I'd say anyone who tells you they have "scientific" information on media bias isn't being objective. But it is entirely possible to present a "systematic" information on media bias, and at least one online source with that goal is https://mediabiasfactcheck.com. They have an explicit methodology, which says:

When determining bias, there isn’t any true scientific formula that is 100% objective. There are objective measures that can be calculated, but ultimately there will be some degree of subjective judgement to determine these. On each page we have put up a scale with a yellow dot that shows the degree of bias for each source. Each page also has a “detailed report” section that gives some details about the source and an explanation of their bias. When calculating bias we are not just looking at political bias, but also how factual the information is and if they provide links to credible, verifiable sources. Therefore, the yellow dot may indicate political bias or how factual a source is, or in many cases, both.

Mediabiasfactcheck has a high level summary graphic which may be helpful: For example, MSNBC looks like this:

msnbc factual

msnbc graphic

Fox News looks like this:

factual on fox news

fox news graphic

But mediabiasfactcheck.com goes into detail as well, and includes a systematic categorization of different kinds of journalistic bias.

If what you are looking for is unbiased news, recognize that's not the goal of either organization. Some news sources actually make reduction of bias and emotionally loaded reporting a policy goal. Take for example: Reuters News looks like this:

reuters

reuters graphic

Note well: its impossible to to express an opinion on news source bias without having some bias, and getting accused of being biased. Mediabiasfactcheck.com is no exception: some other organizations rate mediabiasfactcheck.com as biased. Oh well...

Burt_Harris
  • 8,013
  • 1
  • 26
  • 59
  • 20
    note that, while mediafactcheck rates both about the same re. Media Bias, when it comes to factual reporting they say: Fox: We also rate them Mixed factually and borderline Questionable, MSNBC: Although pundits and hosts have failed a few fact checks, straight news reporting is usually factual and sourced, therefore we rate them Mostly Factual for news reporting. – Italian Philosophers 4 Monica Jan 28 '20 at 21:30
  • 65
    Being factual isn't enough. The facts reported and the facts not reported matter, too. For example, must US TV networks showed the House impeachment managers' making their case to remove President Trump live, but did not show Trump's lawyers making their case live. Is that bias? Or was that done because ratings were terrible for the live coverage last week? You'd have to be a mind-reader to know for sure. – Just Me Jan 28 '20 at 21:47
  • 14
    For me, being "usually factual" isn't enough either. Too many reporters want to express their opinions, and without strong journalistic standards, it get's out of hand. Compare w/ Reuters Handbook of Journalism section on Freedom from Bias, which for lays out what it means to avoid bias, e.g.: Reuters journalists do not express their opinions in news stories, voiced video or scripts, or on blogs or chat rooms they may contribute to in the course of their work. – Burt_Harris Jan 28 '20 at 23:30
  • 4
    You're both right, but the not-facts go in the Media Bias bucket and neither MSNBC/Fox come out all that well out of it. Given that CNN is pretty much out leftfield with MSNBC, at least according to mfc, I wonder who is providing a Center/Least-Biased viewpoint for general US consumption. – Italian Philosophers 4 Monica Jan 29 '20 at 01:33
  • How is it determined that reporting is fact-based? There are allegations that "sources" are sometimes being made up. It is very hard to confirm or disprove that. – usr Jan 29 '20 at 09:54
  • True facts usually tend to be very basic. For example, it is a fact that some politician said specific words at some point in time. But what precisely he meant by those words is open to interpretation. It gets subjective very quickly. – usr Jan 29 '20 at 09:55
  • 6
    @Burt_Harris I disagree. Sometimes politicians lie or intentionally mislead. Is it factual reporting to simply write "Bob said that 'pigs fly'?" Is it more or less factual to instead publish "Bob said that 'pigs fly'. Pigs do not fly?" At a certain point the average consumer needs some clarification and context from the reporter. – pip install frisbee Jan 29 '20 at 14:42
  • 28
    Ugh, those graphics are nasty. It looks like the position of the spot on the arrows is marking the position in the truth-reporting table, and reading it like that (I did, at first, until I looked harder) suggests that Fox News has a rating of High/Very High and MSNBC has a rating solidly in Low. – Richard Ward Jan 29 '20 at 15:15
  • 15
    @JustMe do you have evidence for your claim that most US TV networks didn't air Trump's lawyers? CSPAN, ABC, MSNBC and CNN certainly did – Shaz Jan 29 '20 at 21:31
  • @RichardWard Agreed. And it could be solved by using a capital lambda shape ( Λ ) instead of a horizontal bar. – VGR Jan 30 '20 at 03:44
  • 3
    @Shaz The Washington Post made a big deal how CNN, MSNBC, et al covered the House managers live in their entirety but Fox didn't. That documents the majority of networks did show the House managers making their case live. There's no such evidence for multiple networks showing Trump's defense live, probably because it's impossible to prove something doesn't exist. – Just Me Jan 30 '20 at 18:28
  • 2
    Who says Mediabiasfactcheck isn't biased? How can you confirm or deny that? – sfors says reinstate Monica Jan 30 '20 at 19:32
  • 2
    I never claimed Mediabiasfactcheck was unbiased. I said it was systematic. – Burt_Harris Jan 30 '20 at 22:16
  • 7
    @JustMe The link you provide says the exact opposite of what you're saying. "CNN and MSNBC carried the Democrats’ case in its entirety. On Saturday morning, they did the same as Trump’s legal team began its arguments." Meaning they showed the entirety of Trump's Counsel's arguments. And the screenshots provided prove CNN, Fox, and MSNBC - "The Big Three" all showed the arguments of Trump's Counsel. In fact, your own source shows that Fox is the one news network that didn't air the whole proceedings by going to their regularly scheduled programming during the House Manager's presentation. – Shaz Jan 31 '20 at 02:24
  • 12
    @JustMe Your own source shows Fox news 'missing' 31 out of 52 time slots for the House Manager's presentation, while CNN and MSNBC 'missed' 9 of 52 - and all 3 'missed' the exact same 9 time slots (likely due to being on break in the Senate). All three covered Saturday's session equally - but that's likely due to it being so short. I happen to have watched Trump's counsel present their side on MSNBC and CNN so I know they've been showing the proceedings. Your statement "must [sic] US TV networks [...] did not show Trump's lawyers making their case live" is factually incorrect. – Shaz Jan 31 '20 at 02:32
  • 2
    Are mediabiasfactcheck's bias ratings relative to what is considered "left wing" in the country of that news organization? I find it hard to believe that MSNBC is more left wing than the BBC, the CBC, NPR and even The Guardian. – cjs Jan 31 '20 at 16:00
  • "Some source actually make reduction of bias, and emotionally loaded reporting a policy goal." You should remove the comma after "bias". As it is, "reduction" applies only to "bias" and not to "emotionally loaded reporting". – Acccumulation Feb 01 '20 at 01:02
  • I'm astonished your measure of factualness achieves "Mixed" and not "Low" for Fox News. Their habit of using brainwashing repetition for totally false and ungrounded claims is an example of Propaganda, and it is utterly strange that this does not emerge from your analysis. I say this from a foreigner point of view. – Stephane Rolland Feb 03 '20 at 14:42
  • 2
    @StephaneRolland One issue is that they do a poor job of distinguishing between "news" and "editorial" content. The editorial content is generally of low accuracy but the news is held to a higher standard in general. The fact that the average viewer doesn't recognize that distinction is problematic in a way that wouldn't show up in the methodology above. – John Feb 03 '20 at 19:40
  • 3
    Failure to distinguish between news and opinion is the biggest problem with bias in the self-styled "news" media today. When I was in school (U.S. 60's-70's), recognition of that distinction was part of the curriculum, starting as early as 5th or 6th grade. Has that changed?

    I think the ranking sites do take that into account, for example by putting A.P. and Reuters at the top both in terms of factual and unbiased. Those organizations exercise discipline in avoiding editorializing the news.

     – Burt_Harris Feb 03 '20 at 22:09
21

I'll take you through a Thought Experiment to show why this isn't really possible.

Imagine you were trying to judge whether networks A, B, and C were 'biased' on the coverage of, say, "Should the US offer aid to the rebels of [country name]?"

So, you might think, well, they should have facts from all sides. Present all the facts. Have people on to discuss any point of view on the issue. Give equal time to each side. Don't exclude any news that sheds either side in a positive light. Etc.

... now let's change those 'rebels' to a genocidal group that utilizes a child-army and practices slavery.

Suddenly, things change. If your network gives facts/opinions/equal-time to the genocidal/slaver group, it suddenly feels a lot less 'unbiased' and starts to feel a lot more like they're giving legitimacy to something horrible.

Do you now see the discrepancy? You're now in a position of having to figure out, for this particular issue: how legitimate is each side? Because you can't cover both sides 'equally' and have it be balanced if only one side has a legitimate cause. It's not like someone would expect News Co. XYZ to be balanced by giving equal time to a NASA Spokesman and a rep from the Flat Earth Society.

Let's take an actual issue: Brexit.

News Organization A covers Brexit by giving the facts from both sides, giving equal time to both proponents and dissenters. Organization B has more pro-Leave airtime/arguments; Organization C has more pro-Remain. So which ones are biased?

Well, how legitimate is each side? You could say Organization A is non-biased... except it means you've given each side roughly equal legitimacy. Is that fair? How do you tell? In any sort of non-subjective manner?

Now multiply this by the number of issues that people argue about. And... add in a dash of an uncomfortable truth: generally most popular political topics do indeed have at least some legitimacy on both sides (which a lot of people don't like admitting, since it makes it harder to viscerally hate the people that disagree with them.)

So... how would you design that experiment now? You can't - because "balance" depends on your subjective measure of each side of each issue.

Kevin
  • 2,431
  • 8
  • 12
18

Many studies have been conducted to test exactly this question. This study from 2012 asked participants 4 factual questions about international events, and 5 questions about domestic ones (questions like "It took a long time to get the final results of the Iowa caucuses for Republican candidates. In the end, who was declared the winner?")

Nobody did very well on the survey-- even the most well-informed group averaged less than 2 correct answers in each category. The highest average were people who got their news primarily from NPR. Fox News performed the worst in both domestic issues and international ones, even worse than the "no news" group. MSNBC was the second-lowest, coming in just under "no news" for international issues and just above "no news" for domestic issues.

Studies like this one usually produce similar results. Some researchers call this "the Fox News effect", though a similar case could be made for calling it "the MSNBC effect". So the best empirical evidence we have suggests that Fox is worse than MSNBC-- but not by much.

PlutoThePlanet
  • 583
  • 3
  • 9
  • 21
    I'm not sure this really answers the question, as interesting as that study is. Being non-informative isn't necessarily the same as biased... – the gods from engineering Jan 28 '20 at 19:53
  • 30
    It's also not clear if watching Fox News leads to ignorance, or ignorance leads to watching Fox News, or some mixture there of. – CrackpotCrocodile Jan 29 '20 at 06:10
  • 4
    @CrackpotCrocodile It can be a vicious cycle, at both ends of the political spectrum. People gravitate towards outlets that reinforce their preconceptions. Flat-Earthers can find plenty of websites that tell them that they're correct. – Barmar Jan 29 '20 at 16:07
  • 1
    I really think that study drew the wrong conclusions. Fox News and MSNBC had terrible averages... but take a look at what brought down those averages on page 5. Liberals watching Fox News and Conservatives watching MSNBC. Liberals watching Fox scored a whole point less than Conservatives on Fox; Conservatives on MSNBC scored nearly two whole points less than Liberals on MSNBC. And compared with how "Independents" (3.20) did vs "Not Independent" (3.74), I think they missed the boat. – Kevin Jan 29 '20 at 20:11
  • 1
    @Kevin I disagree. You have to remember that very few conservatives watching only MSNBC or liberals watching only fox news, so they were a small enough number that they didn't skew the overall numbers significantly. If you compare the "liberals watching MSNBC" to "conservatives watching Fox" statistics the same trend of MSNBC watchers doing better then FOX watchers (the numbers of answers correct is much larger in this group, but that's because it includes people who got media from multiple sources, rather then just one source, which likely increased their overall knowledge) – dsollen Jan 29 '20 at 20:52
  • 4
    @Kevin On the other hand the study does not appear to specify how many got their news exclusively from fox or MSNBC. Considering the study only covered a little over a thousand people if most people got their news from multiple sources the sample size for people who got their news exclusively from one source may be a bit small to draw a conclusion. Likewise the total number of questions asked is rather small. Both may lower how much confidence we place on the exact numbers. – dsollen Jan 29 '20 at 20:57
  • 7
    One issue with surveys of this nature, though, is that the questions being asked can themselves be biased towards one group or another, even if they remain entirely factual. One frequent tactic of media on both sides is constantly harping on a few talking points, including whatever facts may support it (but typically omitting most or even any that don't.) As such, it would be pretty easy to design a study that says just about any group you want is the 'most informed' by choosing the questions according to which group is more likely to use those facts to support one of their positions. – reirab Jan 29 '20 at 21:02
  • So the argument could be made that NPR is least biased? – Burt Jan 29 '20 at 21:24
  • One of the questions was what the purpose of the sanctions against Iran are for, and responses are categorized into "anything about WMD", "anything else" and "don't know". But other reasons are terrorism and human rights. So would someone who responds with one of those be considered "wrong"? – Acccumulation Feb 01 '20 at 01:13
6

Along the lines of the excellent answer from Burt_Harris, you might be interested in Ad Fontes Media

They rank a wide array of news (... and, not-really-news) sites.

The reason I reference the answer from Burt_Harris is because I agree that it's difficult to have a strictly scientific approach to this, but Ad Fontes Media has a systematic approach, meant to produce reliable results. For detailed information, see their methodology page.

landru27
  • 169
  • 2
  • 1
    Interesting, but a bit confusing. For example, "Fox News" online is listed as less biased than MSNBC, but "Fox News" on cable is more. That makes OP's question, at least as it is currently formulated, impossible to answer with this source. – Michael W. Jan 30 '20 at 18:40
  • 3
    @MichaelW. : that's one way to look at it; another way is to realize that this source shows that "Fox News" is not a monolithic, monochromatic thing; the same is true of CNN (per that chart); in that sense, I feel that this source is very pertinent to the question, because it helps illuminate subtleties that are not made obvious by the question as-asked; IMHO, providing an answer that examines / challenges the assumptions of a given question is a great way for us all to learn from each other – landru27 Jan 30 '20 at 22:57
  • 1
    I think you misunderstand me. I think this source, if taken as true, illuminates a flaw in the original question. "Is MSNBC or Fox News more biased" can only be responded with "which MSNBC and which Fox News?" – Michael W. Jan 31 '20 at 22:41
5

AllSides.com has a nice list of bias ratings along with a community rating of how accurate that label is.

From their Media Bias Ratings page:

Our Media Bias Ratings are determined using multiple methods and represent the average judgment of Americans. They are based on blind surveys of people across the political spectrum, multi-partisan analysis, editorial reviews, third party data, and tens of thousands of user feedback ratings. Our scientifically-generated ratings are fluid and subject to change as new information is gathered and biases change over time.

MSNBC rating (2020/02/03): MSNBC AllSides Bias Rating

Fox News rating (2020/02/03): Fox News AllSides Bias Rating

Nate Lowry
  • 151
  • 1
3

Ad Fontes Media produces the following chart, which seems to be well researched:

enter image description here

MSNBC seems to be roughly on par with the FOX News channel as a whole in terms of partisan slant, though MSNBC is positioned as being a more reliable news source. Note that FOX news programming (separated from the punditry of the rest of the channel) is viewed as equally reliable to MSNBC, and slightly less partisan.

Ted Wrigley
  • 69,144
  • 23
  • 179
  • 235
  • I ended up checking out that source as part of answering an SE.Skeptics question. The overall goal of the site looks very interesting, and the analysis isn't without some merit, though the data seemed far too sparse to be representative. Assuming that the project has been conducted in-good-faith, I'd consider it to be interesting in an academic sense as initial groundwork in the field, but not something that people should regard as providing reliable scientific analysis in a general context. – Nat Jan 31 '20 at 17:35
  • 1
    I didn't fully investigate their methodology and such, but their results struck me as suspiciously clean, especially for the amount of data that they appear to have collected. So while I wouldn't necessarily disagree with their results, I'm skeptical of the methodology used to arrive at those results. – Nat Jan 31 '20 at 17:39
  • Ridiculous. The Weather Channel has clear bias. It regularly shows that winds largely flow to the right across the globe on our standard north-up map orientation. A politically balanced view should obviously show an equal amount of left-flowing winds. – Ben I. Jan 31 '20 at 17:45
  • That's just because people on the right are bigger blowhards... – Ted Wrigley Jan 31 '20 at 17:51
3

EDIT

some other answers suggest political bias is not objective, probably because they are confounding "preference" with "observation of preference (bias)". Ambiguity in natural language obscures the issue, but objectivity is possible. Preference is subjective, but measurement of preference (i.e. observed bias) is objective. Consider supremacy of the color red (a subjective qualial preference), in contrast to current preference (i.e. bias) for red amongst infants (difficult to measure, but still objective). Likewise, favorability of party X is subjective, but observed favor for party X, though hard to measure, is still objective. Also, objectivity is not confined to non-stochastic phenomena. In cases where objective phenomena are difficult to measure, clever scientists will search for plausible proxy metrics that are easier to measure, as is the case with the following:

ORIGINAL POST

Please see this publication from UCLA, which estimates ADA scores for major media outlets. According to the introduction:

...we count the times that a media outlet cites various think tanks. We compare this with the times that members of Congress cite the same think tanks in their speeches on the floor of the House and Senate. By comparing the citation patterns we can construct an ADA score ... These findings refer strictly to the news stories of the outlets [and not] editorials, book reviews, and letters to the editor from our sample

The publication also contains a section Previous Studies of Media Bias

It's worth considering possible shortcomings of the article's described methods. Firstly, you have to be willing to accept that think-tank citation ratios that coincide with those of congressional members are an acceptable proxy for media outlet bias. This requires that we accept other assumptions prima facie, like a consistent manner of think-tank citation behavior across the political spectrum, etc.

Still, this is, to date, the most serious analysis I've seen. As regarding objectivity, the described methods seem sufficiently detailed so as to be repeated, and hence the results independently measurable and verifiable. I've intentionally omitted their conclusions - if you want to know, read the article ;)

EDIT

Here are two other studies I found:

  1. WHAT DRIVES MEDIA SLANT? EVIDENCE FROM U.S. DAILY NEWSPAPERS
  2. Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis

The first tries to compare "unique phrase usage" analysis of media outlets to members of Congress. I'm skeptical of the second, which tries to recognize bias with machine learning techniques, trained with data procured by a set of volunteers. They might have sufficed with just generalizing the volunteers' results since the AI can only ever (likely) perform as well and no better than it's training set provided by flawed humans.

And here is a data source of news chyron archives used in some more serious analyses.

Finally, an article probing the meta-concerns of determining media bias.

rcgardne
  • 31
  • 3
2

I am not looking for opinion here, I'm looking for serious analysis that presents a criteria that can be (at least reasonably) objectively measured

Here's an objectively measurable criterion: what percentage of people employed by the agency cut their toe nails once a week?

Asking "has anyone come up with an objective measure of X?" makes no sense. If X is itself not an objective measure, there is no objective way to assess if your objective measure is a meaningful measure of X.

The question you should really be asking in situations like this is "What would be the purpose of such a measure?" What makes lower bias preferable? This will tell you what you should expect from an objective measure of bias. The same applies, by the way, to "factual reporting". How exactly do you wish to weight each statement's truth value? What counts as a statement by a news organisation/how discrete do you go?

I don't think there's a good definition here, though. Gotcha side-by-side comparisons of analogous articles by the same outlet are the best you're ever likely to see on this.

something
  • 141
  • 4
-4

I don't think that measuring bias could make any sense.

A lot of studies showed that it is much easier to fool a reader or a viewer if the beginning part of the news is credible and properly argued. It is a natural weakness of the human mind, once the reporter caught our trust we are more prone to believe what follows in the reporting. So measuring the bias in some news is tricky, because a news that is largely accurate, but at the end contains some biased details might be more effective in letting the public believe that those details are true.

So. The measure of how accurate a news is does not give a measure of its capability to fool the public.

FluidCode
  • 7,140
  • 1
  • 15
  • 48