Truth Goggles

(via Boing Boing) One of the many awesome applications online that exists now but probably won't work for about 5 years is "Truth Goggles," an app produced by Dan Schultz, an MIT master's graduate.  In theory, the app highlights any factual claim in a news article, ranking its trustworthiness and explaining the context.

In practice, the app barely works.  It uses PolitiFact to fact check, and that gives it a library of only about 5,500 facts.    Schultz hopes that the app might ultimately cooperate with other resources, like FactCheck.org and Snopes.  Further, it only highlights for exact matches of the phrases as they're stored.  The code is open-source, but (based on my incredibly low level of qualification to make this call) this seems more like the kind of problem you need Google to solve, not the crowd.

In the article in Nieman Journalism Lab, Andrew Phelps reports the three major problems Schultz has identified:

 1. Paraphrase detection

You’re unlikely to see Truth Goggles work on the vast majority of news articles. Truth Goggles matches only exact instances of fact-checked phrases. Taking the example from the top, a reporter could have written: Romney said the unemployment rate for Hispanics has increased from 10 percent to 10.3 percent since President Obama took office. That sentence would be invisible to Truth Goggles.

Figuring this out is the Holy Grail of automated fact-checkers, Schultz said.Natural language processing is advancing in its quest for code to understand language the way we do, but truly reliable NLP is a long way off. And if the software gets close but still messes up, highlighting the wrong claim would just confuse the user.

2. Scale

Truth Goggles is limited to those claims which PolitiFact has checked — an impressive corpus of journalism, sure, but a wimpy number compared to all of the things politicians have ever claimed. You could add FactCheck.org’s database to the mix. And Snopes, if it ever released an API. Say that gets the number up to 15,000. “That’s not nearly enough to create a system that will be actually relevant on a regular basis,” Schultz said. “Let’s say everything was perfect…you’d still rarely see a highlight.”

[...]

3. User interface

Setting aside the back-end wizardry, the front-end design of Truth Goggles proved to be a massive project of its own. For Truth Goggles to work, the software has to interrupt a user’s reading without driving him or her crazy.

[...]

Then there is the matter of color. Truth Goggles always highlights text in neutral yellow. Red and green are automatic cues — False! True! — which can defeat the purpose of the software. Red and green are so final, literally opposites on the color spectrum. That reflects the false polarity of truth, not the continuum. (In fact, PolitiFact uses six flavors of “true” and “false.”)