How the cops watch your tweets in real-time

from arstechnica.com: Recent leaks about the NSA's Internet spy programs have sparked renewed interest in government surveillance, though the leaks touch largely on a single form of such surveillance—the covert one. But so-called "open source intelligence" (OSINT) is also big business— and not just at the national/international level. New tools now mine everything from "the deep Web" to Facebook posts to tweets so that cops and corporations can see what locals are saying.  

Due to the sheer scale of social media posts, many tools don't even aim at providing a complete picture. Others do.

For instance, consider BlueJay, the "Law Enforcement Twitter Crime Scanner," which provides real-time, geo-fenced access to every single public tweet so that local police can keep tabs on #gunfire, #meth, and #protest (yes, those are real examples) in their communities. BlueJay is the product of BrightPlanet, whose tagline is "Deep Web Intelligence" and whose board is populated with people like Admiral John Poindexter of Total Information Awareness infamy.

BlueJay allows users to enter a set of Twitter accounts, keywords, and locations to scan for within 25-mile geofences (BlueJay users can create up to five such fences), then it returns all matching tweets in real-time. If the tweets come with GPS locations, they are plotted on a map. The product can also export databases of up to 100,000 matching tweets at a time.

A look at the BlueJay interface shows it to be a fairly basic tool, but one that gets its power from full access to the Twitter "firehose" of all tweets. Users who want to search the Twitterverse have three basic options: Twitter's search API, Twitter's streaming API, or full firehouse access to Twitter from third party providers like GNIP and DataSift. The first two are free but limited; BrightPlanet notes that even the broader streaming API returns somewhere between one and 40 percent of the relevant tweets depending on Twitter's load at the moment. The firehose requires some serious infrastructure and a paid contract with Twitter, but it provides all relevant tweets.

BlueJay partners with firehose providers and touts this fact in its marketing copy: "BlueJay captures tweets from the entire Twitter stream, unlike all other products on the market that only get a fraction of the tweets that are being posted." BlueJay is meant only for local monitoring of tweets, but this also keeps the price in reach of local police departments—$150 a month.

Of course, once you have this basic data, you may want to do more with it. Say a suspect is tweeting from a GPS-enabled phone client and appears to be dealing drugs. Forget bothering with the paperwork needed to track the phone through a cell phone provider. BrightPlanet also offers GeoTime, a separate data visualization tool that can take exported BlueJay data and mine it to show where and when the target travels, what he tweets about at various locations, and where his phone resides at night. (BrightPlanet describes this as using "pattern recognition to automatically detect and annotate time-space behaviors, such as meetings, gaps, connections, clusters, and motion.)

"The bad guys are out there, and they’re talking to each other online," BrightPlanet says in its sales pitch to law enforcement. "We’re intimately familiar with OSINT needs and deliverables, and we will deliver the Deep Web Intelligence that finds the bad guys and lets you get them behind bars."

As with most general purpose surveillance tools, though, the tech can be turned to any purpose. In a sales flyer, BrightPlanet suggests using BlueJay to "monitor large public events, social unrest, gang communications, and criminally predicated individuals" to "identify potential witness and indicators for evidence" and "track department mentions." The company has also expanded its tools to provide OSINT to the pharma and financial services industries.

Do people really tweet about things like #meth? They certainly do, though not all tweets provide any sort of actionable intelligence (and many are about Breaking Bad):

Criminals do just come right out and tweet about their crimes, but BlueJay appears to be more useful as a way to "listen in" on people who would not ordinarily be talking to police. Used well, such tools should make police departments more aware of both local problems and complaints about their own work. Used less than well, it can be a bit creepy, sort of on par with having a kid's uncle listen outside her bedroom during a slumber party. And used badly, it can make a nice tool for keeping an eye on critics/dissenters.

BlueJay is a reminder that tweets are public. It's an obvious point, but one that's easy to forget. No matter how many followers you have, your tweet stream is really being broadcast to the world. And the world is watching.