You might think I’m over the top with this little tale. But the (hopefully) fictitious Narrative Data Inc could be the offspring of existing large consumer research firms, combined to semantic and data-mining experts such as Recorded Future. This Gothenburg (Sweden)-based company – with a branch in… Cambridge, Massachusetts – provides real-time analysis of about 150,000 sources (news services, social networks, blogs, government websites). The firm takes pride in its ability to predict a vast array of events (see this Wired story).
Regarding the “de-anonymising” the web, two years ago in Paris, I met a mathematician working on pattern detection models. He focused on locating individuals simply through their mobile phone habits. Even if the person buys a phone with a fake ID and uses it with great care, based on past behaviour, his/her real ID will be recovered in a matter of weeks. (As for Facebook, it recently launched a snitching program aimed at getting rid of pseudonyms – cool.)
Expanding such capabilities is only a matter of refining algorithms, setting up the right data hoses and lining up the processing power required to deal with petabytes of unstructured data. Not an issue any more. Moore’s Law is definitely on the inquisitors’ side.