The ways in which “big data” is revolutionizing business are much discussed. However, less appreciated are the possible implications for security and intelligence. In future, it may become possible to confidently predict criminal or terrorist activity by combining observations from sources such as social media activity, internet searches, shopping habits and smartphone location. How can we balance society’s interest in greater safety with the risks this poses to privacy, misuse and the human right to be presumed innocent?

We are generating more and more data, from our smartphones, appliances, satellites, terminals, cameras and a range of other sensors on vehicles and machinery. Anything we do online can be traced, stored and analysed. People are increasingly willing to give up private information, linked to their real-life identities, in exchange for faster or better services. 

At the same time, computing processing power to run statistical algorithms on that data is becoming both more powerful and cheaper. A June 2014 survey by Gartner found that 73% of the surveyed organizations had already invested or planned to invest in the next two years in predictive analytics systems, to capture and analyse data in order to make better decisions.⁹

Predictive analytics encompasses a variety of statistical techniques from modelling, machine learning and data mining that analyse current and historical facts to make predictions about future, or otherwise unknown, events. Everyday examples of current uses of predictive analytics include electricity providers anticipating peaks in demand, credit scores reflecting probabilities that given individuals will default on debts, and companies profiling customers to target offers they are predicted to be likely to buy.

Increasingly, it will become possible to analyse more and more data from diverse sources: social media postings reveal who we are interacting with; credit card or e-cash purchases reveal what we are purchasing; internet searches reveal what we are thinking about; our mobile phone locations reveal where we are and in which direction we are heading; while the movies and TV programmes we stream reveal our preferences in such broad areas as love, violence, travel, culture and sport.

Predictive analytics could increasingly be used in a range of fields, such as parole boards evaluating which inmates are most likely to reoffend; the military predicting which soldiers will be able to handle life in the special forces; life insurance providers predicting when individuals will die; or governments tracking the spread of pandemics in real time to better prepare healthcare facilities and to track the migration of people to prepare infrastructure. 

A specific domain of application of predictive analytics emerging today is its use to anticipate security threats and criminal behaviour. Several police forces already analyse Twitter and text messages as part of their activities to anticipate criminality and use “predictive policing” algorithms to decide where officers should patrol by analysing what areas of a city typically see what kinds of crime at what time of day. 

In 2013, it was revealed how much information from online communications is collected by intelligence agencies such as the US National Security Agency for purposes such as attempting to predict terrorist activity. Sir David Omand argues that social media intelligence, referred to as SOCMINT, should become a full member of the intelligence and law enforcement family. The capacity of predictive analytics to prevent acts of crime and terrorism will likely be among the incentives for government agencies to collaborate on standardizing and sharing data. 

As more and more data become available and more powerful processors are able to link the information in different ways, we can only begin to imagine what kinds of security-related predictions we will be able to make from this data in the coming years. 

The growing power of predictive analytics in the realm of crime and terrorism prevention will have a profound impact on our society and democratic rights. Predictive analytics based on big data is all about correlation, not causation. Even when algorithms are highly accurate at linking patterns of behaviour to future acts of crime or terrorism, there will still be innocently intentioned individuals who match these patterns of behaviour only through coincidence. How should we balance the presumption of innocence for these individuals with society’s interest in minimizing criminality and terrorism?

There will also be the temptation for governments to use the capacity to predict and forestall behaviour more widely than crime and terrorism to encompass legitimate protest and political dissent. What forms of democratic control on the use of this type of big data application will be appropriate to limit the capacity for authoritarian use?

It is likely that different societies will reach different kinds of consensus on the appropriate limits for such technology. To what extent will it be acceptable for governments to monitor and analyse data generated by foreign visitors to their country?

Predictive analytics could significantly decrease the incidence of crime and terrorism and make the world a safer place – but only if we can find ways to minimize the risk of misuse and undesired side effects.