Big Data is coming. We have databases running into terabytes (TB), containing every click we've ever made, our weekly and monthly purchases recorded on loyalty cards, our ins and outs of buildings and the country. All of these data can be analysed, and decisions made. 

Is "Big Data" the same as small data, but ? And if not, what is "Big Data"?

In traditional statistical analysis, an analyst asks a question of a bunch of data. For example, a healthcare business analyst looks at 1 million records on hospital admissions, and asks "which specialties are my most expensive?", or "which patients are my most expensive patients?" This is great for the kind of datasets that we are used to working with, where we are merely going from a few hundred records to a few million – a question of scale, not capability.

financial institutions are leading the way on this. The credit card company wants to reduce fraud, so they want to know what "normal" represents for each of their 6 or 7 million customers, and then in near real-time, they want to check each transaction against this "normal behaviour" model.I've had calls two or three times, and at least two have been transactions that I didn't recognise, that turned out to be fraudulent – stopped before the money had left my account. With traditional statistical techniques, the sheer number of people needed to ask the specific questions would ruin a bank's profitability. So banks use "Big Data", software that itself analyses the data, doing the kinds of things that someone might do with a few idle processor cycles, instead of going out for a doughnut at Starbucks. 

Making use of the same techniques in healthcare

The NHS is a profit centre for the UK. Not in the sense that itself makes money, but in the sense that it makes the whole country more productive, more effective, and more consumer friendly. Perhaps the best analogy is Kaiser Permanente,the healthcare provider arm of a large construction company.

Kaiser needed a way to attract workers to work on its large construction projects all over the world. Being a US company, it offered free healthcare workers and their families, and that proved attractive. Then someone at Kaiser realised that getting people well after an illness meant that the company had paid for someone to be sick for 3, 4, 5 weeks.surely it made more sense to keep them well? So they put more emphasis on primary care – on general practitioners and family practitioners – and rewarded hospitals are supporting the general practitioners to keep people at work. It had a knock-on effect; by keeping people's families and children (and elderly parents) well, Kaiser Permanente kept Kaiser's workers free from worry so they could do their best to the job at hand. nobody is saying that Kaiser Permanente is as good as the NHS, but Kaiser Permanente has clear aims, there is the NHS's original aims have been lost in the midst of time and targets.

Kaiser Permanente has become so successful, that health insurers in the US (the equivalent of our NHS commissioners) use Kaiser Permanente's services to look after people with long-term conditions, to prevent the deteriorating. It's worth looking at motivation here – on the one hand, health insurers want to make sure that their customers get good enough care that they keep getting more customers; on the other hand, they want that care to cost as little as possible. Kaiser Permanente uses "big data" statistical software – same software used by the financial institutions above: SAS for health providers.

We can do the same. we want our population to live healthy fulfilling lives.  We want them to be made while when they are sick, and NHS is good at that. but we want to go further than that – we want to keep them well, not only because it keeps them contributing to society and productive, but also because it costs less.

Traditional statistical analysis to reduce hospital admissions is focused on the only data we could obtain them, hospital data. Tools such as PARR, and the latest offering from Canada – LACE, require that someone has already been into hospital before they will start analysing their likelihood of being readmitted. We want to identify people who have lifestyles  or early symptoms that indicate that their health is deteriorating.

The Kings Fund developed a tool called CPM (Combined Predictive Model) which use data from hospital and from primary care to work out how to predict which people of the population of East and West Surrey were most likely to need hospital treatment in the next year. It was great, but it hasn't been updated to 5 years.  Croydon used it as the basis for their Virtual Ward. They used SAS to build this.

we're hoping to go one stage further. We're hoping to include social care data and Mosaic data along with hospital and primary care and community care data, to make better predictions. These predictions will then be given to a multidisciplinary admissions team will provide that human sanity check. We'll then intensify the coordination of the different services, to keep people well, productive, and contributing to society. And we expect that the total cost of health care will be less, freeing up resources for other people who need them.

We expect that Big Data will make possible conclusions that we haven't even imagined yet.