My Big Data Resolution for 2015

Figure 1. NewVantage Partners study of Fortune 1000 technology executives.

I generally like precision, and I strive for it in my writing and thinking. It certainly helps in communicating ideas., However, I have been pretty relaxed in how I use the term, Big Data. I’ve used the term to refer to the size of the data set (e.g., “We have Big Data.”). I have even used it to describe a method (e.g., “We are using a Big Data approach.”). Still other times, I have used it as a general label (e.g., “Big Data is the future.”).

In a survey of Fortune 1000 technology executives, 30% of the executives found the term, Big Data, confusing and misleading and 53% said it was overstated (see Figure 1). With my inconsistent use of the term, perhaps I am merely contributing to their dislike of the term.

So, I am making a New Year’s resolution for 2015. I will be more precise when I use the term, Big Data.

So, how will I use the term, Big Data? A good place to start is to see how others use it and define it. Berekely solicited over 40 data experts for their definition of Big Data. These definitions could be reduced into six broad areas. These six areas represent the problems/challenges we face in the realm of Big Data. When you tackle a data-intensive project, you will likely need to answer questions in each of these six areas:

1. Processing of 3Vs: The first area is about the nature of the data including the size (i.e., volume), speed (i.e., velocity) and complexity (i.e., variety) of the data and the technology to process it all.


2. Analytics: Analytics is about the types of analyses we perform on the data. There are two general approaches when analyzing data: the exploratory approach (machine learning) and the hypothesis-driven approach (human judgement, critical). I look at these two approaches as complementary to each other. While the hypothesis-driven approach requires scientists to set up and test pre-determined hypotheses, machine learning can augment that process by quickly analyze millions relationships that might lead to potential areas of interest that can then be reviewed, and perhaps pursued, by scientists with more formal testing. For example IBM Watson helped researchers at Baylor College ingest and review 70,000 research papers on a cancer protein. It identified six new proteins to target for cancer research.


3. Insights: The analytics leads to insights. When we consider the insights we gain from our analytics, we need to ensure the results are reliable, valid and useful. We often times turn to data scientists to help us make sense of the analytics and to verify its veracity.


4. Data Integration: Because analytics is all about finding relationships among variables in our data sets, businesses are looking to integrate their data silos to uncover interesting relationships across different metrics. The value of integrated data sets is greater than the sum of its parts.


5. Communication: We need to be able to share our results to interested parties. Toward this end, technology vendors are creating visualization tools to improve how we communicate our results.


6. Security/Privacy/Ethics: Finally, the era of the quantification of everything has resulted in issues surrounding the misuse of data as they relate to security breaches, and privacy and ethics concerns. In fact, a recent study of Chief Information Officers found that security is the number one challenge of Big Data projects (51% indicated it as a top challenge). Some major brands that have been the target of security breaches include Target, Home Depot, Neiman Marcus and JP Morgan Chase. Other examples include the US National Security Agency’s monitoring of individuals worldwide as well as Facebook’s experiments on attitudes using unsuspecting users.

“Big Data” is a Phenomenon

As these definitions of Big Data show, Big Data is not one thing; it is about a lot of different things. Each of these things is not new or unique to Big Data; we have had to address each of them decades ago. These things are simply more relevant in today’s overly-quantified world. So, in 2015 and beyond, when I’m asked to define Big Data, here is how I’m going to answer:

“Big Data is a phenomenon that is the result of the quantification of everything.”

My Big Data Resolution for 2015