Doing Business in a Big Data World

The Internet and cloud computing have revolutionized the nature of data capture and storage, tempting many companies to adopt a new ‘Big Data’ philosophy: collect all the data you can; all the time. But this new world of Big Data is proving to be much more demanding and complex than expected, requiring companies not only to adopt different technologies, but also to make significant changes to their business strategy, internal skillsets, and organizational structures.

Big Data is Not Just More Data: That’s because the nature of the data we can now collect has changed. Big Data involves not just the structured data (customer name and details, products purchased, how much was spent and when, etc.) that every company is used to capturing, but also unstructured data (data scraped from the Internet and social media channels that may come in a wide variety of formats, from video to voice). It may seem an obscure data management distinction, but this shift toward collecting unstructured data – which is a large part of what Big Data is all about – is sending shock waves across traditional organizations.

Why?

New technologies: The consistency and predictability of structured data is what gave rise to today’s centralized IT departments – running SQL/Relational Database Management Systems, and ERP and CRM software. Structured data is clean and predictable; easy to collect, organize and analyze – and is usually securely stored on a company’s internal servers.

Unstructured data is very different. It is messy, variable, and difficult to interpret; captured from a wide variety of sources and usually stored on distributed computing nodes through the Cloud. In order to capture, filter, and analyze huge amounts of unstructured data, an organization needs to leverage the new technologies that include massively parallel processing (MPP) and NoSQL/Hadoop-like (non-relational) database and software frameworks.

But getting access to these Big Data technologies means either buying it and building it into your organization’s current IT enterprise structure, or accessing the tools and data storage through Cloud-based offerings (or a combination of both). Whatever method your organization chooses, it will place a significant strain on your – probably already overwhelmed – central IT department, because they will need to construct a complimentary architecture with these two very different platforms.

New Skills: These new technologies also require very different skills, because in order to convert unstructured, randomly collected data into meaningful intelligence, organizations now need data engineers who understand the new NoSQL/Hadoop-like programming languages, and data scientists (often with PhDs in mathematics or statistics) who can set up the algorithms and correctly interpret the data. Employees with these skills are still expensive and hard to find: nearly 80 percent of companies complain that they’re already finding recruitment for these jobs either ‘challenging’ or ‘extremely difficult.’

New Organizational Structures: And because 80% of Big Data projects cross business lines or functions, surveys show that an alarming one third of Big Data projects are already being pursued without the support of the company’s central IT function, by adventurous departments tempted to access Big Data tools unilaterally over the Cloud – often then running that data through the (unsecured) smartphones of their employees. All of this means that CIOs and centralized IT department staff no longer ‘own’ company data the way they did in the past; giving rise to new concerns about data integrity and security in a more distributed and anarchic company structure.

As a result, Big Data is forcing CIOs to relinquish some of their centralized authority in favor of a power-sharing arrangement – not only with departments (such as Sales and Marketing) who want to run their own customer-focused Big Data projects independently, but with important new C-level rivals: the Chief Marketing Technology Officer, or the Chief Data Officer.

A New Company Data Strategy: This means that a Big Data company strategy requires a good deal of equanimity on the part of these various competing and overlapping roles to create policies that allow for departmental independence, but don’t expose the company to data breaches and misuse of customer data. And that’s why adopting a Big Data strategy also requires senior management to be much more involved in IT and data-related matters – providing clear guidance on what data can be used, in what way, and by whom.

In short, there is a lot more to doing business in the Big Data world than simply collecting and analyzing more data. It is a disruptive paradigm shift that most companies have yet to make.

Return to the Business Performance Assessment Program Forum

About the Author

Dale Neef is a technology advisor, and author of Digital Exhaust: What Everyone Should Know About Big Data, Digitization and Digitally Driven Innovation (FT Press). A veteran of knowledge management, business intelligence, and large-scale technology implementation projects with more than fifty companies worldwide, he has been a technical consultant for the Asian Development Bank, has worked for IBM and Computer Sciences Corporation, and was a fellow at Ernst & Young’s Center for Business Innovation. A frequent contributor to journals, and a regular speaker at technology conferences, he earned his doctorate from Cambridge University, was a research fellow at Harvard, and has written or edited eight books on the economics of knowledge and data management and the use of information technology to mitigate risk. Learn more at www.daleneef.com.