Big Data, Data Analytics and Now... Dark Data

A few days ago the world woke up to the news that Apple had acquired Lattice, an AI company specializing in dark data management at the “bargain” price of a cool 200 million USD.

But beyond the sum or Steve Jobs’ infamous brand lies the spectre of dark data.

What is it? Where is it? What can we do with it? What exactly makes it “dark”?

Today everything revolves around the exploitation of data and the value it brings to business, especially when it comes to driving profits.

What is Dark Data?

Dark data refers to information that is in our possession but is somehow “hidden” from us. It’s the gigabytes of information created every day by our businesses and systems that, for whichever reason, are not used to our benefit. Big consulting firms and technology companies like Gartner, Forrester and IBM have already singled out dark data as one of Big Data’s greatest challenges.

Datumize technology enables the capture of different dark data sources. One of the most important of these is called ephemeral data (temporary or in transit),  which refers to query operations or interactions between people and computer systems that are not stored anywhere.

Think of all the times you check something like the price, availability, or characteristics of a product, or even a customer profile.

Datumize technology captures the ephemeral data that comes up within the CPD in a non-intrusive manner, so it neither affects the performance nor develops anything new. It is basically the same technology currently used by intelligence and espionage services to identify and capture data and even conversations, but applied to business purposes.

The data can be used to obtain information, analyze it and make decisions.

Where Is Dark Data Found?

The nature of dark data is wide-ranging and varied, from the billions of transactions that occur in e-commerce (B2B, B2C) where Datumize captures complete web services dialogues, web searches or mobile applications, as well as internal systems like a CRM or ERP. Dark data can also refer to information about the movement of people and devices, usually by leveraging the ubiquity of WiFi or other positioning systems. By tracking movements or concentrations of people and vehicles, dark motion data can complement existing business intelligence in order to make new or better informed decisions.

The world of machines is another exciting source of dark data.

The majority of machines (for example, in a factory or a hospital) was not designed to be connected to the Internet, or even interoperable with modern artificial intelligence and data analysis systems. In most cases, accessing the information residing in these systems is not possible through a traditional data load. That’s why Datumize technology also includes a device that enables users to capture dark data from the outside world and connect it to different machines, no matter how complex or varied their protocols.

The creepy control center moves into capture dark data.

What Can Dark Data Do?

We are all aware that our increasingly digital world generates a very high data volumes, but we rarely exploit them. Large consulting firms claim that more than 80% of the data stored in our Big Data systems are not actually used in business management. So why capture and store data without extracting any value from it? Keep in mind that dark data are not necessarily huge in volume; sometimes they can be very manageable and even stored in a conventional database. Take product queries that have not materialized in a sale, for instance: a careful analysis of these queries could provide the company with an immediate sales boost.

Herein lies the beauty of dark data: it’s sitting right before our eyes, but like a jewel behind bulletproof glass it has been too costly or complex to capture… until now.

Datumize helps to identify, capture and process dark data from all kinds of sources, extract the most valuable insights and send them to storage systems. Once combined with existing business intelligence, companies can use dark data insights to reveal new dimensions of information, which is the key to making more profitable decisions.

Stay connected with us via Linkedin and Twitter

written by Javier Uria