Today is both exciting and humbling for the team here at Ayasdi. This morning, the World Economic Forum (WEF) announced that Ayasdi has been selected as a member of the 2015 class of Technology Pioneers, considered by many to be the most prestigious honor that an emerging company can garner.
The WEF is an extraordinary organization. It is the pre-eminent global institution for public-private cooperation and its aim is to improve the state of the world by engaging business, political, academic, and other leaders of society to shape global, regional, and industry agendas.
Past Technology Pioneers chosen by the Forum include some of the most transformational companies in the world, companies like Google, Twitter, AirBnB, Kickstarter, Dropbox, Altassian and Palantir. We are delighted to join such esteemed company at such an extraordinary time in our company’s development and in the development of technology more broadly.
Think about where we stand today. We capture more data in an hour today than was captured in decades, even centuries previously. The data gets the headlines – but it is not the story.
The story is what lies hidden in that data.
What lies hidden within that data represents the opportunity to solve some of our most pressing societal challenges – from disease, to climate change, to healthcare and customer understanding.
Underpinning this data explosion are major advances in data storage technology and architecture. Quality-adjusted prices for data storage equipment fell at an average annual rate of nearly 30 percent from 2002 to 2014. With an incremental cost to store data effectively at zero, institutions have responded by capturing everything possible – with the idea that what lies within will produce meaningful value for the enterprise.
Despite the technical advances in collection and storage – knowledge generation lags.
This is a function of how companies approach their data, how they conduct analyses, how we automate learning through machine intelligence.
At its heart, it is a mathematical problem.
For any dataset (n) there are a number of questions or hypotheses that accompany that dataset. Mathematically, that is 2n – an exponential function.
Exponential functions are difficult enough for humans to comprehend, however, to further complicate matters, data is growing exponentially, and is about to hit another inflection point as the IoT kicks in.
What that means is that we are facing double exponential growth in the number of questions that we can ask of our data.
If we choose the same approaches that have served us over time – iteratively asking questions of the data until we get the right answer – we will at the very least delay the discovery of answers to our most pressing question – and may not find others at all as combinatorial complexity pushed the finish line further away.
So much of what is developed today seeks to facilitate question asking or hypothesis development. While highly tuned algorithms or “code free” queries are noteworthy, they miss the central premise – asking questions of today’s large and complex data sets is not a viable approach. This method, one that has served science for centuries, has been rendered obsolete by the explosion of data.
To truly unlock the value that lies within our data we need to turn our attention to the data, setting aside the questions for later.
This too, turns out to be a mathematical problem.
Data, it turns out has shape. That shape has meaning. The shape of data tells you everything you need to know about your data from its obvious features to its secret secrets.
In understanding the shape of data we understand every feature of the dataset, immediately grasping what it is important in the data – dramatically reducing the number of questions to ask, thereby accelerating the discovery process. If you start with answers, the questions get profoundly easier to manage.
By changing our thinking – and starting with the shape of the data, not a series of questions (which very often come with significant biases) we can extract knowledge from these rapidly growing, massive and complex datasets.
Changing thinking doesn’t come easily – particularly if you don’t have the capacity to derive shape from data. It leads you down the same road, again and again. You may get there faster but you end up in the same place every time.
The opportunity presented by the World Economic Forum is to use their platform to educate enterprises, academics and governments on how to think about the problem of complexity differently – to challenge the long held notions about how to develop knowledge from data.
We are delighted to receive such a significant acknowledgement – but also realize that such acknowledgements come with enhanced expectations. On behalf of the entire team at Ayasdi – we accept the challenge.