Discovery is the ability of an intelligent system to learn from data without being presented with an explicit target. It relies on the use of unsupervised and semi-supervised machine learning techniques (such as segmentation, dimensionality reduction, anomaly detection, etc.), as well as more supervised techniques where there is an outcome or there are several outcomes of interest.
In complex datasets, it is nearly impossible to ask the “right” questions. To discover what value lies within the data one must understand all the relationships that are inherent and important in the data. That requires a principled approach to hypothesis generation.
Ayasdi’s unique technology, topological data analysis (TDA), is exceptional at surfacing hidden relationships that exist in the data and identifying those relationships that are meaningful without having to ask specific questions of the data. Ayasdi’s output is able to represent complex phenomena, and is therefore able to surface weaker signals as well as the stronger signals. This permits the detection of emergent phenomena.
As a result, enterprises can now discover answers to questions they didn’t even know to ask.
Once the data set is understood through intelligent discovery, supervised approaches are applied to predict what will happen in the future. These types of problems include classification, regression, and ranking.
For this pillar, Ayasdi uses a suite of supervised machine learning algorithms including random forests, gradient boosting, and linear/sparse learners. The discovery capabilities of our technology are highly useful in that they generate relevant features for use in prediction tasks or finding local patches of data where supervised algorithms may struggle.
Prediction is generally where the rubber hits the road in terms of business value, however, there exists a notion that this is the sum total of machine learning. This is not the case. In fact, those that claim to be doing AI by only executing against the Predict pillar – aren’t really doing AI. Prediction requires other elements to be meaningful in an operational environment.
It requires explanation and trust.
We believe that intelligence needs to support interaction with humans in a way which makes outcomes recognizable and believable to them. For example, when one builds a predictive model, it is important to have an explanation of how the model is doing what it is doing, i.e. what the features in the model are doing in terms that are familiar to the users of the model. This is because this kind familiarity is important in generating trust. Similarly, in the same way that automobiles have mechanisms not just for detecting the presence of a malfunction, but also for specifying the nature of the malfunction and suggesting a method for correcting it, so one needs to have a “nuts and bolts” understanding of how an application is working in order to “repair” it when it goes awry.
For prediction to have value it must be able to justify and explain its assertions, as well as to be able to diagnose failures. These capabilities are what make prediction intelligent.
Justification and transparency build trust.
No business leader should deploy intelligent and autonomous applications against critical business problems without a thorough understanding of what variables power the model. Enterprises cannot move to a model of intelligent applications without trust and transparency.
The process of operationalizing an intelligence within the enterprise requires some change in the organization, an acceptance that the application will evolve over time and that will demand downstream changes – automated or otherwise.
For this to happen, intelligence (likely presented through an application) need to be “live” in the business process, seeing new data and automatically executing the loop of Discover, Predict, and Justify on a frequency that makes sense for that business process. For some processes that may be quarterly, for others daily. That loop can even be measured in seconds.
Intelligent applications are designed to detect and react when data distributions evolve. As a result, they need to be “on the wire” in order to detect that phenomena before it becomes a problem.
Too many solutions provide an answer in a point of time, an intelligent system is one that is always learning through the framework outlined here.
Ayasdi’s technology is built around these pillars – from the capability to combine supervised and unsupervised learning with full justification and wrap it in an application that continues to learn from the underlying data.
The core of that technology is a framework for machine learning called Topological Data Analysis. While only one component of the overall technology stack at Ayasdi, it is an important one – and it is unique to the company. To learn more about TDA visit our Resources section and look over the several hundred published papers that speak to its power and explainability.