We live in a changing world.
The amount of data that we create is growing at a rate that is far outpacing our ability to extract business value from it. It is not just the volume of the data, it is the complexity of that data. Most data science solutions are designed for volume. Few are designed for complexity. Fewer still excel at both volume and complexity.
At Ayasdi we are focused on the challenge of extracting business value from massively complex data through the design, development and deployment of intelligent applications.
Our approach, our underlying technology and our products are expressly crafted to deliver against an enterprise’s requirements in this area. The reason is that for intelligence to be broadly consumed within an enterprise it needs to be presented in the framework that enterprises consume most broadly. Applications are that framework.
Like the ubiquitous Office suite or the new school apps like Slack or Concur, applications dominate what we do and how we work. They will all benefit from intelligence. But with every company claiming to be an AI company these days – how does one distinguish truth from fiction?
We think that intelligence is far more than just prediction. We think that intelligent applications have multiple facets. To achieve this we operate on five pillars. We consider these to be critical to the successful deployment of intelligence within an organization. Some of these elements may seem self-evident, but a single element – standing by itself, will not allow an enterprise to successfully transform itself.
To be successful you need all of the elements working in conjunction with each other.
The five pillars of an intelligent system are:
Let’s take each of these concepts in turn.
The Five Pillars
Discovery is the ability of an intelligent system to learn from data without being presented with an explicit target. It relies primarily on the use of unsupervised and semi-supervised machine learning techniques (such as segmentation, dimensionality reduction, recommendation systems, anomaly detection etc.).
Usually, in enterprise software, the term discovery refers to the ability of ETL/MDM solutions to discover the various schemas of tables in large databases and automatically find join keys etc.
Our use of the term is very different and has important implications.
In complex datasets, it is nearly impossible to ask the “right” questions. To discover what value lies within the data one has to understand all of the relationships that are inherent and important in the data. That requires a principled approach to hypothesis generation.
Ayasdi’s unique technology, topological data analysis (TDA), is exceptional at surfacing hidden relationships that exist in the data and identifying those relationships that are meaningful without having to ask specific questions of the data.
As a result, enterprises can now discover answers to questions they didn’t even know to ask.
Once the data set is understood through intelligent discovery, supervised approaches are applied to predict what will happen in the future. These types of problems include classification, regression and ranking.
For this pillar, Ayasdi uses a standard set of supervised machine learning algorithms including random forests, gradient boosting, linear/sparse learners. The discovery capabilities of our technology are highly useful in that they generate relevant features for use in prediction tasks or finding local patches of data where supervised algorithms may struggle.
The learn and predict phase is an important part of the business value associated with data science, however, generally, in predictive analytics, there exists a notion that this is the sum total of machine learning.
This is not the case by far. Prediction, while important, requires other elements to be meaningful in an operational environment.
It requires explanation and trust.
Prediction without justification does not meet the standard of intelligence. It is far easier to build a black box prediction engine with high accuracy and low explainability, however, if the predictive qualities of the model are unrelated to the actual drivers of the business it will fail when conditions change.
For prediction to have value it must be able to justify its assertions. This capability is what makes prediction intelligent.
Justification and transparency build trust.
No business leader should deploy intelligent and autonomous applications against critical business problems without a thorough understanding of what variables power the model. Enterprises cannot move to a model of intelligence applications without trust and transparency.
The process of operationalizing an intelligent application within the enterprise requires some change in the organization, an acceptance that the application will evolve over time and that will demand downstream changes – automated or otherwise.
For this to happen, intelligent applications need to be “live” in the business process, seeing new data and automatically executing the loop of Discover, Predict, Justify on a frequency that makes sense for that business process. For some processes that may be quarterly, for others daily. That loop can even be measured in seconds.
Intelligent applications are designed to detect and react when data distributions evolve. As a result, they need to be “on the wire” in order to detect that phenomena before it becomes a problem.
Too many solutions provide an answer in a point of time, a intelligent system is one that is always learning through the framework outlined here.
Intelligent systems are designed to detect and react as the data evolves. An intelligent system is one that is always learning, live in the workflow and constantly improving. Anything less is simple analytics.
Additional Elements for Consideration
For an organization considering how to apply machine intelligence in their enterprise, the five pillars represent the starting point. To scale that effort, however, requires additional considerations involving scale, strategy and organizational transformation. These are discussed briefly below:
Seek Enterprise Scale
Performing small scale experiments creates a false sense of security for many enterprises. Wins executed against sterile data or in an operational vacuum are not likely to translate well when asked to scale to real world scenarios. Seek solutions that are proven in enterprise environments for scalability, performance and compute.
Enterprise scale also means that your AI strategy needs to integrate into your existing infrastructure – this means Active Directory for User Management, leveraging existing data lake infrastructure and respecting the security and compliance guidelines for operational systems. Only with these considerations in place can AI permeate the enterprise.
For most solutions the business level interfaces are constrained to Microsoft Office products and as a result the create barriers between business leaders and the data. Further, these tools are not sufficient for real time collaboration and cannot accommodate the scale of truly operationalized systems. To deliver value to the business, the user experience must connect the business user with their data in an meaningful way.
Putting an organization on the path to intelligence requires support from the business, based on real business problems. While innovation groups are excellent scouts to the research and evaluation of new platforms, they are typically not tasked to direct show business value. Mainstream your initial applications into your businesses, as you seek to build trust, experience and deliver results.
Companies don’t need to startups nor be in Silicon Valley to possess the agile development mindset. Enterprises that commit to fast timelines learn faster. The speed at which an organization learns will define their competitiveness in the coming decade. Commit to moving as rapidly as possible.
Focus on Data Availability over Data Quality
Our experience is that enterprises spend too much time fretting over the quality of their data when they should be determining how to feed these intelligent systems more data. More data means better outcomes and “exhaust” data or “old” data have far higher utility than most organizations are willing to give credit for. Worry less about null values and think more about about adding additional sources. Intelligent systems grounded in unsupervised learners will determine what is valuable and explain why.
Be Ready for Process Change
Intelligent systems will change how you perform certain business processes. Recognizing this fact ahead of time will enable the enterprise to capitalize on the knowledge and to consolidate the wins – thereby building momentum for the future applications of intelligence.
Build a Center of Excellence
At the heart of a successful transformation sits a center of excellence. The COE is where best practices are developed, process change is accelerated and prioritizations are made based on operational readiness, business need and other considerations.
For the COE, select the use cases that have the right ‘fit” along these dimensions:
- Sponsorship – sufficient business sponsorship to support the organizational changes that will be a result of operationalizing the use case
- Value – ensure this is a high value use case for the line of business (not a science project)
- Measurability – the use case can deliver quantifiable value (ROI)
- Fit & Feasibility – the use case fits within the organizational mission
- Operational – use case can be made operational
- Environment – there is infrastructure that supports the use case
- The next generation of leadership will come from an enterprises AI COE. Staff it accordingly.
Enterprises that adopt machine intelligence will outperform enterprises that don’t. Intelligence in business applications is not a fad, it is an inflection point. Intelligence is not reserved for Google, Facebook, Amazon and Microsoft, it is available to any enterprise with the sophistication and drive to transform their business.
Intelligence will define the winners and loser over the coming decade – perhaps less.
As a company, Ayasdi has pioneered the development of these intelligent systems. The five pillars outlined at the start of this paper: Discover, Predict, Justify, Act and Learn represent all the necessary components of an intelligent system. Choosing one, two or even three components will deliver but a fraction of what the entire system would deliver.
A commitment to establishing an intelligence driven enterprise requires a commitment to every principle – however, the payoff for those enterprises making that investment will be immense.