Fellow Bloggers – My role is to create & deliver digital products and solutions that help deliver value to the customer and increase customer loyalty. As an architect of these solutions I am constantly striving to effectively leverage Big Data, NLP and data science techniques. However, when it comes to data science I always struggle with the concepts of machine learning (ML) and artificial intelligence (AI). In this blog I embark on a quest to find a way to set apart the concepts of ML & AI and to simplify the decision of when to apply which of these two concepts.
In just the past couple of years, ML/ AI have magically penetrated into all aspects of our service industry - from automating a manual process to driving cars to offering self-help assistance to recommending next best offers to automation of complex decision making. So the question becomes are these algorithms "simulating" the human or just "mimicking" the human. Do they better the human ability or just replicate the human? Every leading source I have read agrees that ML is a subset of AI but it is not always clear that ML is not AI. Furthermore, all of us are bombarded with claims of solutions where they tout the use of ML /AI and invariably these concepts are used interchangeably.
Knowing the difference is key as it drives the nature of the solution starting with the acquisition of data to choice of the algorithms to the data science model chosen to product placement and product positioning decisions. One also needs to have this awareness to decide if these techniques are suitable for being incorporated into a mission critical auditable enterprise process or if the outcome of the solution is a socially responsible outcome.
The following two definitions from the post The Difference Between AI & ML really helped lend great clarity.
"AI stands for artificial intelligence, where intelligence is defined as the ability to acquire and apply knowledge.
ML stands for machine learning where learning is defined as the acquisition of knowledge or skills through experience, study, or by being taught."
Here is a synthesis of the literature on these topics. It is not by any means a comprehensive study but enough to help me make choices.
ML is the algorithm that can learn from labeled data corpus. This data corpus captures decisions previously made by human experts. Applying ML techniques that are taught "domain decision" patterns allow one to arrive at predictable outcomes with great accuracy and speed. The caveat is that the data used for training is a good representation of the variations in the "domain". ML is attractive in problem domains that have "repeatable" patterns and solution paths but where there are large scale transaction volumes/ interactions. ML has the ability to deal with immensely complex data sets to sift through to a) find the relevant parameters b) make sense of large scale variations in the parameter values. A human would find it tedious to deal with these data volumes to find these patterns that lead to a judgement. Again, this only works long as the data sets have the similar characteristics as the data on which the ML algorithm received training.
On the other hand, if the algorithm involves constantly "identifying" and "incorporating" external influences into the decision making process, especially if the algorithm has not been “trained” in making the connection between these external inputs /influences and the response it is likely to qualify to be in the realm of AI. In addition, these external influences are completely unique, and not just variations in values of known data parameters. The idea is that the AI model has to figure out a way to deal with these unknowns and stumble upon a solution pathway. Here the algorithm has just undertaken a leap that is typical of what would be done by a human. Analogous to the human intelligence it is sensing the surroundings, with a stream of unique inputs and is using these to constantly reshape the runtime nature of the algorithm which results in an outcome that may or may not be appropriate or effective. Thus, the realm of AI is an attempt to simulate human cognitive behavior that results in evolving outcomes that are not always predictable or optimal.
How well an AI does in terms of simulating human behavior, how broad a task is being attempted and how culturally/ socially appropriate the response is to the environment are all topics that are beyond the scope of this discussion.
Is this too much of a simplification? Is this abstraction that is common to architects a benefit or disservice?
Please stay in touch ...
Popular posts from this blog
It is one thing to read about Internet of Things (IoT) and get dazzled by the commercial opportunities it offers based on the stats like the number of connected devises there are in the world today. Or how more and more consumer products are getting connected to the "grid" to enable remote monitoring/ operations. Despite all that, it is unclear as to how an enterprise would be able to make a strategic decision about benefits of an investment in IoT. How would it know if their business model or product portfolio or customer base would gain from this investment? I am looking for any case studies, market research material that might help in this analysis. Are there players (established industrial giants and/ or manufacturing heavy-hitters) who have adopted this technology and gained market share or helped make significant product improvements and /or branch out into services not possible before the advent of IoT. Some notable companies come to mind in this space -
A lot of talk has been heard lately about the concept of data lake. Variously known as, data refinery, data factory etc. I find it interesting that we now hear logical architectural terms that speak to the concepts and to the purpose of the big data technologies such as Hadoop / HDFS and Apache distributed database technologies such as HBase / Cassandra . This may be indicative of a shift. What I am not sure of is does this mean that there is a level of maturity that has been achieved by this suite of open source technologies? Or could this point to the fact that these technologies have practical applications that solve enterprise scale problems? Or does it show that enterprises have realized that they are no longer able to just deal with "structured data" and that a vast majority of information lies in the space of "unstructured content" leaving them no choice but to venture into the realm of big data technologies? Not really sure! The fact re