This content is locked. Please login or become a member.
Predictive AI or enterprise machine learning, that’s the technology that learns from data to predict in order to improve pretty much all the main large-scale operations that organizations conduct. The problem is that most of these projects actually usually fail to achieve successful deployment. So that’s why I conceived of the playbook, the titular playbook of my book, “The AI Playbook” called bizML, the business practice for running machine learning projects successfully through the deployment. I break it down into six steps to enforce deep collaboration between data scientists and business stakeholders so that everyone can get on the same page, speak the same language, and run the project end to end through to successful deployment.
The first step is to reverse plan for deployment. You have to start with the business goal, the intended purpose. That is to define the value proposition, the whole point of the project. What’s predicted and what’s done about it? Who’s going to buy, therefore market to them. Which transaction is most likely to be fraudulent, therefore investigate it or block it, et cetera. There’s so many pairs like that.
Step two of bizML is to define the prediction goal. That is the target of prediction. What the predictive model that’s learned from data will output or, that is to say, what the probability it outputs is trying to predict. That needs to be defined in greater detail. It’s one thing to say, I’m going to predict defection, which customer is going to cancel, but you actually have to be a lot more specific about that. Which customers who’ve been around for three years and have at least this kind of value to us are going to decrease their spend by ninety percent in the next three months and not increase their spend in some other channel because that doesn’t really count as losing the customer, etc., etc., etc. All the caveats and qualifiers from a business standpoint, the practical pragmatic aspects, if I’m going to be sending an expensive discount offer to these customers at risk of leaving, I better be more specific accordingly about exactly what I mean by leaving or defection. These decisions cannot be made by just data scientists alone in a cubicle. The stakeholders have business expertise that the data scientists don’t know, and that informs the decisions about all those aspects.
Step three of bizML is to establish the evaluation metrics, the performance metrics, the benchmark, the particular number. It’s just arithmetic, but it’s very particular arithmetic to define exactly how well it works, which is to say basically two things. How well does it predict sort of in a pure predictive performance sense? And those tend to be what are called technical metrics, like accuracy, precision, recall, and the data science favorite, area under the curve. But you also need to deal with business metrics, the profit, the savings, something that means something to the business, aligns with organizational goals that is understandable and relevant to the business side stakeholders. So you need to decide which metrics are most pertinent for the project and what are the goals, what level of performance needs to be attained before we’re ready for deployment.
Step four of bizML is to prepare the data. Now data prep, as data scientists always call it, is actually the biggest technical bottleneck of the project. It’s sort of the janitorial service. It’s a necessary evil. It’s unavoidable. You need to do a certain really specialized data, database programming, or data manipulation process so that the data is in the right form and format. That basically comes down to a bunch of positive and negative examples of whatever it is you’re trying to predict and everything known about each of those cases, such as what’s known about a transaction that may or may not be fraudulent.
Step five of bizML is to actually train the model. That is to apply machine learning methods, use the machine learning software on that data that you prepared in the previous step. This is the fun part. Right? This is the rocket science. It’s where it actually somehow gains insights. It ascertains patterns or formulas from that historical data that are then going to apply over new, unseen cases. And, of course, we need to use those metrics to benchmark exactly how well it does succeed in doing so, but it’s still one step short of achieving value.
Step six of bizML is to deploy the model. Its job now is to take as input what’s known about one individual case, a transaction that might be fraudulent, a customer that might buy or cancel, a delivery address that might be receiving a package tomorrow, right, and then put a number on it, a predictive score, probability of the chances of that outcome or behavior, whatever it is that we’re predicting for this project. So that’s what we’ve got. Now we need to use it. That’s what deployment is, to operationalize it, to put it into production, to actually start making those predictions on a case-by-case basis and use that prediction to directly inform those operational decisions. Only then will we be changing operations and potentially improving them that way. Only then do we gain efficiencies and, therefore, organizational value.
So after that culminating final step, step six of bizML, where you’ve deployed the model, you’re not done. You’ve actually only just started doing operations in a new way. So there is an ongoing process after that successful deployment, which is to constantly monitor and periodically refresh the model. The model was trained over historical cases where you knew the answer. Those were examples from which to learn. But those cases that were used to develop the model are becoming further and further a part of the distant past. That model, if you never change it, its performance is just going to be slowly degrading. So periodically, you refresh it. You train a new model over more recent data. You don’t necessarily have to do that very often. It really depends on the project. Sometimes it’s once a week or once a month. For the leading credit card payment fraud detection model of the world, it’s literally done once a year.