November 10, 2026 | 8 Minute Read
AI models rarely break loudly. The model continues to serve predictions and the infrastructure appears to be functioning normally. Warning signs, such as customers receiving less relevant recommendations, fraud cases slipping through, or business forecasts becoming less reliable, emerge gradually. When teams finally investigate, they often discover that the model's performance started declining weeks or even months earlier, with no clear record of when the drift began or why it happened.
MLOps best practices can help prevent these issues. In this blog post, we cover six ways model drift and related failures emerge in production and introduce a practical fix borrowed from code quality practices: the ratchet.
MLOps Best Practices
MLOps is the set of practices that run a machine learning model as an operated system, including automated training and deployment, monitoring of prediction quality, and named ownership of both.
Google's guide to MLOps pipelines and automation levels and Microsoft's documentation on model management and deployment in Azure Machine Learning cover the full lifecycle.
The MLOps best practices below target the period after launch, when a model meets data it was not trained on and nothing in the infrastructure reports a problem.
1. Recognize Six Ways Models Degrade in Production
The six failure modes below run from the data, through the system, to the organization.
Data drift: The statistical properties of the input data shift away from what the model trained on, a common cause of model drift. A model trained on pre-pandemic consumer spending patterns does not know that behavior changed permanently in several categories. Drift is the normal condition of any system trained on a snapshot of a world that keeps moving.
Feature skew (also called training-serving skew): The features computed at training time and the features computed at serving time diverge. Two different code paths, often maintained by two different teams, compute the same feature. "Orders in the last 30 days" can mean a rolling calendar window in the batch training pipeline and a fixed billing cycle in the real-time serving layer, and the two produce meaningfully different numbers. The model is fine. The inputs it receives in production differ from the inputs it trained on.
Silent failures: An upstream component degrades. A feature pipeline starts returning nulls that get imputed to zero. A third-party data feed changes its schema and a join starts failing. The model keeps producing confident-looking predictions on degraded or missing inputs, because nothing in the stack was told to object.
Missing retraining pipelines: The team ships the model, celebrates, and never builds the automated path to retrain it as new ground truth arrives. A model trained on last year's labels keeps scoring this year's customers. Retraining becomes a manual, occasional, deprioritized project, and the gap between what the model knows and what is currently true widens every week.
Poor monitoring: Most teams watch infrastructure health, meaning latency, uptime, and error rates, and call it observability. Those signals say nothing about whether the model's predictions are still good. Model monitoring has to track the quality signal itself: prediction distributions, calibration against outcomes as they arrive, and performance broken out by segment.
Governance gaps: MLOps governance means named owners and documented decisions. Without them, quality regressions sit unresolved while teams with competing priorities debate whose problem they are.
2. Verify the MLOps Platform Actually Exists
We worked with a construction hardware manufacturer that believed it already had an MLOps platform. The client was building an AI-powered bidding tool that reads architectural drawings and RFP documents and generates a bill of materials automatically. When the engagement started, the client's team described the platform as in place.
Once we asked the basic operating questions, it turned out none of the pieces existed:
no system for tracking model accuracy in production
no error collection
no human-in-the-loop path for feeding corrections
The platform existed only as a line on a slide. Building the governance layer, including error collection, a retraining cadence, and clear ownership of the quality floor, became a bigger part of the engagement than the machine learning work itself. The client had never assigned anyone to own these responsibilities.
A serious custom AI and ML development engagement has to close that gap.
3. Apply the Ratchet to Model Quality
The ratchet pattern is a mechanism that stops quality metrics from decreasing. A checked-in file records the current floor for each metric, such as coverage percentage (which cannot fall) or lint violations and type errors (which cannot rise). Any change that would push a metric past its recorded floor is blocked at the merge gate.
The quality gate sits in front of model deployment and, ideally, in front of the retraining pipeline's decision to promote a new model version. A retrained model that scores meaningfully worse than the deployed model on any tracked metric (outside the tolerance bands described below) does not get promoted unless a human signs off on the tradeoff explicitly.
In most organizations a model degrades and nothing stops it, because nothing was built to stop it. With a ratchet in place, a drop below the recorded floor is the event that requires action, and staying at or above the floor happens automatically.
4. Replace Hard Floors With Tolerance Bands
Lint violations and coverage percentage are deterministic. Measure an unchanged codebase twice and you get the same number twice. Model quality metrics are not. Accuracy on a rolling window of live traffic fluctuates from sampling variance alone, even when the model is unchanged and the data is stationary. A direct port of the code ratchet produces a hard floor that treats noise as regression.
A worked example shows the scale of the problem:
Baseline accuracy: 91%
Weekly predictions: 2,000
Standard error from sampling: √(0.91 × 0.09 / 2,000) ≈ 0.64 percentage points
Problem with a hard floor: Setting the threshold at 91% means roughly half of ordinary weeks will fall below it due to natural sampling variation.
A more practical threshold: Use a band of three standard errors around the baseline: 91% ± (3 × 0.64%), giving a range of approximately 89.1% to 92.9%.
What this means: The band helps flag changes that exceed normal sampling noise, reducing unnecessary deployment blocks and repeated threshold resets.
The fix is to treat the floor as a tolerance band around the baseline, borrowing from statistical process control. A metric that stays outside its historical band for several consecutive windows signals a genuine change, while a single dip followed by a recovery is more likely to be noise. A ratchet that cannot distinguish between the two will either generate constant false alarms or lose the team's trust and eventually be disabled.
Common mistake: A common mistake is building the band from the weeks right after launch. Build it from the held-out validation set or a labeled window you have verified as healthy. If drift started in those weeks, the band encodes the degraded state as the acceptable normal, and every later alert is measured against a baseline that was already wrong.
The defense is the same discipline the code ratchet requires for coverage-lowering commits. Floor and band resets need an owner, a recorded reason, and ideally a second reviewer. Recalibrating the baseline becomes a deliberate, audited decision instead of a quiet way to keep a degrading model shipping under the label of expected variance.
5. Map the Ratchet to the Six Failures
Each failure mode maps to a specific piece of this infrastructure, which helps prioritize where to invest first.
6. Accept What the Ratchet Cannot Judge
None of this replaces judgment about qualitative model behavior.
Do the model's errors matter to the business?
Is a slight accuracy drop in a low-stakes segment worth blocking a deployment that fixes a real problem in a high-stakes one?
Is a technically calibrated model making decisions nobody would defend if asked to explain them out loud?
Goodhart's law applies as it does in code: a tracked metric becomes a target, and a target can be optimized without improving the thing you care about. A ratchet on accuracy alone can reward a model that gets more confidently, precisely wrong on the segment nobody thought to track.
Choosing what to track deliberately is the first job. The ratchet is the floor beneath which quality cannot fall silently, and a human still has to ask periodically whether the floor measures the right thing.
Give the Pipeline Authority to Object
MLOps best practices come down to one structural change. Models die after deployment because nothing objects when they start to. Once something in the pipeline has the authority to object, the quiet three-month decline becomes a decision someone had to make on the record.
If you are earlier in the process and still deciding what to build rather than what to protect, the same habit of tracking the right thing applies to model selection and design. For your highest-stakes production model, who would get paged if its accuracy quietly dropped ten points, and how would they find out?
If you want a second set of eyes on your highest-stakes production model, talk to Improving's AI and ML team. We will review your monitoring, retraining, and ownership setup and show you where a quiet decline could slip through.
