Start With A Product Objective And A Baseline
Define the decision the model supports, the user outcome, and the cost of false positives and false negatives. A single accuracy score can hide class imbalance and unequal error costs. Establish a simple rule-based baseline so the team can prove that machine learning adds enough value to justify its complexity.
Choose offline metrics that relate to the product objective, then define online guardrails such as latency, complaint rate, conversion, or manual-review load. Record which populations and conditions the evaluation set represents. A model can improve an aggregate score while harming an important segment.
Make Data And Training Reproducible
Version code, feature definitions, training data references, configuration, and model artifacts. The same feature must have the same meaning during training and serving. Validate types, ranges, missingness, categories, and freshness at pipeline boundaries. Stop or quarantine a run when data violates the contract rather than silently training on corruption.
Separate experimentation from the repeatable pipeline. A notebook is excellent for exploration but often depends on hidden state and manual steps. Move accepted transformations and evaluation into reviewed code with tests. Apply the practices from maintainable software to feature logic and data interfaces.
Engineer The Inference Path
Decide between batch and online inference based on freshness and latency. Online prediction needs bounded preprocessing, model loading, concurrency, capacity, and timeouts. Batch prediction needs scheduling, partitioning, idempotent output, and recovery after partial failure.
Define behavior when features are late, the model service is unavailable, or the input is outside the training domain. A cached result, simpler model, rule-based fallback, or manual review may be safer than an error. The API reliability guide applies directly to online model endpoints.
Monitor Data, Predictions, Outcomes, And Infrastructure
Service metrics reveal latency, errors, saturation, and resource use. Data metrics reveal missing features, distribution shifts, new categories, and freshness. Prediction metrics reveal score distribution and confidence. Outcome metrics reveal whether the product objective is being met, although labels may arrive days or weeks later.
Drift is a signal to investigate, not automatic proof that performance fell. A seasonal change may be expected; a pipeline bug may produce similar statistics. Compare with training and recent reference windows, segment results, and connect alerts to an action. Avoid dashboards with no owner or threshold rationale.
| Layer | Example signal | Possible action |
|---|---|---|
| Data | Missing feature rate rises | Stop pipeline or use defined fallback |
| Model | Prediction distribution shifts | Segment and compare inputs |
| Service | Tail latency exceeds budget | Scale, optimize, or shed load |
| Outcome | Business guardrail degrades | Rollback or reduce exposure |
Roll Out, Roll Back, And Retrain Deliberately
Compare a candidate with the current model on fixed evaluation data and recent shadow traffic. Deploy gradually, keep model and feature versions visible, and define automatic or manual rollback criteria. A/B tests need enough duration and careful interpretation when users influence one another or behavior changes over time.
Retraining should not deploy automatically merely because it completed. Re-run validation, compare metrics, and verify data lineage. Set ownership for model approval, service operations, fairness or safety review where relevant, and incident response. When something fails, use the production debugging process across data, code, model, and infrastructure.
Production readiness comes from repeatable evidence. The goal is not a model that scored well once, but a system that can explain which model ran, what data shaped it, how it behaves now, and how to recover when conditions change.
Design Human Review Where Consequences Demand It
Some predictions should recommend rather than decide. Define when low confidence, unusual inputs, or high-impact outcomes move to a qualified reviewer. Give that reviewer enough context, a way to disagree, and a clear escalation path. Human review is a product workflow, not a vague safety promise.
Analyze overrides and appeals as data while respecting privacy and access controls. They can reveal missing features, unclear policy, or harm concentrated in a subgroup. Document who can approve a model, change a threshold, stop serving, and communicate an incident.
Cost also belongs in the release decision. Track compute per prediction, accelerator utilization, queue time, and the expense of feature retrieval. A model that improves one metric slightly may not justify a large operational increase, especially when a simpler baseline remains easier to explain and recover.
Common Production ML Questions
How Often Should A Model Be Retrained?
When new data and changed conditions justify it, not simply on a calendar. Monitor outcome, drift, label availability, and retraining cost.
Does Drift Always Require A New Model?
No. Investigate whether the shift is expected, harmful, or caused by a pipeline issue. Retraining on bad data can make the problem worse.
Can We Deploy The Highest-Accuracy Model?
Only if it also meets latency, cost, interpretability, safety, and operational requirements. A slightly less accurate model may create a more dependable product.
