Case study 02 / Data Science · ML Engineering
From experiments to monitored models.
Three connected studies in causal inference, inventory forecasting, and churn prediction, built on one synthetic e-commerce dataset.
Synthetic-data case studies
The problem
How can a data scientist decide whether a product change works, forecast inventory needs, and keep a churn model useful over time?
What I built
Built ExperimentIQ for treatment-effect analysis, ForecastIQ for forecast-driven inventory simulation, and ModelGuard for churn prediction, model registration, API serving, and drift monitoring. All three use a shared synthetic dataset for a fictional company.
How it works
A shared deterministic data generator supplies the studies. ExperimentIQ checks assignment and leakage before estimating effects. ForecastIQ compares forecasting models and inventory policies. ModelGuard uses historical customer snapshots, an MLflow registry, and a FastAPI prediction endpoint.
- Synthetic data
- Analysis & features
- Evaluation
- Decisions & serving
Engineering decisions
Audit before interpreting
ExperimentIQ checks sample-ratio mismatch and device-switching leakage before analyzing revenue and conversion. CUPED uses a pre-period revenue covariate.
Separate signal from certainty
Conversion increased by 6.9% relative in the synthetic experiment (p=0.036). The revenue change was not statistically significant (p=0.373). They support different conclusions.
Monitor more than drift
ModelGuard compares a classifier with a recency baseline on a later time window. It tracks feature and prediction drift alongside observed predictive performance.
Results & evidence
Out-of-time model AUC
0.740
12,330 synthetic customer snapshots in the current-period evaluation; naive recency baseline AUC 0.565.
Relative conversion lift
+6.9%
10,000 synthetic customers; p=0.036. This is a simulated experiment, not revenue or impact at an employer.
Results are documented in the linked project artifacts. They have not been independently reproduced for this portfolio.
Limits & lessons
ShopSphere is fictional and all business outcomes are simulated. Results depend on the data generator, baseline choices, and evaluation windows. They demonstrate analytical methods rather than customer impact.
The recommendation should follow the evidence, including uncertainty and cases where a simpler baseline is competitive.