Case study 05 / ML Engineering · Data Science
When fine-tuning does not win.
A controlled comparison of LoRA, DoRA, and IA3 against zero-shot and few-shot math-reasoning baselines.
ML experiment
The problem
Does parameter-efficient fine-tuning improve multi-step math reasoning when training data and compute are constrained?
What I built
Built a training and evaluation harness for LoRA, DoRA, and IA3, with answer extraction, per-category reporting, saved predictions, and zero-shot/five-shot baselines.
How it works
The adapter runs use Llama-3.2-3B-Instruct, 6,000 examples, two epochs, seed 42, and a single V100 GPU. Evaluation records accuracy, extraction failures, latency, and output length. A ten-epoch IA3 follow-up probes longer training.
- GSM8K
- Shared training budget
- Adapters & baselines
- Held-out evaluation
Engineering decisions
Keep the baseline visible
All three adapters underperformed the zero-shot model. IA3 is the best adapter in the recorded comparison, but not the best overall method.
Inspect more than accuracy
Track answer extraction failures and generation length alongside correctness to distinguish task failure from output-format behavior.
Test the longer-training hypothesis
IA3 accuracy fell from 0.6975 at two epochs to 0.6262 at ten. More epochs did not repair the gap under these conditions.
Results & evidence
Zero-shot accuracy
75.59%
Recorded GSM8K test evaluation, Llama-3.2-3B-Instruct, seed 42.
Best adapter: IA3
69.75%
6,000 training examples and two epochs; below the zero-shot baseline. LoRA: 63.99%; DoRA: 63.38%.
Results are documented in the linked project artifacts. They have not been independently reproduced for this portfolio.
Limits & lessons
This is one model, dataset, seed, and constrained training recipe. The experiment does not establish a universal ranking of adapter methods. Identical hyperparameters are a controlled comparison, not per-method tuning.
A strong experiment can conclude that the simplest method is best for the tested budget.