Parameter-Efficient Fine-Tuning Benchmarks for Sub-8B Small Language Models

1. Introduction & Motivation

Hyperscale 70B+ parameter models dominate media headlines, yet production infrastructure demands predictable latency, privacy, and cost boundaries. Small Language Models (1B–8B parameters) can match or exceed large models on specialized domain tasks when fine-tuned with high-quality token hygiene.

However, fine-tuning SLMs presents distinct failure modes:

1. Catastrophic Forgetting: Due to fewer redundant parameters, aggressive learning rates quickly destroy the base model's general reasoning.
2. Quantization Noise in 4-bit (NF4): In models under 3B parameters, 4-bit weight rounding can degrade token probability distributions more severely than in 70B models.
3. Dataset Format Drift: Mismatched chat template markers (e.g. <|im_start|> vs [INST]) introduce silent loss spikes.

We engineered SLMGen (github.com/eshanized/SLMGen) to automate dataset diagnostics and synthesize memory-optimal training recipes for developer workstations.

---

2. Experimental Setup

We evaluated three base model architectures across coding and reasoning tasks:

• Phi-3.5-mini-instruct (3.8B parameters)
• Llama-3.2-3B-Instruct (3.2B parameters)
• Qwen-2.5-Coder-3B-Instruct (3.1B parameters)

Each model was trained on a specialized synthetic systems programming dataset (Rust POSIX and kernel bindings, 25,000 instruction pairs) using an NVIDIA RTX 4090 (24GB VRAM).

+---------------------------------------------------------------+
|                       RAW JSONL DATASET                       |
+---------------------------------------------------------------+
                                |
                                v
+---------------------------------------------------------------+
|              SLMGen DATASET HYGIENE ENGINE                    |
|   - Token Length Distribution Check (99th percentile trim)    |
|   - Deduplication via MinHash LSH                             |
|   - Chat Template Alignment (Auto-detect BOS/EOS markers)     |
+---------------------------------------------------------------+
                                |
                                v
+---------------------------------------------------------------+
|             AUTONOMOUS RECIPE SYNTHESIZER                     |
|   - Compute optimal LoRA Rank r in {8, 16, 32}                |
|   - Target modules: q_proj, k_proj, v_proj, o_proj            |
|   - Generate standalone Google Colab / local Python notebook  |
+---------------------------------------------------------------+

---

3. Empirical Results & Findings

3.1 LoRA Rank vs Memory Footprint

Base Architecture	LoRA Rank ($r$)	LoRA Alpha ($\alpha$)	VRAM Usage (GB)	Epoch Time	Target Loss
Qwen-2.5-Coder-3B	8	16	7.4 GB	18 min	0.842
Qwen-2.5-Coder-3B	16	32	8.6 GB	21 min	0.812
Qwen-2.5-Coder-3B	32	64	11.2 GB	27 min	0.806
Phi-3.5-mini (3.8B)	16	32	10.1 GB	25 min	0.824
Llama-3.2-3B	16	32	9.2 GB	22 min	0.835

Observation: For models under 4B parameters, increasing LoRA rank beyond $r = 16$ yields negligible loss improvement (+0.006 delta) while increasing peak VRAM by 30%. Thus, $r = 16$ with $\alpha = 32$ represents the optimal Pareto frontier for 3B SLMs.

3.2 Full Precision 16-bit vs QLoRA 4-bit (NF4)

Contrary to conventional assumptions, QLoRA NF4 quantization incurred an imperceptible 1.2% quality loss on coding benchmarks while reducing baseline VRAM by 52%, allowing full batch size 4 training on consumer hardware.

---

4. Conclusion & Software Integration

These benchmark configurations are directly baked into SLMGen's automated recipe generator, ensuring developers can fine-tune compact language models with validated hyperparameters. See github.com/eshanized/SLMGen for code and notebook generators.

@techreport{roy2026slm,
  title={Parameter-Efficient Fine-Tuning Benchmarks for Sub-8B Small Language Models},
  author={Roy, Eshan},
  institution={Tonmoy Infrastructure & Vision},
  number={SLM-RES-2026-004},
  year={2026},
  month={October},
  url={https://eshan.tonmoyinfrastructure.org/research/parameter-efficient-fine-tuning-slm}
}