Experimental Tooling·AI & Tooling·Primary: TypeScript & Python

SLMGen

Dataset intelligence and fine-tuning infrastructure for Small Language Models.

Tooling and pipelines for fine-tuning small language models (SLMs) with dataset analysis, explainable model recommendation, and automated Google Colab notebook synthesis.

Official Repository

Problem Statement

Fine-tuning compact models (1B–8B parameters) often fails due to bad dataset curation, mismatched hyperparameters, and confusing tooling aimed solely at hyperscalers.

Intended Audience

Machine learning engineers, researchers, and indie builders fine-tuning compact language models on consumer or single-GPU hardware.

Core Architecture & Mental Model

Combines an interactive configuration interface with automated Colab notebook generation and dataset validation pipelines.

Verified System Capabilities

Dataset Intelligence: Heuristic validation of token distributions and format errors

Model Matching: Explainable scoring for selecting base SLMs (Phi, Llama, Qwen, Gemma)

Automated Colab Artifacts: Ready-to-execute Python notebooks with memory-efficient LoRA/QLoRA recipes

Quantization Configurations: 4-bit and 8-bit NF4 bitsandbytes configurations

Architectural Decisions & Rationales

Focus on sub-8B parameter models

Rationale: Allows practical deployment on edge devices and single developer workstations without multi-million dollar compute budgets.

Heuristic MinHash token deduplication

Rationale: Eliminates redundant synthetic sequences before quantization to prevent catastrophic forgetting.

Technical Trade-offs & Honest Limitations

Automated notebook generation targets Google Colab and single-node Jupyter environments; distributed multi-node clusters require custom orchestration.
Context window length is bounded by available GPU VRAM (up to 8,192 tokens on 24GB VRAM).

Empirical Verification Evidence

Public repository at github.com/eshanized/SLMGen with automated recipe generators.

Related Research & Architectural Writing

Research Paper

Parameter-Efficient Fine-Tuning Benchmarks for Sub-8B Small Language Models

Deploying generative intelligence on edge hardware and single-GPU developer workstations requires rigorous parameter-efficient fine-tuning (PEFT) of Small Language Models (SLMs) in the 1B to 8B parameter range. This report documents the empirical evaluation underlying SLMGen. We analyze LoRA rank sensitivity, 4-bit NormalFloat (NF4) quantization degradation, and dataset token hygiene heuristics across Phi-3.5, Llama-3.2, and Qwen-2.5 base models.