Yoshida is a high-performance, modular General LLM Reinforcement Learning (RL) and Post-Training Framework built natively in pure PyTorch with Zero External RL Library Lock-in (No TRL, No Ray, No Deepspeed-RL dependencies).
Designed for both agile single-GPU research and multi-GPU cluster scaling, Yoshida supports DeepSeek-R1 style GRPO (Reasoning & Math RL), Online PPO (Actor-Critic), Direct Preference Optimization (DPO), Bradley-Terry Reward Modeling (RM), and Supervised Fine-Tuning (SFT).
It pairs exact mathematical formulations with modern spectral optimizers (Dion3 with Gram Newton-Schulz, Muon, Fused AdamW), colocated in-process vLLM PagedAttention rollouts (3,300+ tokens/sec), native multi-GPU FSDP (ZeRO-3), and local structured experiment workspaces.
Explore our comprehensive guides in docs/:
| Guide | Description |
|---|---|
| π Getting Started Guide | Step-by-step installation, hardware verification, and your first 30-second training run. |
| π Dataset Preparation | Standard JSON/JSONL formatting rules for SFT, DPO, RM, and GRPO reasoning/alignment datasets. |
| βοΈ Configuration Guide | Declarative PyTorch Lightning-style YAML configuration and dynamic dot-notation CLI overrides. |
| π³ Training Recipes | Ready-to-use copy-paste recipes for GRPO reasoning RL, DPO, Multi-GPU FSDP, and vLLM acceleration. |
| ποΈ SLURM Cluster & HPC Guide | Multi-node torchrun batch scripts, InfiniBand tuning, and spot preemption auto-resuming. |
| π‘οΈ Evaluation & Benchmarks | Automated multi-axis evaluation for reasoning accuracy, instruction following, and guardrails. |
| β Troubleshooting & FAQ | Resolving CUDA Out-of-Memory (OOM), optimizer selection guidance, and NCCL tuning. |
-
Zero RL Library Lock-in: Exact mathematical formulations of:
-
GRPO: DeepSeek R1-style Group Relative Policy Optimization with Schulman
$k_3$ non-negative KL penalty. - Online PPO: Generalized Advantage Estimation ($\text{GAE}(\gamma, \lambda)$) with Value Head Critic.
- DPO: Direct Preference Optimization with zero-memory reference switching.
- Bradley-Terry Reward Modeling: Pairwise log-sigmoid preference scoring.
- SFT: Masked sequence cross-entropy.
-
GRPO: DeepSeek R1-style Group Relative Policy Optimization with Schulman
-
High-Throughput Rollout Subsystem (
align_core/rollout/):- Colocated vLLM Engine: Continuous batching and PagedAttention delivering 3,300+ tokens/second in-process on a single GPU.
- Native PyTorch Engine: Zero-dependency FlashAttention SDPA baseline (190 TPS) that runs anywhere without serving daemons.
-
First-Class Trajectory Provenance: Full tracking of behavior policy
$\mu(a|s)$ ,policy_version,base_id, and exact importance ratios.
-
Modern Spectral Optimizers:
-
Dion3: Gram Newton-Schulz (GNS) with Fractional Row Selection (up to
$6\times$ lighter than vanilla Muon with identical validation quality). - Muon: Quintic Newton-Schulz matrix polar decomposition (NeMo-RL aligned).
- Fused AdamW: PyTorch native single-kernel CUDA acceleration.
-
Dion3: Gram Newton-Schulz (GNS) with Fractional Row Selection (up to
-
Multi-GPU Distributed Engine: Native
torchrunsupport with DDP and FSDP (ZeRO-3) parameter + gradient + optimizer state sharding with auto-wrapping transformer decoder layers. -
Local-First Experiment Workspace: Structured
<project_slug>/<experiment_slug>folders containingconfig.yaml,metrics.jsonl,summary.json, rotating checkpoints, and synchronized WandB namespaces. -
Fault-Tolerant Checkpointing: Exact state restoration (weights, AdamW/Muon moments, LR scheduler curves, AMP scalers, CPU/CUDA RNG seeds) with graceful
SIGINT/SIGTERMpreemption emergency saving. - Hardware Profiler & Smoketest: Empirical GEMM matrix benchmarks (TOPS / TFLOPs), tokens/sec, and Model FLOPs Utilization (MFU %).
python smoketest.py --config configs/smoketest.yamlpython train.py \
--method grpo \
--rollout_backend vllm \
--optimizer dion3 \
--project llm-reasoning \
--experiment grpo-qwen-reasoning \
--group_size 4python train.py \
--method dpo \
--optimizer dion3 \
--project llm-alignment \
--experiment dpo-dion3-pilot \
--max_steps 20torchrun --nproc_per_node=4 train.py --config configs/fsdp_grpo_safety.yamlpython evaluate.py \
--base_model Qwen/Qwen3-0.6B \
--sft_adapter ./experiments/llm-alignment/sft-baseline/final_model \
--dpo_adapter ./experiments/llm-alignment/dpo-dion3-pilot/final_model \
--grpo_adapter ./experiments/llm-reasoning/grpo-qwen-reasoning/final_model# Build Docker image
docker build -t yoshida:latest .
# Run hardware smoketest & TFLOPs profiler
docker compose run --rm smoketest
# Run GRPO RL Training with GPU passthrough
docker compose run --rm train-grpo
# Run Multi-GPU FSDP (ZeRO-3) across all GPUs
docker compose run --rm train-fsdpyoshida/
βββ align_core/ # Core Framework Library
β βββ rollout/ # RolloutManager, PolicyVersion, Trajectory provenance, and backends (vLLM, NativeTorch)
β βββ configs/ # Dataclass & PyTorch Lightning-style YAML config parser
β βββ distributed/ # Multi-GPU context, NCCL collective comms, DDP & FSDP (ZeRO-3)
β βββ hardware/ # NVIDIA GPU auto-detection, TF32, SDPA, FLOPs/MFU profiler
β βββ logging/ # Workspace slug manager, local JSONL streaming, & WandB tracker
β βββ models/ # Native PyTorch loss math (GRPO, DPO, PPO/GAE, BT, SFT)
β βββ trainers/ # SFT, RM, DPO, PPO, GRPO trainers, CheckpointManager & Optimizers (Dion3/Muon)
β βββ data/ # Safety datasets, reasoning prompts & collators
βββ configs/ # Ready-to-use YAML configuration files
β βββ dpo_safety.yaml # DPO configuration preset
β βββ grpo_safety.yaml # GRPO configuration preset
β βββ fsdp_grpo_safety.yaml # Multi-GPU FSDP (ZeRO-3) preset
β βββ smoketest.yaml # Hardware profiling preset
βββ docs/ # Documentation Suite
β βββ GETTING_STARTED.md
β βββ DATASET_PREPARATION.md
β βββ CONFIGURATION_GUIDE.md
β βββ TRAINING_RECIPES.md
β βββ EVALUATION_AND_BENCHMARKS.md
β βββ TROUBLESHOOTING_AND_FAQ.md
βββ train.py # Unified CLI & torchrun trainer
βββ evaluate.py # Evaluation benchmark
βββ smoketest.py # Hardware GEMM TOPS/TFLOPs & training simulation profiler
βββ Dockerfile # Production NVIDIA CUDA / PyTorch image
βββ docker-compose.yml # Multi-GPU container orchestration
MIT License