Add veRL GRPO training recipe for gpt-oss-20b on g5.12xlarge

[nkumaraws]

Summary

• Add complete veRL GRPO (Group Relative Policy Optimization) recipe for training openai/gpt-oss-20b (20B MoE) on g5.12xlarge (4x A10G 24GB GPUs) with HyperPod EKS
• Custom multilingual language compliance reward function — model must reason and answer in the same language as the user's question
• Full evaluation pipeline with vLLM batch inference (50-question test set, 5 languages)

Architecture

• 6 nodes: 3 GPU workers (12 GPUs total) + 1 Ray head (CPU only) + 2 spare
• veRL + FSDP2: Full CPU offload (offload_policy=True) for both params and optimizer on 24GB GPUs
• vLLM inline inference: gpu_memory_utilization=0.45, enforce_eager=True (no CUDA graphs)
• Ray orchestration via KubeRay on HyperPod EKS

Training Results

Validated on HyperPod EKS cluster (trl-gptoss-eks), 80 steps:

Metric	Value
Steps completed	80
Final val reward	5.65 / 6.0
Step time	~2.5 min
Total training time	~3.5 hours
Checkpoint size	~117 GB (FSDP shards)

Evaluation (GRPO vs SFT baseline):

• GRPO: 98% reasoning correct, 80% answer correct
• SFT baseline: 96% reasoning correct, 74% answer correct
• GRPO improved answer language compliance by +6%

Files

File	Description
Dockerfile	Updated with gpt-oss-20b dependencies
README.md	Expanded with GRPO recipe docs, memory optimization, troubleshooting
recipe/run_gptoss_grpo.sh	Training launcher (ray job submit)
recipe/language_reward.py	Custom reward function (language detection + scoring)
recipe/evaluate_gptoss.py	50-question vLLM evaluation across 5 languages
recipe/evaluate_gptoss.sh	Evaluation wrapper script
setup/env_vars.example	g5.12xlarge + p5en.48xlarge config templates
setup/load_data_gptoss.sh	HuggingFaceH4/Multilingual-Thinking data prep
setup/raycluster.yaml	Updated KubeRay manifest (num-gpus=0 head, OOM killer disabled)

Testing

• Trained 80 steps on 3x ml.g5.12xlarge workers (12 GPUs)
• HF checkpoint saved and evaluated successfully
• No OOM during training or checkpoint save
• EFA networking validated (NCCL over OFI)

Related

• Companion PR: OpenRLHF GRPO recipe (same task, different framework) — for framework comparison

[KeitaW]

Review — veRL GRPO Recipe for gpt-oss-20b on g5.12xlarge

Clean, well-scoped PR. Three minor findings below (1 inline on the Dockerfile tag, 1 on pre-existing EFA version, 1 trailing newline). The code quality and documentation are excellent.

---

Things That Look Great

• The run_gptoss_grpo.sh header is outstanding. The "WHY THESE SETTINGS?" section with 10 numbered lessons learned from real OOM debugging is exactly the kind of documentation that prevents users from repeating the same mistakes. Each point explains not just what to set but why, with specific memory numbers.
• The troubleshooting section in the README covers 8 real failure modes (OOM during vLLM init, backward pass, NCCL hang, disk full, zombie jobs, fp32 default, expandable_segments, EFA config) — each with symptom, root cause, and fix.
• Shell scripts consistently use set -xeuo pipefail — the strictest safety settings.
• The env_vars.example dual-config pattern with commented p5en and active g5 blocks, each with inline explanations, makes it easy for users to switch between instance types.
• The reward function (language_reward.py) follows the veRL compute_score API cleanly, with a well-documented scoring formula and clear separation of concerns.
• The raycluster.yaml parameterization (replacing hardcoded resource values with ${WORKER_CPU}, ${WORKER_MEMORY}, ${FI_EFA_USE_DEVICE_RDMA}) makes the manifest work across instance types without forking.

[KeitaW]

Base image tag vllm011.latest resembles a rolling tag

The .latest suffix is ambiguous — it could be a fixed tag that was named with "latest" in it, or a rolling tag that the verlai maintainers update in place. If it's the latter, builds become non-reproducible.

Could you confirm this is a fixed tag? If so, a comment noting that would help. Otherwise, I'd suggest switching to a more explicitly versioned tag or a digest-based pin.

Reference: CONTRIBUTING.md — "External dependencies must be pinned to a specific version or tag (no latest)."

[KeitaW]

Pre-existing EFA version 1.43.3 is below CI-enforced minimum

This line isn't changed by the PR, but since the Dockerfile is being modified, CI's version check may scan the file and flag EFA_VERSION=1.43.3 (minimum is 1.47.0). I'd suggest bumping it while the file is being touched to avoid a potential CI failure.

Suggested change

	ARG OPEN_MPI_PATH=/opt/amazon/openmpi/
	ENV EFA_VERSION=1.47.0

[KeitaW]

Missing trailing newline

The .editorconfig specifies insert_final_newline = true. Just needs a newline at the end of the file.

Suggested change

	Without `NCCL_NET=ofi` and the correct `LD_LIBRARY_PATH`, NCCL silently falls back to TCP, giving much worse inter-node bandwidth.
	Without `NCCL_NET=ofi` and the correct `LD_LIBRARY_PATH`, NCCL silently falls back to TCP, giving much worse inter-node bandwidth.

[KeitaW]

Few nits