Support for Qwen3.5 with verl

[TtLuckyyy]

Question

Hello, may I ask whether the verl framework currently supports Qwen3.5? I downloaded the rllm repository in early March. I noticed that after upgrading my environment and modifying rllm to train Qwen3.5, the results were relatively poor, whereas Qwen2.5 showed improvement. I’m not sure if something wasn’t properly adapted or configured, and I would appreciate your guidance.

Context

None

Relevant Code / Config

Environment

No response

[JinyanSu1]

Hi, which dataset are you using? and what's the generation length you are setting? Qwen3.5 response is pretty long, maybe you could check your response cut off rate?

[listar2000]

Hi @TtLuckyyy do you mind sharing which dataset/training script you are using?

[TtLuckyyy]

I am working on CUDA code generation. Below is my rLLM configuration script for training Qwen3.5-4B.

TRAIN_ARGS=(
"algorithm.adv_estimator=grpo"
"data.train_batch_size=32"

"data.max_prompt_length=8192"
"data.max_response_length=16384"

"actor_rollout_ref.model.path=$QWEN35_4B"
"actor_rollout_ref.hybrid_engine=True"
"actor_rollout_ref.actor.optim.lr=1e-6"
"actor_rollout_ref.model.use_remove_padding=True"
"actor_rollout_ref.actor.loss_agg_mode=seq-mean-token-mean"

"actor_rollout_ref.actor.ppo_mini_batch_size=4"
"actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=1"
"actor_rollout_ref.actor.use_dynamic_bsz=True"
"actor_rollout_ref.actor.ppo_max_token_len_per_gpu=24576"

"actor_rollout_ref.actor.clip_ratio_high=0.28"
"actor_rollout_ref.actor.kl_loss_coef=0.001"
"actor_rollout_ref.actor.kl_loss_type=low_var_kl"
"actor_rollout_ref.actor.entropy_coeff=0"

"actor_rollout_ref.actor.ulysses_sequence_parallel_size=1"
"actor_rollout_ref.model.enable_gradient_checkpointing=True"

"actor_rollout_ref.actor.fsdp_config.param_offload=True"
"actor_rollout_ref.actor.fsdp_config.optimizer_offload=True"
"actor_rollout_ref.ref.fsdp_config.param_offload=True"

"actor_rollout_ref.rollout.name=vllm"
"actor_rollout_ref.rollout.mode=async"
"actor_rollout_ref.rollout.tensor_model_parallel_size=1"
"actor_rollout_ref.rollout.enforce_eager=False"
"+actor_rollout_ref.rollout.engine_kwargs.vllm.disable_cascade_attn=True"

"actor_rollout_ref.rollout.temperature=0.6"
"actor_rollout_ref.rollout.gpu_memory_utilization=0.45"

"actor_rollout_ref.rollout.n=4"
"actor_rollout_ref.rollout.val_kwargs.n=4"
"actor_rollout_ref.rollout.val_kwargs.temperature=0.6"
"actor_rollout_ref.rollout.val_kwargs.top_p=0.95"

"actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=1"

"algorithm.kl_ctrl.kl_coef=0.001"

"rllm.mask_truncated_samples=True"
"rllm.agent.max_steps=3"
"rllm.stepwise_advantage.enable=True"
"rllm.stepwise_advantage.mode=per_step"
"rllm.kernel_group_reward.enable=True"

"trainer.critic_warmup=0"
"trainer.logger=['console']"
"trainer.resume_mode=disable"

"trainer.project_name=$TRAINER_PROJECT_NAME"
"trainer.experiment_name=$TRAINER_EXPERIMENT_NAME"

"trainer.val_before_train=False"
"trainer.n_gpus_per_node=8"
"trainer.nnodes=1"

"trainer.save_freq=50"
"trainer.test_freq=0"

"trainer.default_hdfs_dir=null"
"trainer.default_local_dir=$DEFAULT_LOCAL_DIR"

"trainer.total_epochs=1"

)

The dataset and corresponding prompts cannot be shared for now, as my research is still ongoing and I’m not yet in a position to disclose them. However, I used the same dataset when training both Qwen2.5-7B-Instruct and Qwen3.5-4B.

[TtLuckyyy]

Hi, which dataset are you using? and what's the generation length you are setting? Qwen3.5 response is pretty long, maybe you could check your response cut off rate?
Hello, my average output token length is between 5,000 and 10,000, so it seems that no truncation has occurred, as I set the maximum output length to 16,384.

[jacklanda]

Any update? Please show us an off-the-shelf example for running Qwen3.5 series models with verl backend.