Guide to install and run Bagel on Windows 11 with RTX 5090 32GB Vram

[prartio]

This was done on Windows 11, RTX 5090 32 GB Vram, Nvidia Driver 576.52

Follow the guide carefully step by step:

1) Download Bagel repository (lets assume it's "Bagel" folder on your PC) and remove
flash_attn==2.5.8, torch==2.5.1, torchvision==0.20.1 from "requirements.txt" file inside "Bagel" folder

2) Download everything from https://huggingface.co/ByteDance-Seed/BAGEL-7B-MoT/tree/main
Of course main file is "ema.safetensors", 29.2 GB
Put all files into "_\Bagel\models\BAGEL-7B-MoT_" , create all subfolders if needed

2) Install miniconda. Start "Anaconda Prompt" from Start, all next commands must be typed there:

conda create -n BagelEnv python=3.10

3) conda activate BagelEnv

All next commands must be with BagelEnv enviroment being active

4) Install Cuda Toolkit 12.8 and cudnn into BagelEnv miniconda enviroment, non system wide, to avoid conflicts with
other projects:

conda install cuda=12.8 cudnn -c nvidia -c conda-forge

5) pip install torch==2.7.0+cu128 torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

6) pip install -r requirements.txt

7) a) download "flash_attn-2.7.4.post1%2Bcu128torch2.8.0cxx11abiTRUE-cp310-cp310-win_amd64.whl" from https://huggingface.co/lldacing/flash-attention-windows-wheel/tree/f1d19914b4b710823709e11e2a88d052216a110b

pip install flash_attn-2.7.4.post1%2Bcu128torch2.8.0cxx11abiTRUE-cp310-cp310-win_amd64.whl

It successfully worked for me even though it's for Torch 2.8.0 and we've installed 2.7.0

OR

b) Later @petermg released wheel for torch 2.7.0, so you can instead download it:

"flash_attn-2.7.4.post1+cu128.torch270-cp310-cp310-win_amd64.whl" from https://github.com/petermg/flash_attn_windows/releases/tag/01

pip install flash_attn-2.7.4.post1+cu128.torch270-cp310-cp310-win_amd64.whl

8) pip install gradio

9) To start Bagel, still inside "Anaconda Prompt" and with active BagelEnv enviroment, change folder to "Bagel"
repository you downloaded in step 1) then:

python app.py

I got next output in console:

The safetensors archive passed at models/BAGEL-7B-MoT\ema.safetensors does not contain metadata. Make sure to save your model with the save_pretrained method. Defaulting to 'pt' metadata.

• Running on local URL: http://127.0.0.1:7860
• To create a public link, set share=True in launch().

---

Then browser started and Bagel UI appeared, everything works fine. CPU is not utilized much but 5090 is
under 95-100% load during image generation, editing, understanding. VRAM is almost full, like 30-30.5 GB used.
Image generation and editing takes around 55-65 seconds, image understanding takes about 0.5-5 seconds depending on
image size.

[Andy1621]

Cool!

[pupba]

I use an RTX 4090 with CUDA 12.4 and DFloat11, and created this repository for others with the same setup.
https://github.com/pupba/BAGEL-DFloat11-Windows

[Andy1621]

Excellent

[georgel]

Thank you for the guide. I followed the instructions and received this error when I tried to generate.

RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Do you have any advice on how I might try to debug this?

[prartio]

Thank you for the guide. I followed the instructions and received this error when I tried to generate.

RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Do you have any advice on how I might try to debug this?

Try to use different wheel at step 7, if you tried a) then pip uninstall flash_attn and install wheel from b) , or vice versa

[hirokith]

I followed the instructions but got ouput:

(bagel) PS D:\dev\BAGEL> python app.py
The safetensors archive passed at models/BAGEL-7B-MoT\ema.safetensors does not contain metadata. Make sure to save your model with the `save_pretrained` method. Defaulting to 'pt' metadata.
language_model.model.embed_tokens.weight:   0%|               | 0/1223 [00:00<?, ?w/s, dev=cuda:0]

and then it exists, do you know how to solve this issue? Thank you!

By the way my GPU is RTX 3060, should I use other versions for cuda or torch to resolve the issue?

[prartio]

I followed the instructions but got ouput:

(bagel) PS D:\dev\BAGEL> python app.py
The safetensors archive passed at models/BAGEL-7B-MoT\ema.safetensors does not contain metadata. Make sure to save your model with the `save_pretrained` method. Defaulting to 'pt' metadata.
language_model.model.embed_tokens.weight:   0%|               | 0/1223 [00:00<?, ?w/s, dev=cuda:0]

and then it exists, do you know how to solve this issue? Thank you!

By the way my GPU is RTX 3060, should I use other versions for cuda or torch to resolve the issue?

The title of the guide is quite clear I think - you need RTX 5090 with 32 GB of VRAM.

[cleverestx]

I use an RTX 4090 with CUDA 12.4 and DFloat11, and created this repository for others with the same setup. https://github.com/pupba/BAGEL-DFloat11-Windows

pip install triton-windows-3.2.0.post11 is failing for me with
ERROR: Could not find a version that satisfies the requirement triton-windows-3.2.0.post11 (from versions: none)
ERROR: No matching distribution found for triton-windows-3.2.0.post11

(using default install on that page), What is missing? My CUDA is 12.4 and all steps up to that worked fine.

[iChristGit]

I use an RTX 4090 with CUDA 12.4 and DFloat11, and created this repository for others with the same setup. https://github.com/pupba/BAGEL-DFloat11-Windows

pip install triton-windows-3.2.0.post11 is failing for me with ERROR: Could not find a version that satisfies the requirement triton-windows-3.2.0.post11 (from versions: none) ERROR: No matching distribution found for triton-windows-3.2.0.post11

(using default install on that page), What is missing? My CUDA is 12.4 and all steps up to that worked fine.

Download the whl from here:
https://huggingface.co/madbuda/triton-windows-builds

Place it in the bagel main folder and execute pip install triton-3.0.0-cp310-cp310-win_amd64.whl

[georgel]

I use an RTX 4090 with CUDA 12.4 and DFloat11, and created this repository for others with the same setup. https://github.com/pupba/BAGEL-DFloat11-Windows

pip install triton-windows-3.2.0.post11 is failing for me with ERROR: Could not find a version that satisfies the requirement triton-windows-3.2.0.post11 (from versions: none) ERROR: No matching distribution found for triton-windows-3.2.0.post11
(using default install on that page), What is missing? My CUDA is 12.4 and all steps up to that worked fine.

Download the whl from here: https://huggingface.co/madbuda/triton-windows-builds

Place it in the bagel main folder and execute pip install triton-3.0.0-cp310-cp310-win_amd64.whl

Installing triton solved the problem I was experiencing above. I don't quite understand how they are related but it works now.

[prartio]

Guys, I found interesting fact about using Bagel from own Python scripts. There is line of code:

max_mem_per_gpu = "40GiB" # Modify it according to your GPU setting. On an A100, 80 GiB is sufficient to load on a single GPU.

Of course I set it to "32GiB" for my 5090, and when I tried to use image understanding from my python script, I noticed next messages in the console output:

Auto device map: OrderedDict([('language_model.model.embed_tokens', 0), ('language_model.model.layers.0', 0), ('language_model.model.layers.1', 0), ('language_model.model.layers.2', 0), ('language_model.model.layers.3', 0), ('language_model.model.layers.4', 0), ('language_model.model.layers.5', 0), ('language_model.model.layers.6', 0), ('language_model.model.layers.7', 0), ('language_model.model.layers.8', 0), ('language_model.model.layers.9', 0), ('language_model.model.layers.10', 0), ('language_model.model.layers.11', 0), ('language_model.model.layers.12', 0), ('language_model.model.layers.13', 0), ('language_model.model.layers.14', 0), ('language_model.model.layers.15', 0), ('language_model.model.layers.16', 'disk'), ('language_model.model.layers.17', 'disk'), ('language_model.model.layers.18', 'disk'), ('language_model.model.layers.19', 'disk'), ('language_model.model.layers.20', 'disk'), ('language_model.model.layers.21', 'disk'), ('language_model.model.layers.22', 'disk'), ('language_model.model.layers.23', 'disk'), ('language_model.model.layers.24', 'disk'), ('language_model.model.layers.25', 'disk'), ('language_model.model.layers.26', 'disk'), ('language_model.model.layers.27', 'disk'), ('language_model.model.norm', 'disk'), ('language_model.model.norm_moe_gen', 'disk'), ('language_model.model.rotary_emb', 'disk'), ('language_model.lm_head', 'disk'), ('time_embedder', 'disk'), ('vae2llm', 'disk'), ('llm2vae', 'disk'), ('latent_pos_embed', 'disk'), ('vit_model', 'disk'), ('connector', 'disk'), ('vit_pos_embed', 'disk')])

See all those 'disk" words? I also noticed that only 16 GB of my VRAM was occupied during my python script execution, and it was doing image understanding like 5-10 times slower that in browser gui! I realised that most likely it doesn't fully load 29.2 GB weights file into memory when I use Bagel from script.

So, I just changed that line of code to:

max_mem_per_gpu = "80GiB"

80 GB! Even though I only have 32, but the file with weights, ema.safetensors is only 29.2 GB, so it should be okay. And it solved all problems! No more that console log output with 'disk' words which probably means that part of the model were reading from disk during inference. VRAM is fully ocuppied, same as when I use Bagel from browser manually. And main thing that image understanding from my python script is as fast as in browser, i.e. 0.5 second per image.

TLDR: if you are using Bagel from your own scripts, and you have 32 GB of VRAM (i.e. RTX 5090 GPU), set:

max_mem_per_gpu = "80GiB"

[TurtlesAI]

while I had been using bagel on my 5090 before this, I'm now using the d11 version which is much faster and should not be lossy. Try it yourself.

[iChristGit]

while I had been using bagel on my 5090 before this, I'm now using the d11 version which is much faster and should not be lossy. Try it yourself.

I wonder how fast it is on your 5090.
My 3090Ti feels so slow although the 24 gigs

[wyn1015]

相同的问题：ema.safetensors does not contain metadata. Make sure to save your model with the save_pretrained method. Defaulting to 'pt' metadata.

• Running on local URL: http://127.0.0.1:7860 80GB显存，这个端口能打开，但是生图的时候无法生成，图像编辑也不可以，是为什么呢