We need Offline support. proper offline support. internet access is not only uneccesary, it doesnt even work as intended when it connects.

[poopinyap]

Did you already ask in the discord? -- No - I do not use discord

You verified that this is a bug and not a feature request or question by asking in the discord? - No

Running 1 process
Loading Flux2 model
Loading transformer
Moving transformer to CPU
Loading Qwen3
torch_dtype is deprecated! Use dtype instead!
torch_dtype is deprecated! Use dtype instead!
config.json: 100%|##########| 728/728 [00:00<?, ?B/s]
model.safetensors.index.json: 32.9kB [00:00, 155kB/s]
WARNING:huggingface_hub.file_download:Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: pip install huggingface_hub[hf_xet] or pip install hf_xet
WARNING:huggingface_hub.file_download:Xet Storage is enabled for this repo, but the 'hf_xet' package is not installed. Falling back to regular HTTP download. For better performance, install the package with: pip install huggingface_hub[hf_xet] or pip install hf_xet
model-00001-of-00005.safetensors: 1%|1 | 52.4M/4.00G [01:22<1:43:27, 635kB/s]
Error running job: An error occurred while downloading using hf_transfer. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.

Result:

• 0 completed jobs
• 1 failure
  ========================================
  Traceback (most recent call last):
  Traceback (most recent call last):
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\file_download.py", line 474, in http_get
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\file_download.py", line 474, in http_get
  hf_transfer.download(hf_transfer.download(
  ExceptionException: : Error while removing corrupted file: The process cannot access the file because it is being used by another process. (os error 32)Error while removing corrupted file: The process cannot access the file because it is being used by another process. (os error 32)
  The above exception was the direct cause of the following exception:
  The above exception was the direct cause of the following exception:
  Traceback (most recent call last):
  Traceback (most recent call last):
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\run.py", line 120, in
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\run.py", line 120, in
  main()main()
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\run.py", line 108, in main
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\run.py", line 108, in main
  raise eraise e
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\run.py", line 96, in main
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\run.py", line 96, in main
  job.run()job.run()
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\jobs\ExtensionJob.py", line 22, in run
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\jobs\ExtensionJob.py", line 22, in run
  process.run()process.run()
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\jobs\process\BaseSDTrainProcess.py", line 1604, in run
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\jobs\process\BaseSDTrainProcess.py", line 1604, in run
  self.sd.load_model()self.sd.load_model()
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\extensions_built_in\diffusion_models\flux2\flux2_model.py", line 184, in load_model
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\extensions_built_in\diffusion_models\flux2\flux2_model.py", line 184, in load_model
  text_encoder, tokenizer = self.load_te()text_encoder, tokenizer = self.load_te()
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\extensions_built_in\diffusion_models\flux2\flux2_klein_model.py", line 43, in load_te
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\extensions_built_in\diffusion_models\flux2\flux2_klein_model.py", line 43, in load_te
  text_encoder: Qwen3ForCausalLM = Qwen3ForCausalLM.from_pretrained(text_encoder: Qwen3ForCausalLM = Qwen3ForCausalLM.from_pretrained(
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\transformers\modeling_utils.py", line 277, in _wrapper
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\transformers\modeling_utils.py", line 277, in _wrapper
  return func(*args, **kwargs)return func(*args, **kwargs)
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\transformers\modeling_utils.py", line 4900, in from_pretrained
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\transformers\modeling_utils.py", line 4900, in from_pretrained
  checkpoint_files, sharded_metadata = _get_resolved_checkpoint_files(checkpoint_files, sharded_metadata = _get_resolved_checkpoint_files(
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\transformers\modeling_utils.py", line 1200, in _get_resolved_checkpoint_files
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\transformers\modeling_utils.py", line 1200, in _get_resolved_checkpoint_files
  checkpoint_files, sharded_metadata = get_checkpoint_shard_files(checkpoint_files, sharded_metadata = get_checkpoint_shard_files(
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\transformers\utils\hub.py", line 1084, in get_checkpoint_shard_files
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\transformers\utils\hub.py", line 1084, in get_checkpoint_shard_files
  cached_filenames = cached_files(cached_filenames = cached_files(
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\transformers\utils\hub.py", line 567, in cached_files
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\transformers\utils\hub.py", line 567, in cached_files
  raise eraise e
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\transformers\utils\hub.py", line 494, in cached_files
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\transformers\utils\hub.py", line 494, in cached_files
  snapshot_download(snapshot_download(
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\utils_validators.py", line 114, in _inner_fn
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\utils_validators.py", line 114, in _inner_fn
  return fn(*args, **kwargs)return fn(*args, **kwargs)
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub_snapshot_download.py", line 330, in snapshot_download
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub_snapshot_download.py", line 330, in snapshot_download
  _inner_hf_hub_download(file)_inner_hf_hub_download(file)
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub_snapshot_download.py", line 306, in _inner_hf_hub_download
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub_snapshot_download.py", line 306, in _inner_hf_hub_download
  return hf_hub_download(return hf_hub_download(
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\utils_validators.py", line 114, in _inner_fn
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\utils_validators.py", line 114, in _inner_fn
  return fn(*args, **kwargs)return fn(*args, **kwargs)
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\file_download.py", line 1014, in hf_hub_download
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\file_download.py", line 1014, in hf_hub_download
  return _hf_hub_download_to_cache_dir(return _hf_hub_download_to_cache_dir(
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\file_download.py", line 1175, in _hf_hub_download_to_cache_dir
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\file_download.py", line 1175, in _hf_hub_download_to_cache_dir
  _download_to_tmp_and_move(_download_to_tmp_and_move(
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\file_download.py", line 1742, in _download_to_tmp_and_move
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\file_download.py", line 1742, in _download_to_tmp_and_move
  http_get(http_get(
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\file_download.py", line 485, in http_get
  File "F:\AI-Toolkit-Easy-Install\AI-Toolkit\venv\lib\site-packages\huggingface_hub\file_download.py", line 485, in http_get
  raise RuntimeError(raise RuntimeError(
  RuntimeErrorRuntimeError: : An error occurred while downloading using hf_transfer. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.An error occurred while downloading using hf_transfer. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.

I use an offline PC to train. it trains perfectly fine with Z-image Turbo and Base with the Huggingface repo that i clones using Git clone

thinking it will work the same, I git cloned the flux.2 base 9b repo. it didnt work, said "no such file" even though i used the "path to repo"

so i did it the hard way, i connected the internet.
I knew I needed a read token, so i set that up.
the first run, it downloaded 20% of only model 00001 of 00005.safetensors for the qwen 3 cache.
then it stopped downloading and failed. it Always does this for all hf_hub things, that is why i prefer git cloning and using it offline.

anyway, i thought since the cache had a file for Qwen 3, I could git clone the repo into that folder. no blob ref or snapshots folders, just straight clean git clone.

it did not work. so i connected again. this time, it created the blob ref and snapshots folders and just failed without downloading anything.

how do i get the correct models I need?
the only way that seems to make sense is to download it file by file and save it in its proper folder, but toolkit wants snapshots and blobs with md5 hashes

and I Can do it. i can save and symlink everything, but theres absolutely no reason to download everything AGAIN after the 3rd time.

[shawnlau23-coder]

If you have the required files in your .cache folder or where Huggingface stores them, make an offline start batch file. Add these lines to it:

set HF_HUB_OFFLINE=1
set TRANSFORMERS_OFFLINE=1

[poopinyap]

If you have the required files in your .cache folder or where Huggingface stores them, make an offline start batch file. Add these lines to it:

set HF_HUB_OFFLINE=1 set TRANSFORMERS_OFFLINE=1

I used an online PC to clone the cache and copied it over, so i have the files.
next is to just try your bat suggestion
i'll Try THANK YOU🙏​🙏​

[q5sys]

I just ran into this as well. AI-toolkit was working fine... partially through several trainings... I pulled the latest changes to get the fix for the jobs clobbering each other with a locked database... and now I cant start any job.
Even with setting those env variables mentioned above... it keeps trying to download from huggingface even though I've pointed it to my local directory with the files.

None of my flux2 and LTX2 jobs will work now even though they worked perfectly fine 15 minutes ago.

ideally we should be able to define where ai-toolkit should look for files. If that field isn't populated... then reach out to huggingface, but if it is, then use the local model.
Would just take an extra field based on which model we're using, and a few if statements for where to look.

[SRStwo]

No program should ever force people to have an Internet connection running unless it truly requires Internet to function (when it's truly unavoidable for the basic functionality of the program). It is so much more secure to not have an Internet connection on a production system.

[q5sys]

in trying to fix this I ended up down a weird rabbit hole... found what might be related in the transformers code.

Traceback (most recent call last):
  File "/zstor/ai/training/ai-toolkit/run.py", line 135, in <module>
  File "/zstor/ai/training/ai-toolkit/run.py", line 135, in <module>
        main()main()
  File "/zstor/ai/training/ai-toolkit/run.py", line 123, in main
  File "/zstor/ai/training/ai-toolkit/run.py", line 123, in main
        raise eraise e
  File "/zstor/ai/training/ai-toolkit/run.py", line 111, in main
  File "/zstor/ai/training/ai-toolkit/run.py", line 111, in main
        job.run()job.run()
  File "/zstor/ai/training/ai-toolkit/jobs/ExtensionJob.py", line 22, in run
  File "/zstor/ai/training/ai-toolkit/jobs/ExtensionJob.py", line 22, in run
        process.run()process.run()
  File "/zstor/ai/training/ai-toolkit/jobs/process/BaseSDTrainProcess.py", line 1616, in run
  File "/zstor/ai/training/ai-toolkit/jobs/process/BaseSDTrainProcess.py", line 1616, in run
        self.sd.load_model()self.sd.load_model()
  File "/zstor/ai/training/ai-toolkit/extensions_built_in/diffusion_models/ltx2/ltx2.py", line 411, in load_model
  File "/zstor/ai/training/ai-toolkit/extensions_built_in/diffusion_models/ltx2/ltx2.py", line 411, in load_model
        tokenizer = GemmaTokenizerFast.from_pretrained(self.ltx_te_path)tokenizer = GemmaTokenizerFast.from_pretrained(self.ltx_te_path)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/zstor/conda/envs/zstor-training/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained
  File "/zstor/conda/envs/zstor-training/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2113, in from_pretrained
        return cls._from_pretrained(return cls._from_pretrained(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/zstor/conda/envs/zstor-training/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2395, in _from_pretrained
  File "/zstor/conda/envs/zstor-training/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2395, in _from_pretrained
        tokenizer = cls._patch_mistral_regex(tokenizer = cls._patch_mistral_regex(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/zstor/conda/envs/zstor-training/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2438, in _patch_mistral_regex
  File "/zstor/conda/envs/zstor-training/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2438, in _patch_mistral_regex
        if _is_local or is_base_mistral(pretrained_model_name_or_path):if _is_local or is_base_mistral(pretrained_model_name_or_path):
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/zstor/conda/envs/zstor-training/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2432, in is_base_mistral
  File "/zstor/conda/envs/zstor-training/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2432, in is_base_mistral
        model = model_info(model_id)model = model_info(model_id)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In digging into it, the _patch_mistral_regex function calls model_info() which makes an unconditional API call that doesn't respect the offline cache. It only skips the call if the path is recognized as a local directory, and that doesn't always seem to happen. Not sure why.

The _patch_mistral_regex function runs for every tokenizer loaded via from_pretrained, regardless of model type. It calls model_info() on the HuggingFace API to check if it's a Mistral model, before deciding whether to apply a the Mistral-specific regex patch. So even when loading a Gemma tokenizer like with LTX2, it tries to phone home to HuggingFace just to confirm it's not Mistral, then moves on.

I was able to patch around it with transformers/tokenization_utils_base.py around line 2432 and change:
python model = model_info(model_id)
to:

python            try:
                model = model_info(model_id)
            except Exception:
                return False

Wrapping the call in try/except and returning False, wont hurt anything since I'm in an local env that's only set up for AI-Toolkit and wont be using Mistral for anything.
All that said, there hopefully is better way to handle this entirely within the AI-Toolkit code than having to locally patch the transformers code. I'm just posting it here unless there's someone else that runs into this and wants to hack on their own system.