[Web] Can't create session loading large models

[edhyah]

Describe the issue

Hello,

I'm trying to load a UNet model (> 2GB), which means I have a model.onnx file as well as a lot of .weight and .bias files.

Using onnxruntime-web, I'm attempting to load it into a Node/Javascript environment using the following code:

const ort = require('onnxruntime-web');
try {
    const session = await ort.InferenceSession.create('./unet/model.onnx')
} catch (e) {
    console.log(e);
}

When I do this, I'm faced with this warning:

Deserialize tensor onnx::MatMul_12479 failed.tensorprotoutils.cc:640 TensorProtoToTensor External initializer: onnx::MatMul_12479 offset: 0 size to read: 409600 given file_length: 32 are out of bounds or can not be read in full.

And then this error:

Error: Can't create a session.

Using this same method and code I've been able to load smaller models < 2GB without errors and warnings.

Is there anything I can do to investigate why I'm running into this issue?

To reproduce

UNet model can be accessed here: https://drive.google.com/drive/folders/1l65XlDrYYE7m1EUPdEZ6SDGWjrlIljJu?usp=sharing

You can use this code (copied and pasted from above):

const ort = require('onnxruntime-web');
try {
    const session = await ort.InferenceSession.create('./unet/model.onnx')
} catch (e) {
    console.log(e);
}

Urgency

It's somewhat urgent.

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.8.0 (installed onnxruntime-web through npm)

Execution Provider

Other / Unknown

[shalvamist]

Hi,

Thanks for the detailed issue report.
I tried to download the model you are referencing but the link leads to a folder with a lot of files - can you please share a link to the specific model file you are using to produce the issue ? this will help us to make sure I am following and using the exact setup you have on your end.

Thanks again

[edhyah]

Hello,

Thanks for the reply!

In that folder, the main model file is model.onxx. Since it's a large model (> 2GB) though, I had to use external data, which is why you see all of the other .weight and .bias files. I think the problem is I can't seem to load onnx models with these external data files.

Let me know what you think.

[edhyah]

@shalvamist Any thoughts? Sorry, just a bit urgent on my end. Thanks!

[shalvamist]

Hey,

I understand the urgency and limitations, no worries :).
Is there a way you can share a description of how to download and reconstruct the model on my end ? that will be the fastest route to get it on my end

[edhyah]

Yes of course, thanks! I'm using Python 3.7; works best in a virtualenv.

Install the following packages.

pip install torch diffusers transformers

Then, clone the github repo.

git clone git@github.com:huggingface/diffusers.git
cd diffusers

Run the torch to onnx script to convert the large model.

python scripts/convert_stable_diffusion_checkpoint_to_onnx.py --model_path="CompVis/stable-diffusion-v1-4" --output_path="./stable_diffusion_onnx"

The model I'm having trouble with is located in ./stable_diffusion_onnx/unet. All of the files in that unet folder can also be found in the link I shared above: https://drive.google.com/drive/folders/1l65XlDrYYE7m1EUPdEZ6SDGWjrlIljJu?usp=sharing

[shalvamist]

Hey @edhyah

Thanks for sharing the details - I was able to reproduce the issue on my end.
The error was exactly the same -

Deserialize tensor onnx::MatMul_12479 failed.tensorprotoutils.cc:640 TensorProtoToTensor External initializer: onnx::MatMul_12479 offset: 0 size to read: 409600 given file_length: 32 are out of bounds or can not be read in full.

Let me try to dig a little deeper into this issue and I will try to get back to you in the following days.

[edhyah]

Awesome thank you! Really appreciate it :)

[shalvamist]

Hi,

So we looked at the error and we think the cause is due to the size of the model it hits the 4GB address space limitation of WASM. Unfortunately at this time there are several limitation on getting this model running on ORT Web. The main challenge is at this stage is WASM 32bit addressing space (4GB limit).
There are several threads on the topic working on WASM 64bit - here are a few -

• https://github.com/WebAssembly/memory64
• Implement the memory64 proposal in Wasmtime bytecodealliance/wasmtime#3153

I also checked if we have a downsized version of Unet to try but as of now there isn't one that meets the accuracy requirements. So until WASM 64bit will be supported officially (we'll need to enable this support in ORT Web) or we'll have a trimmed down version of the model we can't run this on WASM. There might be a possibility to use WebGL but there are other concerns there that are related to performance (20> sec per inference on a very strong GPU)

Sorry I couldn't be of more help, please let me know if you have any comments or ideas to can try to enable your progress.
We'll be happy to help.

[edhyah]

Ah gotcha. Thank you for all of the work! Really appreciate it :)