Android support

[Young-Flash]

Are there any plans to support Android devices in the future?

[mazzzystar]

Hi I'm not good at Android dev, and my energy is also devoted to other projects, so I'd glad if someone could help : )

[sunlin-xiaonai]

i will have a try, now i am do somethings about model

[greyovo]

I split the model into ImageEncoder and TextEncoder, like the author did, and successfully ran the demo on Android using PyTorch Mobile.

Running result under a Xiaomi 12S @ Snapdragon 8+ Gen 1 SoC:

• Inference speed: around 10 seconds for 100 images and 250 seconds for 2000 images (much slower than that on iOS). All photos are 400*400px JPEGs.

• The exported ImageEncoder and TextEncoder are also very large (335 MB and 242 MB, respectively).

The efforts I have tried before:

1. Quantifying the model, but it failed because I am not familiar with Pytorch's quantization operations
2. Converting to NNAPI mode for operation, but no luck.

I observed the device status in profile mode and found that the CPU usage is only about 45%. I am not sure if NPU was used during inference.

I'm still evaluating whether it's worth to keep going, or anyone could give any advice would be appreciated 😄

Refs:

• (beta) Efficient mobile interpreter in Android and iOS — PyTorch Tutorials 2.0.1+cu117 documentation
• Pytorch Mobile Performance Recipes — PyTorch Tutorials 2.0.1+cu117 documentation
• https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html

[x97425]

I split the model into ImageEncoder and TextEncoder, like the author did, and successfully ran the demo on Android using PyTorch Mobile.

Running result under a Xiaomi 12S @ Snapdragon 8 Gen 2 SoC:

• Inference speed: around 10 seconds for 100 images and 250 seconds for 2000 images (much slower than that on iOS). All photos are 400*400px JPEGs.
• The exported ImageEncoder and TextEncoder are also very large (335 MB and 242 MB, respectively).

The efforts I have tried before:

1. Quantifying the model, but it failed because I am not familiar with Pytorch's quantization operations
2. Converting to NNAPI mode for operation, but no luck.

I observed the device status in profile mode and found that the CPU usage is only about 45%. I am not sure if NPU was used during inference.

I'm still evaluating whether it's worth to keep going, or anyone could give any advice would be appreciated 😄

Refs:

• (beta) Efficient mobile interpreter in Android and iOS — PyTorch Tutorials 2.0.1+cu117 documentation
• Pytorch Mobile Performance Recipes — PyTorch Tutorials 2.0.1+cu117 documentation
• https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html

i've send you an email,pls check out if you have time. respect!

[williamlee1982]

I split the model into ImageEncoder and TextEncoder, like the author did, and successfully ran the demo on Android using PyTorch Mobile.

Running result under a Xiaomi 12S @ Snapdragon 8+ Gen 1 SoC:

• Inference speed: around 10 seconds for 100 images and 250 seconds for 2000 images (much slower than that on iOS). All photos are 400*400px JPEGs.
• The exported ImageEncoder and TextEncoder are also very large (335 MB and 242 MB, respectively).

The efforts I have tried before:

1. Quantifying the model, but it failed because I am not familiar with Pytorch's quantization operations
2. Converting to NNAPI mode for operation, but no luck.

I observed the device status in profile mode and found that the CPU usage is only about 45%. I am not sure if NPU was used during inference.

I'm still evaluating whether it's worth to keep going, or anyone could give any advice would be appreciated 😄

Refs:

• (beta) Efficient mobile interpreter in Android and iOS — PyTorch Tutorials 2.0.1+cu117 documentation
• Pytorch Mobile Performance Recipes — PyTorch Tutorials 2.0.1+cu117 documentation
• https://pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html

look forward to your work, really want this on Android

[Young-Flash]

I split the model into ImageEncoder and TextEncoder, like the author did, and successfully ran the demo on Android using PyTorch Mobile.

Running result under a Xiaomi 12S @ Snapdragon 8+ Gen 1 SoC:

• Inference speed: around 10 seconds for 100 images and 250 seconds for 2000 images (much slower than that on iOS). All photos are 400*400px JPEGs.
• The exported ImageEncoder and TextEncoder are also very large (335 MB and 242 MB, respectively).

The efforts I have tried before:

1. Quantifying the model, but it failed because I am not familiar with Pytorch's quantization operations
2. Converting to NNAPI mode for operation, but no luck.

I observed the device status in profile mode and found that the CPU usage is only about 45%. I am not sure if NPU was used during inference.

I'm still evaluating whether it's worth to keep going, or anyone could give any advice would be appreciated 😄

Refs:

• (beta) Efficient mobile interpreter in Android and iOS — PyTorch Tutorials 2.0.1+cu117 documentation
• Pytorch Mobile Performance Recipes — PyTorch Tutorials 2.0.1+cu117 documentation
• https:pytorch.org/tutorials/prototype/nnapi_mobilenetv2.html

Maybe you can briefly introduce the required technology stack and technical route, so that others can participate 😄

[mazzzystar]

Inference speed: around 10 seconds for 100 images and 250 seconds for 2000 images (much slower than that on iOS). All photos are 400*400px JPEGs.

@greyovo for the ViT-B-32 CLIP model, the required image size is 224x224, so maybe do some preprocess would make the indexing faster

[greyovo]

@greyovo for the ViT-B-32 CLIP model, the required image size is 224x224, so maybe do some preprocess would make the indexing faster

@mazzzystar Thanks for the advice. I actually did preprocess (i.e. resizing to 224px, center croping, normalizing..., like CLIP's preprocess() function did) before encoding images. Since I tried with 3000px*2000px images and got the same result, I dont think that's the main problem :(

[greyovo]

Maybe you can briefly introduce the required technology stack and technical route, so that others can participate 😄

@Young-Flash In fact, most of the work relies on the Collab script provided by the author @mazzzystar, and I just exported the two encoders to Pytorch-JIT models respectively instead of CoreML.

Possible improvements in my opinion:

1. Converting the model for NNAPI execution
2. Pruning or quantifying the model, which requires a deep understanding of the model structure and may require retraining
3. Distilling knowledge from the model, which requires familiarity with deep learning techniques and also need retraining
4. Looking for other multimodal models similar to CLIP, but I searched around and couldn't find anything more efficient and smaller than Vit-B/32 :(

Perhaps the easiest way is to convert the model to be suitable for NNAPI in order to speed up executing encoders. I tried by following pytorch's official tutorial but failed. It seems to require an ARM64 processor PC. I'm not sure if I missed something.

[mazzzystar]

Pruning or quantifying the model, which requires a deep understanding of the model structure and may require retraining

I think you may be able to do quantization when exporting pytorch model. pytorch/pytorch#76726

[greyovo]

I think you may be able to do quantization when exporting pytorch model. pytorch/pytorch#76726

@mazzzystar Yes I also tried quantization but encountered several problems that I cannot solve, and hence the quantization was failed, not to mention the NNAPI convertion (which needs a quantized model). I may later share a jupyter notebook I used to see if anyone would help.

[greyovo]

An interesting thing is that I found some efforts on distilling the CLIP model:

• ImageSearchLightningCLIP

  • It's an Android app using CLIP model. @x97425 @Young-Flash @williamlee1982 in case you may need it.
  • It used a model distilling from CLIP. The speed can be doubled compared to the original Vit-B/32 (that is, 100 pictures for ~5 seconds), but unfortunately, this will sacrifice accuracy (even unable to pass the example zero-shot prediction in CIFAR100).

• clip-models-for-distillation

  • They outlined the general idea in the their blog but no specific code for distillation was provided.

At least they proved that knowledge distillation may be a feasible direction, but requires a notable effort to do so.

[mazzzystar]

@greyovo I'm not sure if distillation is a good idea compared to quantinazation. Even you don't quantize the model, using FP16 would significantly increase model speed.

I might try to export a lightweight Android version if I have time in the future : ) Is ONNX okay?

[greyovo]

I might try to export a lightweight Android version if I have time in the future : ) Is ONNX okay?

@mazzzystar You are right! There's another option - ONNX. It seems they have complete docs and demos. So yes, it's worth a try! Thanks :)