是否支持多卡并行推理

[Whjie-LLM]

我现在有多张4090显卡，能否使用多卡并行推理

[mp3pintyo]

The 4090 currently has a low amount of memory. If you have multiple cards, the memory amounts will not add up.

[KimbingNg]

Hi @beizhai35 , we have released fp8-quantized models, which support generating images with < 24 GPU memory (e.g. a single 4090 GPU). Give it a try :)