Is there a way to use a pre-trained controlnet?

[alexanderswerdlow]

Instead of loading from the base U-Net, can we perform LoRA on an already trained controlnet?

[HighCWu]

I think the diffusers library itself already contains this support. You only need to use diffusers to load the controlnet model, and then train the lora of controlnet by referring to the method of training unet's lora in diffusers.

[venkateshtata]

@HighCWu if I am right the implementation in diffusers library is plugging the LoRA to UNet of the StableDiffusionControlNetPipeline and not the ControlNet's attention blocks. Is there any way we can use the pre-trained ControlNet that understands say "Canny" or "HED" conditions and plug the LoRA on top of one such pre-trained model ?

[HighCWu]

@HighCWu if I am right the implementation in diffusers library is plugging the LoRA to UNet of the StableDiffusionControlNetPipeline and not the ControlNet's attention blocks. Is there any way we can use the pre-trained ControlNet that understands say "Canny" or "HED" conditions and plug the LoRA on top of one such pre-trained model ?

Now the official t2i lora sample is unet.add_adapter. I think, for controlnet, it should be done with controlnet.add_adapter

[venkateshtata]

It's throwing AttributeError: 'ControlNetModel' object has no attribute 'add_adapter' error.

[HighCWu]

It's throwing AttributeError: 'ControlNetModel' object has no attribute 'add_adapter' error.

@venkateshtata check my new added file models/controlnet.py, import it. it may work. I wrote some sample code in it, you can check the source code