New optimizer:AdEMAMix8bit and PagedAdEMAMix8bit

[sdbds]

Tested to work better than AdamW.

[kohya-ss]

Thank you! But I think, by updating the dependent library, we can use optimizer_type arg like --optimizer_type bnb.optim.AdEMAMix8bit instead of adding new optimizer types. I don't like to increase the number of optimizer types infinitely😅

[sdbds]

Thank you! But I think, by updating the dependent library, we can use optimizer_type arg like --optimizer_type bnb.optim.AdEMAMix8bit instead of adding new optimizer types. I don't like to increase the number of optimizer types infinitely😅

It's true that you can do this for non-invasive optimizers, but it's a bit of a gateway for non-developers, and the 8bit class of optimizers is sort of more commonly used.
I've actually tested about 76 optimizers using pytorch_optimizer and have no plans to add them in.

[sdbds]

The extra code section is now removed and only the version of the dependency libraries and the comments section are modified.Additional release notes are needed regarding the use of --optimizer_type bnb.optim.AdEMAMix8bit

[FurkanGozukara]

@sdbds which optimizer you find best at the moment for sd 1.5 sdxl and flux?

Individually for each

[rockerBOO]

The extra code section is now removed and only the version of the dependency libraries and the comments section are modified.Additional release notes are needed regarding the use of --optimizer_type bnb.optim.AdEMAMix8bit

Maybe it can be fully qualified path in the help text. Regular users would then easily know what to enter.

AdEMAMix8bit -> bnb.optim.AdEMAMix8bit

[gesen2egee]

Is it possible to combine with schedule free?

[sdbds]

Is it possible to combine with schedule free?

It has been submitted to schedulfree officials.

[sdbds]

@sdbds which optimizer you find best at the moment for sd 1.5 sdxl and flux?

Individually for each

I can tell you if you keep my credits in your videos or whatever:
SD1.5 sdxl prodigy
flux adamWschedulefree and this for now

[kohya-ss]

Thanks you, I merged this PR.

I forgot that optimizer_type needs to be specified with a full path. So it works with --optimizer_type bitsandbytes.optim.AdEMAMix8bit. I also updated the help text.

It has also been merged into the sd3 branch.

[FurkanGozukara]

What is difference between PagedAdEMAMix8bit and bitsandbytes.optim.AdEMAMix8bit

also any Optimizer extra arguments required? thank you

anyway i started testing all i could 🗡️