Add Aim integration to traingenerator

[SGevorg]

Thanks for the great work.
It would be great to add Aim to train generator as an experiment logger option.

Aim is the most advanced open source experiment comparison tool available at the moment.

[jrieke]

Hey @SGevorg,

let's use the existing image classification w/ PyTorch template as an example here. As far as I can see, using Aim would just mean adding

aim.set_params({"lr": lr, "batch_size": batch_size, "num_epochs": num_epochs}, name="hparams")

around line 85 and adding

aim.track(metrics["loss"], name="loss", subset=name, epoch=epoch)
aim.track(metrics["accuracy"], name="accuracy", subset=name, epoch=epoch)

around line 220 (+ adding import and pip install of course).

Questions:

• anything missing/wrong or anything else you'd like to have added?
• is the preferred name for the hyperparmeters "hparams" or "params"? (you use both in the docs)
• should the track method just use subset to differentiate between train/val/test or should the metrics also be named train_loss/val_loss/test_loss etc?

[SGevorg]

@jrieke thanks for the instructions.
Adding @gorarakelyan here too to help with the code.

re train/val/test they are provided as arguments to track - any additional argument converted into a context (key-value pairs). In this case you woul just call aim.track(metrics["loss"], name="loss", epoch=epoch, subset="train") and subsequent ones for test, val too.

[arkkln]

Hi @jrieke ,

All correct, thanks for your effort!

• I suggest using name hparams to be more specific
• as @SGevorg mentioned, the phase should be passed as an argument to track function, which I see has been perfectly done here:
  aim.track(metrics["loss"], name="loss", -> subset=name <- , epoch=epoch)
• nothing more to add as for now

Would be happy to help in case of more questions. Do you want me to open a PR?

This is an awesome project!

[jrieke]

Hi @gorarakelyan,

yes, feel free to open a PR if you have time, otherwise, I'll do it later (not that much work anyway) and will send it to you for testing. Make sure to add Aim as an option in the sidebar here. You can also update the scikit-learn template if you like.

Best, Johannes

[arkkln]

Great, thanks for the instructions, @jrieke. I will work on the PR and ping you once it is ready.

[arkkln]

Hi @jrieke ,

I've added Aim to PyTorch image classification template and opened a PR. Can you have a look at it? Thanks a lot!

[jrieke]

@gorarakelyan Sorry for the slight delay. I only changed 2 small things on your fork (hope that was OK; I added aim to the pip install line at the top of the generated code + moved the aim import a bit further down) and merged. It's now live on the website! 🥳

Thanks again for the contribution ❤️ Feel free to also add it to the scikit-learn template if you like.

---

Btw I noticed 2 minor issues with aim when running on my machine:

1. This warning

   /Users/jrieke/opt/anaconda3/lib/python3.7/site-packages/aimrecords/artifact_storage/storage.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working. 
   from collections import Iterator`
   

2. When I delete the .aim folder before closing aim_session or finishing the script (which is what happens when I run pytest because I create a temporary dir for each test run, which gets deleted before the tests finish), I get this error:

   Something went wrong in _close. The process will continue to execute.
   _close: [Errno 2] No such file or directory: '/private/var/folders/1j/szl2n8g55053cfpnpdr_bpfc0000gn/T/tmp65jegpiw/.aim/default/8a8bbb00-5a00-11eb-8eac-8c8590b82ad8/objects/.aimrecords_storage/loss/current_bucket.bin'`
   

Nothing major but thought I'll mention it in case you want to look into it :)

[arkkln]

Awesome! 🎉

Thanks a lot for the review and for reporting issues.