Poetry support

[caniko]

Description

The requirements.txt + pip is, IMHO, outdated. Instead, Poetry should be used, for proper package management.

Implementation

No poetry project

Automated poetry init when running cookiecutter https://github.com/cookiecutter/cookiecutter-django.

Existing poetry project

Base:

cat base.txt | grep -E '^[^# ]' | cut -d= -f1 | xargs poetry add

Local:

cat local.txt | grep -E '^[^# ]' | cut -d= -f1 | xargs poetry add -D

Production:

cat production.txt | grep -E '^[^# ]' | cut -d= -f1 | xargs -n1 poetry add -E production

Rationale

Why should this feature be implemented? Random article that I found.

[browniebroke]

This is quite similar to #1988 except this is restricting the problem (lock your dependencies) to a specific solution (Poetry). I really like Poetry and use it in my personal project, so that would be really nice to have, but since there are at least 2 other strong alternatives (pip-tools/pipenv), I suggest to keep the discussion there.

One thing that would really help in adopting Poetry is the dependency group feature coming up in version 1.2 (currently in alpha).

A challenge in adopting more advanced package mangers like this is keeping the cookiecutter template up to date. Currently, we use PyUp but we would need to confirm whether it works with Poetry and if we can use templated pyproject.toml.

If it's not supported, we could run the commands you suggest in post-generation hook to generate the pyproject.toml from the existing requirements files... A similar approach could be used for other package managers.

[Andrew-Chen-Wang]

Has anyone tried using poetry with large projects? Or just the base cookiecutter-django template? As much as a lock file is useful, poetry's super slow, especially larger projects. I wouldn't be opposed to offering multiple package managers (ie an option to use pip, poetry, etc).

[caniko]

Has anyone tried using poetry with large projects

I don't see how this is relevant, as the packages required by cookiecutter aren't that many. Making it not so large, imho. If it was relevant, I'd say poetry is more than capable, as I have had projects with 70+ packages run just fine with poetry.

poetry's super slow

What do you need speed for in a development scenario? Not that it matters, but poetry is not slow.

package managers (ie an option to use pip, poetry, etc).

Poetry uses pip, but yes, there are other package managers that try to compete with poetry. I don't use them, but they should be supported. I don't see how this is relevant, as adding support should be simple.

Furthermore, I have already moved all my dependency management in my cookiecutter project to Poetry. Even if support is never added, it is incredibly easy to port a project with requirement.txt to poetry; annoying, but easy.

[Andrew-Chen-Wang]

Has anyone tried using poetry with large projects

I don't see how this is relevant, as the packages required by cookiecutter aren't that many. Making it not so large, imho. If it was relevant, I'd say poetry is more than capable, as I have had projects with 70+ packages run just fine with poetry.

It's a precursor to the speed issue. Also, regardless of the number of packages, when working in a large team that constantly changes the toml file and we're all using docker, caching becomes an issue as well. But I digress; poetry should be an option anyway.

poetry's super slow

What do you need speed for in a development scenario? Not that it matters, but poetry is not slow.

So I don't have to wait several minutes to add a package? Apologies, but not everyone has better-than-McDonalds wifi (and personally when I want to get a new feature in but it requires a new package, I don't want to shift focus temporarily to a bug for 5 minutes before shifting my focus back to working on the new feature). Poetry's dependency resolver is the slowest portion, and there are several GitHub issues in the poetry repo of arguments on how to fix it. I enjoy poetry update as it removes the need for something like dependabot, but having to wait 3-5 minutes to add a package, 7 minutes to upgrade a crucial package that's core to several others like PyJWT, and 10 minutes to update (on average across multiple repos) is a deal breaker for me at the moment.

package managers (ie an option to use pip, poetry, etc).

Poetry uses pip, but yes, there are other package managers that try to compete with poetry. I don't use them, but they should be supported. I don't see how this is relevant, as adding support should be simple.

Furthermore, I have already moved all my dependency management in my cookiecutter project to Poetry. Even if support is never added, it is incredibly easy to port a project with requirement.txt to poetry; annoying, but easy.

Yea, it's annoying, and that's why I'm not objecting to adding an option since poetry is like the python3 in the python 2->3 transition. And it's relevant since I'm assuming an issue like this would sprout a PR that more than likely would ignore the other package managers. cookiecutter-django is kinda opinionated and only mainly supported by one maintainer, but more options is nice and usually forgotten.

[caniko]

This seems to be a very passionate subject for you, I really didn't mean to hit a nerve. I wanted to keep the discussion limited to ways to implement support, and not discuss Poetry's limitations in terms of speed. Those that use Poetry are already familiar with its speed.

Opinion:

• The toml file argument sounds like a team of devs working without a centralized plan. Nonetheless, poetry is a bad fit for this kind of workflow, as the toml file shouldn't change at the rate you are describing. Most of the packages should be defined during poetry init.
• Having a slow Wi-Fi slows down everything, not just Poetry. This limitation should be solved not just for efficiency and speed, but overall happiness. Slow internet during development is dissatisfying, and can lead to burn-out.
• You can still write code while Poetry is updating. Being able to work while something is installing/compiling/computing is a skill that should be developed. My biggest culprit these days is docker and Kubernetes, it takes a long time for these systems to get ready for use, I can't expect my boss to be OK with me waiting 15 minutes without doing anything during this time.

Hope we can move on. Thank you

[luzfcb]

I don't use poetry. I've tried it a few times, but I haven't studied it deeply to the point that my employer and I have adopted it, so I don't have an opinion on how good or not the use of poetry is in the real world. I don't know where the corn hurts.

I have two concerns:

• Will adding poetry increase project complexity and add more workload to maintainers?

• How will the automatic dependency update look like? Dependabot for python dependencies is not an option. dependabot is not able to correctly parse our requirements files as they contain Jinja2 syntax. Instead of dependabot, we use PyUp. PyUp is able to correctly parse files with Jinja2 syntax because years ago one of the creators of PyUp was also a maintainer of this project.
  PyUp doesn't support pyproject.toml yet.

Question: Would a step-by-step tutorial teaching how to migrate to poetry a project generated via cookiecutter-django help alleviate the problem?

[caniko]

These concerns are legitimate indeed. I will try my best to alleviate the two mentioned here.

1. The questionnaire during the initialization of a cookiecutter django project should stay mostly the same. Moreover, the arguments from the existing CLI can be passed directly into poetry init, docs here. I would say it puts close to no extra load on maintenance, since the poetry init integration is ad-hoc to the rest of the code.
2. Auto update is very easy with poetry: poetry update. You can revert using your VCS followed by poetry install. You can even do poetry update --dry-run to see what would be updated. Not sure how this would be implemented on the cookiecutter stack, considering PyUp has no pyproject.toml support, maybe wait for this to be solved. Seems like Pydantic had the same issue back in 2018.

To answer the final question, I think it will be a rubber band, the auto update seems to be problematic. A method to go from requirement.txt files to pyproject.toml; I would refer you to the first post in the issue.

[Andrew-Chen-Wang]

hit a nerve

Sorry, I just don't write very professionally or respectfully so excuse me 😅 Will work to improve plus thanks for suggestions!

I'm just concerned we're fully replacing the requirements files with poetry. I'd rather make it an option than fully replace if that makes what I previously said clearer.

auto update

We can work around this. Anytime an update occurs with PyUp, we can push a commit to the pyup branch that changes the TOML file's Python requirements. It should be trivial as we just read line by line in the TOML file (hopefully there are no differences in package naming syntax that I'm unaware of). Especially since it's our template, we'll know beforehand what "sections" are included in the TOML file so there's no chance to screw up over arbitrary section names.

Edit: actually you're right, poetry update package is prob better. We just need to know the package name in its exact format.

[browniebroke]

I think we should not underestimate that the template has a number of combinations with which one may use poetry, pip (or something else): non-docker vs docker, Heroku vs other. We need to integrate each package manager with the various places, which means to have the appropriate branches everywhere is needed:

{% if cookiecutter.package_manager == "poetry" %}
poetry install
{% else %}
pip install -r requirements/local.txt
{% endif %}

Then, when the template needs to be updated (package/python version, Docker version, Poetry version....) or added (new package), maintainers will sometimes have to test each separate package manager to deploy to prod, this takes time, which I wouldn't call "close to no extra load on maintenance".

To make maintenance easier, we'll definitely to add an extra job in the CI workflow dedicated to that (what happens if dependencies can no longer be resolved after an update?).

Auto update is very easy with poetry: poetry update

Again, this might work great with the generated project, but it doubt that it'll work in the context of the template (Jinja tags all over the place might make the toml invalid, I think).

[Andrew-Chen-Wang]

I think for the template, it might be easier if we run a post-gen hook that runs poetry install rather than save a huge lock file for the template. All that's left is the TOML file itself. Again, we can do some parsing to identify the package that we want to change in the TOML file based on the package that changed in the requirements file.

[Andrew-Chen-Wang]

I'd like to make a quick test script. If anyone can make a quick pyproject.toml and a lock file where there are multiple groups/envs (ie local and prod), that'd be great. Thanks! It doesn't need to have everything, but stuff like psycopg2 in all it's forms for prod and local would be helpful.

[Gkaster]

I'd like to make a quick test script. If anyone can make a quick pyproject.toml and a lock file where there are multiple groups/envs (ie local and prod), that'd be great. Thanks! It doesn't need to have everything, but stuff like psycopg2 in all it's forms for prod and local would be helpful.

as example cool cookiecutter django project with poetry (with docker/+compose)
pyproject.toml

[starascendin]

can someone pls make this poetry version. Its such a pain in the ass to migrate this to poetry

[Andrew-Chen-Wang]

Thanks @Gkaster I simply haven't worked with poetry that much. In my opinion, a simple solution would be the post-gen hook idea where we first generate the necessary requirements, paste them into a pyproject.toml file, and generate a lock file if use_poetry=True.

is there anything I'm missing here? I don't think it's a good idea to be making jinja template comments in the lock file itself due to huge numbers of dependencies.

@starascendin any explanation for why this is a pain? Could you make a PR or provide the usual steps it takes?