Enhancement: --jobs Option - Enable Auto-Detection with Value 0 & Breaking Change Consideration

[daniel-mohr]

I'm proposing an enhancement to the --jobs (or -j) option to increase the flexibility of the program.

Current Behavior:

Currently, the --jobs (or -j) option interprets a value of 0 as 1. This means that regardless of the number of available processors, only one process is used.

Proposed Enhancement:

I suggest that a value of 0 be interpreted as a signal to use the maximum number of available processors. This would allow the user to easily enable automatic parallelization without needing to specify a particular number of processes. This approach is common in build systems like make -j0 and version control systems like git -j0, where 0 often indicates utilizing all available cores.

Implementation:

I am working on a pull request to implement this change. The changes will include:

• Modification of the default logic for the jobs variable to interpret 0 as a signal to use all available processors.
• Updating the documentation to explain the new behavior of the --jobs option.

Breaking Change:

This change will introduce a breaking change. Users who previously used --jobs 0 or jobs = 0 in their configuration to force single-threaded execution will suddenly experience parallelization after this pull request is accepted.

However, I would expect that very few users have configured --jobs 0. The default value via argparse should not result in parallelization and should therefore be adjusted. This keeps the breaking changes within limits.

Benefits:

• Flexibility: Users can easily enable automatic parallelization without manually configuring the number of processes.
• Optimization: The program can optimally utilize available resources.
• User Friendliness: Simplifies configuration for users who want automatic parallelization.

I am open to feedback and willing to adjust the changes as needed.

Pull Request: I am currently working on a pull request and will provide a link as soon as it's available.

[facelessuser]

I'm not against having 0 mean "use as many cores as available". I think when I implemented this, I didn't bother to look close enough at the API of ProcessPoolExecutor to realize that None would utilize as many cores as it could. Using zero for this case seems like a reasonable approach.

I'm not very concerned about changing the behavior of zero as what I was really doing is simply clamping to the value 1. Zero was not meant to be a "real" value when the feature was implemented, just allowing a sane response if when any positive integer was used.

I see that you've already thrown up a PR. I have a few issues with the approach, mainly how None travels through the code path. It's possible I may just commit a solution directly that address my concerns, but if not, I'll make some comments on the PR to provide further guidance.

Thanks for opening this issue, I think this will be a good enhancement.

[daniel-mohr]

So, I followed the given path from the command line arguments via argparse to the only place it is used.

I changed the default value for argparse to None to detect the user did not provide input -- if the used provides this parameter it would be an integer.

For every function which is called with this jobs parameter I changed the default value to 1 -- you initial idea.

At the final point I use the default 1 if the user does not provide anything. But if the user provides a integer, this is used -- only the special case 0 is translated for ProcessPoolExecutor to None.

Seems clear to me.

But let me know about required changes or if I have to adapt some settings to allow to push to this pull request.

[facelessuser]

Yes, the path was followed, but I think it should have been defaulted as None throughout. Additionally, we need to keep handling for negative integer input. It was just easier for me to throw up a PR as the change is pretty simple.

[daniel-mohr]

It's your project.

I didn't want to change the default behavior of all you functions.

Negative integers as jobs was already handled. I did not change much. As an enhancement the input could already be checked by argparse by a user defined function as type. This would through an error for negative input. But I'm not sure if the effort is not too much.

Perhaps the docs/src/markdown/index.md file should also be adapted in the two places where jobs is explained.

Further you did not describe the 0 in the argparse help of jobs.

I'm waiting for the next release.

[facelessuser]

I'm not criticizing your work, that wasn't my intention. I was just noting there were some minor things, and I'd either comment of the PR or just push was I was thinking.

Further you did not describe the 0 in the argparse help of jobs.

I did not, but I can.

[daniel-mohr]

Thanks for clarifying. I appreciate you taking the time to explain your approach. No worries at all – it's your project, and you're right to implement it the way you see fit. I'm happy to see the enhancement implemented.