Adopt multi-phase init (PEP 489)

[AA-Turner]

cc @giampaolo

xref #2576

The mechanical changes for each module are:

1. Change PyMODINIT_FUNC PyInit_{name}(void) to static int {name}_exec(PyObject *mod)
2. Change return NULL to -1 and return mod to 0
3. Remove PyModule_Create and PyUnstable_Module_SetGIL
4. Add a PyModuleDef_Slot slots array containing the exec function & slots to declare subinterpreter and no-GIL support
5. Move the PyModuleDef struct after the slots array & add the new .m_slots member
6. Restore the PyMODINIT_FUNC PyInit_{name}(void) function, now just with the PyModuleDef_Init() call

A

[giampaolo]

There's errors like:

DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=701 -DPSUTIL_FREEBSD=1 -I/usr/local/include/python3.11 -c psutil/_psutil_bsd.c -o /tmp/tmpt5wq7zks.build-temp/psutil/_psutil_bsd.o
2025-05-28T13:38:48.0092731Z       psutil/_psutil_bsd.c:226:30: error: use of undeclared identifier 'moduledef'; did you mean 'module_def'?
2025-05-28T13:38:48.0093197Z         226 |     return PyModuleDef_Init(&moduledef);
2025-05-28T13:38:48.0093482Z             |                              ^~~~~~~~~
2025-05-28T13:38:48.0093727Z             |                              module_def
2025-05-28T13:38:48.0094421Z       psutil/_psutil_bsd.c:216:27: note: 'module_def' declared here
2025-05-28T13:38:48.0094769Z         216 | static struct PyModuleDef module_def = {
2025-05-28T13:38:48.0095021Z             |                           ^
2025-05-28T13:38:48.0095249Z       1 error generated.
2025-05-28T13:38:48.0095492Z       error: command '/usr/bin/cc' failed with exit code 1
2025-05-28T13:38:48.0095761Z       [end of output]

[AA-Turner]

@giampaolo sorry! copy and paste error. I've pushed a commit to fix, please could you approve CI?

A

[giampaolo]

2025-05-28T17:01:52.7843315Z       gcc -pthread -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=701 -DPy_LIMITED_API=0x03060000 -DPSUTIL_LINUX=1 -I/tmp/tmp.Vj6AbJSRak/venv/include -I/opt/_internal/cpython-3.12.7/include/python3.12 -c psutil/_psutil_linux.c -o /tmp/tmpb01gi0rv.build-temp/psutil/_psutil_linux.o
2025-05-28T17:01:52.7845994Z       psutil/_psutil_linux.c:71:6: error: ‘Py_mod_multiple_interpreters’ undeclared here (not in a function)
2025-05-28T17:01:52.7846936Z          71 |     {Py_mod_multiple_interpreters, Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},
2025-05-28T17:01:52.7847554Z             |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-05-28T17:01:52.7848532Z       psutil/_psutil_linux.c:71:36: error: ‘Py_MOD_PER_INTERPRETER_GIL_SUPPORTED’ undeclared here (not in a function)
2025-05-28T17:01:52.7849502Z          71 |     {Py_mod_multiple_interpreters, Py_MOD_PER_INTERPRETER_GIL_SUPPORTED},
2025-05-28T17:01:52.7850131Z             |                                    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2025-05-28T17:01:52.7850976Z       error: command '/opt/rh/devtoolset-10/root/usr/bin/gcc' failed with exit code 1
2025-05-28T17:01:52.7851612Z       [end of output]
2025-05-28T17:01:52.7851919Z   

Since this is all repeated code, I wonder if it makes sense to somehow define it in psutil/_psutil_common.c, and import/reuse it in the individual platform C files. Just a mild suggestion though, not really requesting anything here. ;)

[colesbury]

This pattern doesn't work well with stable ABI extensions. For example, if you compile psutil for cp36+ abi3, it won't have this slot set even when loaded into Python 3.12+.

https://pypi.org/project/psutil/#files

If you want it to work with the stable ABI, I think you need to hard code the constant and dynamically set the slot at runtime (i.e., in PyInit__psutil_posix).

The same would be true for Py_mod_gil, but the free threaded build doesn't currently work with the stable ABI, so it's not currently relevant.

[AA-Turner]

@colesbury Indeed, I've thinking about this but haven't come up with any good solutions. The 'simplest' is for psutil to additionally distribute a wheel built with the limited API set to 3.12+, which would double the number of wheels in the medium term, but still be O(1).

One potential solution would be to define two slots arrays, slots_py36 and slots_py312. We could then choose between them based on the runtime version (though Py_Version was only added in 3.11, so this might have to go via sys.hexversion). This would still need to hard-code the symbols that don't exist in Python 3.6, but avoids dynamic manipulation of the arrays.

cc @encukou for any advice you may have here!

A

[encukou]

I suggest not adding new functionality in this PR. Keep the PyUnstable_Module_SetGIL call, and remove the new slots for now.
Allowing per-interpreter GIL should be its own PR and discussion. (And preparing a good proposal for psutil devs might not be trivial.)

Alas, limited API does limit you to a minimum Python version. It is not designed to dynamically allow newer features.
We can change that, but this is not the place to do it.

[AA-Turner]

Thanks to Petr for the advice, I've removed the Py_mod_gil and Py_mod_multiple_interpreters slots from this PR and added back the PyUnstable_Module_SetGIL() call. @giampaolo please could you run CI?

A

[giampaolo]

@giampaolo please could you run CI?

It looks like there are failures on Windows.

[encukou]

@AA-Turner, the suggestions should fix those.

[encukou]

Suggested change

static int module_loaded = 0;

[encukou]

Suggested change

	.m_methods = mod_methods,
	.m_methods = PsutilMethods,

[giampaolo]

For the record, all the .c files recently changed due to the adoption of clang-format, so a rebase is required.