Update PyBind11 to version 3.0.1

[jstone-lucasfilm]

This changelist updates the default version of PyBind11 to 3.0.1, enabling MaterialX builds with the recently released Python 3.14.

[ld-kerley]

I have no objections to this PR - just a few questions. We don't use the python bindings all that much and I'm certainly no expert here.

What version of pybind11 were we using before? I didn't see anything in the PR diff that changed. It might be helpful to note the version of pybind11 that we're localizing here.

Are there any compatibility concerns with moving from the old version to this one? Places could well be using MaterialX pybind11 bindings in coordination with other pybind11 libraries - I know we would have done that at Imageworks.

Is it possible for someone to provide their own external pybind11? If so do we test this in the CI? And also what is the minimum pybind11 version that works? ie. do we have a compatibility range, is that documented anywhere?

Are we maintaining any local patches on top of the stock pybind11 source? I couldn't see any comments in the PR annotating any local differences. If we aren't maybe we could consider moving this to a git submodule, this would make future upgrades easier.

[jstone-lucasfilm]

Good questions, @ld-kerley, and I'm not aware of any compatibility concerns with this upgrade, as the minimum Python and C++ requirements in PyBind11 3.0.1 align with those of the MaterialX project.

The previous version of PyBind11 in MaterialX was 2.13.6, and the most important limitation addressed in version 3.0.1 is support for Python 3.14, which is now available to MaterialX clients in the industry, and is likely to become popular in the next year.

We don't maintain any modifications to PyBind11, which would potentially allow its integration as a submodule, though I believe the more likely next step is that we'll move to the more GPU-friendly NanoBind, following the lead of the OpenVDB project. That step will require coordination across the ASWF, though, and isn't a change that we would make unilaterally.

[ld-kerley]

Are 2.13.6 pybind bindings compatible with 3.0.1 built bindings? If not then it might be problematic for people who have extensive bindings for other projects already built against an older version. I believe in a prior role we had to be careful to keep pybind11 versions aligned in a single runtime environment - but again I'll caveat that I'm no expert here.

I guess the concern here is that we're implicitly forcing all new builds of MatX python bindings to 3.0.1, even if not building for python 3.14 - which might be problematic if things aren't compatible.

[jstone-lucasfilm]

@ld-kerley That's a good question, though we've upgraded our PyBind11 version continually over the course of the MaterialX project without introducing compatibility issues for clients.

The main concern historically has been that we provide a default version of PyBind11 that supports all of the platforms and C++/Python versions that the current version of MaterialX itself supports, and PyBind11 3.0.1 achieves that goal.

We also support external versions of PyBind11 for clients that need this, though I'm not aware of any specific clients that have chosen this path.

[ld-kerley]

is that external pybind11 path tested in CI - if so then I think it frees us to make these sort of changes with confidence.

I'm pretty sure Imageworks builds of MaterialX were using externally provided pybind11 when I was still there.

Related - @lgritz has an approach for external dependencies that might be interested to look at - I believe he has cmake logic that can download (and build) any depdendencies not provided - an approach like this might mean we can have our cake and eat it - meaning batteries are included - without having to localize source.

[jstone-lucasfilm]

@ld-kerley We're not yet testing the external PyBind11 pathway in our GitHub Actions test suite, though that sounds like a great idea for the future.

Although I like the external dependencies approach in the OSL and OIIO projects, we've intentionally taken a very different, "batteries included" approach on the MaterialX project, minimizing the need for build-time downloading of external content. It's a great discussion for the future as to whether the "batteries included" approach of MaterialX is still ideal, but it's a very intentional design decision that we've made.

[ld-kerley]

Oh glad you're aware of the OSL/OIIO approach - I only mentioned it because in the past there has been concerns around the side of the repository for download - I believe you mentioned Unreal downloads and builds MaterialX every UE build - and if that build doesn't need pybind, this approach could save those downloads. I think you mentioned having number of 1000's of UE builds every day, so it might be significant.

Anyway - as I said I was just highlighting some questions - we don't really use the python bindings. Maybe some of this is worth discussing a little wider than this PR, either on slack in on the TSC -just for visibility of other users.

[lgritz]

I believe the more likely next step is that we'll move to the more GPU-friendly NanoBind, following the lead of the OpenVDB project. That step will require coordination across the ASWF, though, and isn't a change that we would make unilaterally.

I'm afraid that ship has long since sailed -- OpenVDB already unilaterally switched to nanobind.

I don't believe there are compatibility problems of mixing them most of the time... as long as package A's exported Python types aren't also used by packages B's python bindings (as is the case for Imath+Alembic, and for OIIO+OSL). In those cases, I think it can only work if A and B switch to nanobind at the same time. (Since OpenVDB doesn't use another package's python types as parameters to its own python-bound functions, I think what they did turns out to be safe.)

Wenzel hinted to us at some point that future nanobind and pybind11 changes might make it possible to mix and match, but that seemed like a speculative thing and probably all switching to nanobind might be a more prudent approach.

[jstone-lucasfilm]

@lgritz It's true that OpenVDB has already taken the leap, but it's really the integration between MaterialX and OSL/OIIO/OCIO that we're most focused on. So perhaps there's still an opportunity for this set of projects to move forward together in a transition from PyBind11 to NanoBind. There's no hurry from our perspective, as PyBind11 still works very well and is actively developed, but the GPU-friendly aspects of NanoBind are likely to become a stronger draw over time.

[lgritz]

@lgritz It's true that OpenVDB has already taken the leap, but it's really the integration between MaterialX and OSL/OIIO/OCIO that we're most focused on. So perhaps there's still an opportunity for this set of projects to move forward together in a transition from PyBind11 to NanoBind.

Strength in numbers -- I would be happy to try to move all three projects forward simultaneously.

But note that mixing is only a problem in practice if MaterialX's python bindings actually depend on exported OIIO or OSL types in the bindings themselves. For example, if MaterialX has a function (exposed by the python bindings) that has an argument that is an OIIO.TypeDesc. If the APIs are not sharing types, I think they don't need to be in sync.

[JGamache-autodesk]

Looks extremely good to me.

One thing to check. Make sure that GuessPythonExtSuffix worked and that the libraries have the right extension. The Python version should feature prominently.
Windows: PyMaterialXCore.cp313-win_amd64.pyd

Now, for the reason this change is necessary. Let's suppose I have a derivative project and I want to drop a binary MaterialX into it. On Linux this requires the derivative project be compiled on exactly the same compiler and compiler version as was used to build the binary MaterialX distribution.

The failure case is complex though. If you have PyBind11 code in a derivative project that expect to receive a PyElement and have it convert automatically to a Mx::Element::Ptr. This will only work if the compilers match with PyBind11 pre-3.0. With 3.0 you can compile MaterialX with GCC and the derived project with LLVM and the bindings will works because PyBind11 will now only check for C++ ABI compatibility, and this issue is fixed now in all Linux compilers. Last one to update was RedHat with its _GLIBCXX_USE_CXX11_ABI trek into C++ past.

But, strong caveat, the derived project must also upgrade to PyBind11 >= 3.0.0 since this change is ABI breaking. Painful, but necessary.

I discovered the compatibility issue while trying to build LookdevX coverage with LLVM and linking with a UFE built on GCC 11. All the Python bindings in LookdevX failed miserably. Now, both UFE and LookdevX are on PyBind11 >= 3.0.0.

[jstone-lucasfilm]

Thanks for the great analysis, @JGamache-autodesk, and that's very useful context.

I've checked the extensions on the generated Python libraries for this PR, and they match the convention that you describe above.