Update node serialization/deserialization and other Pydantic issues

[edan-bainglass]

This PR addresses several issues with #6255 that came up during updates to the aiida-restapi in aiidateam/aiida-restapi#93.

Major changes

• All repository file serialization in JSON dumps (via Entity.serialize()) are removed due to impracticality.
• Node subclass Models no longer extend Node.Model. Instead, Node.Model.attributes is typed as Node.AttributesModel, which is concretely implemented by Node subclasses with explicit attributes definition (see this issue for details). Code models still extend Node.Model here and there for CLI metadata support.
• Several changes made to support the AttributesModel refactor. Of note is that now node construction from model will pass arguments to Node ORM constructors under the attributes key. As such, constructors are adjusted to account for this. This change also supports Node creation via the RESTAPI.

Note for PR reviewers

There are a few changes that are likely out of scope. These will move to dedicated PRs prior to merge of this PR.

---

Happy to discuss 🙂

[codecov]

Codecov Report

❌ Patch coverage is 89.50564% with 121 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.37%. Comparing base (2068a7d) to head (0e80533).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/aiida/orm/nodes/node.py	85.47%	33 Missing ⚠️
src/aiida/orm/nodes/data/array/kpoints.py	51.22%	20 Missing ⚠️
src/aiida/orm/fields.py	69.24%	12 Missing ⚠️
src/aiida/orm/pydantic.py	84.73%	11 Missing ⚠️
src/aiida/orm/entities.py	93.75%	7 Missing ⚠️
src/aiida/orm/nodes/data/array/array.py	85.42%	7 Missing ⚠️
src/aiida/orm/nodes/data/array/xy.py	88.24%	6 Missing ⚠️
src/aiida/orm/nodes/data/structure.py	83.34%	6 Missing ⚠️
src/aiida/cmdline/groups/dynamic.py	58.34%	5 Missing ⚠️
...rc/aiida/storage/psql_dos/orm/querybuilder/main.py	16.67%	5 Missing ⚠️
... and 8 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6990      +/-   ##
==========================================
+ Coverage   80.28%   80.37%   +0.10%     
==========================================
  Files         577      578       +1     
  Lines       45545    46112     +567     
==========================================
+ Hits        36562    37060     +498     
- Misses       8983     9052      +69     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[edan-bainglass]

@khsrali regarding one of the failed tests...

tests/orm/nodes/data/test_remote_stash.py::test_constructor_invalid_folder[stash_mode-copy] - ValueError: `RemoteStashFolderData` can only be used with `stash_mode == StashMode.COPY`

In this PR, I lift some validation to the top of the constructor (of stash data classes - see 91e4ad5), to avoid any operations if the object is bound to fail. However, this seems to introduce failures in testing them. It suggests that perhaps there is some order to the operations, though I don't see it. Can you please comment?

[edan-bainglass]

@khsrali regarding one of the failed tests...

tests/orm/nodes/data/test_remote_stash.py::test_constructor_invalid_folder[stash_mode-copy] - ValueError: `RemoteStashFolderData` can only be used with `stash_mode == StashMode.COPY`

In this PR, I lift some validation to the top of the constructor (of stash data classes - see 91e4ad5), to avoid any operations if the object is bound to fail. However, this seems to introduce failures in testing them. It suggests that perhaps there is some order to the operations, though I don't see it. Can you please comment?

It was checking for TypeError, which is what happens when you delay the stash_mode check and provide a non-Enum type for it. But there is no need for it, as the stash_mode check covers that issue as well. I updated the test to parameterize also over error, checking for ValueError in the case of the invalid stash_mode.

[edan-bainglass]

@GeigerJ2 okay, this is ready for others to inspect. I did my best to isolate the commits and provided comments on each. Happy to discuss further.

Pinging also @agoscinski @superstar54 if interested.

Pinging @sphuber for input/feedback, if he has time.

[edan-bainglass]

It may be possible to rely on the post models of aiida-restapi as a reference for defining ORM constructor parameters, as the post models are intended to represent serialized objects passed to the REST API for object construction. Looking into this.

[edan-bainglass]

@danielhollas what is this about?

Critical: potential `verdi` speed problem: `aiida.orm.fields` module is imported which is not in: ('aiida.brokers', 'aiida.cmdline', 'aiida.common', 'aiida.manage', 'aiida.plugins', 'aiida.restapi')

Nevermind. I see that importing orm in the cmdline raises alarm bells. Removed...

[danielhollas]

Nevermind. I see that importing orm in the cmdline raises alarm bells. Removed...

Nice, the system works. :-) Feel free to improve the error message here to make it more obvious that this is about the aiida.cmdline module.

[edan-bainglass]

Nevermind. I see that importing orm in the cmdline raises alarm bells. Removed...

Nice, the system works. :-) Feel free to improve the error message here to make it more obvious that this is about the aiida.cmdline module.

#7059

[edan-bainglass]

@danielhollas what do you think about ignoring Ruff N806 - "~variable should be lowercase"? See case below. Model is not an instance, but a class. Naming it model would be misleading, hence I went with Model.

Model = cls.Model.as_input_model()
^^^^^ N806

[danielhollas]

@danielhollas what do you think about ignoring Ruff N806 - "~variable should be lowercase"?

Do you mean ignoring locally (fine) or globally?

[edan-bainglass]

@danielhollas what do you think about ignoring Ruff N806 - "~variable should be lowercase"?

Do you mean ignoring locally (fine) or globally?

Global. There are many cases when the variable is a class, not an instance. I've pushed this in my last commit just to verify that it works. Okay with removing it in favor of local handling, but would like to hear the reason against a global N806 rule.

Update

Nice. Ignoring N806 globally raised a whole lot of RUF100 due to the codebase being littered with local N806 rules. I'd say that supports my case 🙂

[edan-bainglass]

@danielhollas done for tonight. Will revisit this in the morning 😴

[danielhollas]

Nice. Ignoring N806 globally raised a whole lot of RUF100 due to the codebase being littered with local N806 rules. I'd say that supports my case 🙂

Yeah, running git grep N806 indeed does get a lot of hits (although most of them are in tests/).

Seems fine to remove it, or alternatively, use the https://docs.astral.sh/ruff/settings/#lint_pep8-naming_extend-ignore-names configuration to automatically ignore some common patterns (e.g. the one you have here, and common class names produced by factory functions (SinglefileData = DataFactory('core.singlefile')

e.g. something like this in pyproject.toml

[tool.ruff.lint.pep8-naming]
ignore-names = ["[A-Z]*Data", "Model"]

In any case please open a separate PR for that so we don't polute this one with bikeshedding discussion and unrelated changes.

[edan-bainglass]

In any case please open a separate PR for that so we don't polute this one with bikeshedding discussion and unrelated changes.

Thanks @danielhollas. Then for this one, since there are only a few cases in my PR, I will locally ignore them. Will open a PR for the pattern handling shortly after.

[mbercx]

LGTM 🚀

aiidateam/aiida-quantumespresso#1266

[mbercx]

(still) LGTM! 🚀

I had a quick look at the new docs, as we discussed, but it's tricky to meaningfully review them without understanding the changes and how they are used. One should probably try to implement a new data type with the models/fields to see if the documentation has any gaps. If I get around to that, I'll cycle back and give a more useful review/feedback of the documentation.