Yaml Anchors Breaking as of 5.3.0

[alteredtech]

I am working on a toy problem for work with the swift-openapi-generator. I have recently encountered an error where it no longer builds. After some investigation on this issue apple/swift-openapi-generator#746 I was able to narrow it down to this PR #428 for 5.3.0.

When I attempt to build on 5.3.0 I get this message

Error: /Users/alteredtech/Coding/swift-petstore-oas/Sources/openapi.yaml: error: Found neither a $ref nor a Response in .responses.200 for the **PUT** endpoint under `/pet`.

Response could not be decoded because:
Inconsistency encountered when parsing `Vendor Extension`: Found at least one vendor extension property that does not begin with the required 'x-' prefix. Invalid properties: [ yamlAnchor ]..

However, on 5.1.3 and 5.2.0 I do not and the build is successful.

My working commit example: https://github.com/Altered-Tech/swift-petstore-oas/tree/yaml-anchors
My breaking commit example: https://github.com/Altered-Tech/swift-petstore-oas/tree/yaml-anchors-fails

[mattpolzin]

To expound, the problem OpenAPIKit has is that the structure it’s Codable types are being passed appears to have a meta yamlAnchor property (i.e. the property did not appear in the original YAML source) and OpenAPIKit expects to be able to assert that the input structure has no unexpected properties.

[jpsim]

Thanks for the report! @lynchsft this seems related to your changes in #428, could you please take a look?

[lynchsft]

Has this issue been tested with Yams 5.3.1? It contains anchor-related bug fixes.

[lynchsft]

@mattpolzin Do we have contact with the OpenAPIKit maintainers? It sounds like they were pasting-on some anchor processing to a version of Yams prior to the introduction of first-party support. If that's truly the case, they should simply update their logic by deleting the third-party anchor handling.

[mattpolzin]

That'd be me; I maintain OpenAPIKit. I try not to trust my own memory going back too far, but the word yamlAnchor does not appear in the OpenAPIKit codebase and it doesn't appear in the repo linked to in the OP above so I assumed that Yams was injecting that property somehow during decoding.

[lynchsft]

Well hi! 👋🏻 I apologize for not checking the ownership.
I think I'm catching up now. The presence of anchors in @alteredtech's yaml files does add meta-property to the Node representation of the yaml tree. But if the decoded type does not conform to YamlTagCoding, then the anchor does not (cannot) appear in the decoded type.

I need a bit of info/source links from @mattpolzin. Is OpenAPIKit using the Yams Node APIs (lower level) or the top-level Decoder api (higher level)? Can you link to the spot(s) in OpenAPIKit where the decoding is actually happening so I can examine?

[mattpolzin]

OpenAPIKit is using the Codable interface. I suspect the problem OpenAPIKit has would occur for any Decodable type that supports arbitrary hash values. For OpenAPIKit this comes up for many types because OpenAPI allows for "vendor extensions" to define properties not specified by the OpenAPI Standard. You can see the code here, but the crux of it seems to me to be: A Decodable type's decode method cannot know that yamlAnchor was not part of the original YAML file because it shows up the same way as any other dynamically determined decoding key that was part of the source YAML would show up.

[alteredtech]

Has this issue been tested with Yams 5.3.1? It contains anchor-related bug fixes.

Yes it has. You can check the version diff I did in the original issue in this comment that spurred this issue.

I manually updated the package resolve with each yams version until it was able to build.

[lynchsft]

OK! That was highly clarifying. 😁 Thank you Matthew and thank you @alteredtech for the confirmation about tested versions.

The use of Codable's .keys property was the link I was missing.

Because OpenAPIKit specifically handles yaml and because I believe that the more generally helpful (to general use-cases) behavior is to include the meta properties in the Codable keys, it's my conclusion that the most incisive and correct way to resolve the issue is for OpenAPIKit to publish an update the ignores yamlAnchor and yamlTag as keys when performing the now-failing check. This approach would solve the OP's problem while not redacting useful metadata from other use-cases.

Thoughts?

[mattpolzin]

I'd rather not tie OpenAPIKit to a non-standard (as in not YAML specified) naming convention for what is essentially a reserved name. OpenAPIKit uses the Codable Protocols partly to avoid that vendor lock -- we depend on Yams only for the test suite, developers using OpenAPIKit may theoretically choose any YAML decoder or JSON decoder they'd like. To a lesser degree I am also not crazy about ignoring any real property found in the file (whether it's a YAML file or a JSON file) that happens to have one of those reserved words as its name. I see you've recognized this problem yourself and chosen for Yams to give preference to any real property with that name over the meta property if one exists.

Since Yams has a protocol that declares the yamlAnchor property, could Yams check that the type being decoded conforms to that protocol before injecting the yamlAnchor key?

[lynchsft]

That's a really valid argument. I apologize for misunderstanding your connection with Yams.
I'll make a test suit that targets the desired behavior and see what can be done. From my memory the needed type information was not available at the right place....

[lynchsft]

The experiment isn't gong well. As I recalled, Swift's Codable architecture rather intentionally does not pass the not-yet-known decoded type into the construction of the Decoder. Whether or not to retain/expose/parse the yaml tags and anchors cannot be decided by the conformances of the type which will later be decoded.

A different question had occurred to me during the exploration. @mattpolzin , is it not reasonable to consider the OP's use case (the presence of yamlAnchor or yamlTag) as genuine vendor extensions? From this perspective, OpenAPIKit has done its job perfectly. If we take this perspective then the questions becomes why didn't the decoded type declare the properties explicitly. @alteredtech can you confirm (or else explain) that the type you're trying to decode conforms to YamlAnchorCodable?

[mattpolzin]

@lynchsft I think we are still seeing this problem differently. The OP is not defining the types being decoded, OpenAPIKit is. Nothing downstream of Yams has made any mention of the YamlAnchorCodable protocol nor the yamlAnchor or yamlTag properties. The YAML being parsed does use anchors and the OP does want Yams to automatically dereference those anchors but there is no need to expose the anchors or tags to OpenAPIKit. In fact, OpenAPIKit sees yamlAnchor as a property, assumes that means the ingested YAML file literally defined that key, and complains because the OpenAPI specification does not allow vendor extensions without the x- prefix.

Could Yams support a userInfo property passed to the Decoder that turns off the yamlAnchor/yamlTag injection? Again, the support for dereferencing those anchors is desirable, just not the injection of those properties into the downstream decoding process via Codable. Then, although it isn't a desirable user experience, OpenAPIKit's documentation could recommend users of Yams turn that injection off when decoding YAML.

There's another option that would work for OpenAPIKit, but I doubt it is desirable to you because it would arbitrarily tie YAML decoding to the OpenAPI specification which is no more desirable than OpenAPIKit tying itself to Yams. That alternative idea would be to simply name the injected properties something starting with the x- prefix and therefore making them valid as OpenAPI vendor extensions.

[alteredtech]

Sorry for my delayed response, busy finishing up grad school.

@mattpolzin I would say covered my usage accurately. I am not defining types being decoded.

The crude usage explanation I am doing is having an openapi spec that has some yaml anchors, toss it into the Swift-OpenAPI-Generator that uses OpenAPIKit that uses Yams, get out server/client/types code with the yaml anchors all dereferenced.