Support for outputting optional data resource fields

[ialarmedalien]

I'd like to be able to extend the standard frictionless data resource and package to allow the addition and output of extra fields.

I can add extra fields (attributes) to Resource objects, but if I want to output those fields as part of a DataPackage manifest, I have to iterate through the resources and run to_json() or .metadata_export() and manually add in the extra key-value pairs.

Is there a recommended way to put in extra Resource fields or should I make my own Resource subclass / continue to manually interpolate extra data?

[pierrecamilleri]

That is a good question - and sorry if my answer is a little hesitant, I am still discovering some aspects of the code.

I reproduced the loss of custom properties in the descriptor.

resource = TableResource(path="table.csv")
resource.abc = "abc"
print(resource.to_descriptor()) # No "abc" key

In the code, this happens in metadata_export keeping only the keys from the profile -- not sure if this is a feature or a bug… I'd say this is not the desired behavior, I would expect (as I have written in the docstring) "The deserialization and serialization must be lossless".

For the moment, if you know the extra-attributes in advance, I think the easiest is to create a Resource subclass like that :

class MyTableResource(TableResource):
    metadata_profile_patch = {"properties": {"abc": {"type": "string"}}}

resource = MyTableResource(path="table.csv")
resource.abc = "abc"
print(resource.to_descriptor()) # includes "abc"

Note that this does not work on Resource, only on the already specialized resource class (like TableResource) as there is some magic (metaclass = Factory) happening at initialization that casts the result to another class.

Please note that I would like to remove the metadata shenanigans with the next major version, and this may break.

[ialarmedalien]

Thanks for the response!

Is there an ETA on the next major version / metadata shenanigans removal so I can assess whether I should DIY a resource subclass locally or just wait for a new version?

[pierrecamilleri]

No ETA but it is a big project (Metada is tightly coupled with much of the codebase) , so not soon and I wouldn't recommend to wait