Support for pyspark.sql.Column in dataclasses-json

[bhupesh-simpledatalabs]

Hello,

In my dataclasses i am using Column field of pyspark.sql as well. Is it possible to extend dataclasses-json to define json encoding and decoding for Column field types as well?

My encoding and decoding functions will look like below

def encoder(self:Column) -> str:
    return f'{self._jc}'
def decoder(self, colString) -> Column:
    return expr(colString)

I am extending Column class in python to add encoder and decoder method to enable to_json and from_json call on Column fields. I also tried specifying these encoder and decoder methods in my dataclass via below

@dataclass_json
@dataclass(frozen=True)
class ColumnExpression:
    target: str
    expression: Column = field(
        metadata=config(
            encoder=Column.encoder,
            decoder=Column.decoder
        )
    )

I am getting below error when i try to call to_json() on the instance of my own ColumnExpression dataclass

Traceback (most recent call last):
  File "/Users/bhupeshgoel/Documents/codebase/ComponentBuilderPython/cb/server/ComponentBuilderBase.py", line 167, in <module>
    print(cc.to_json())
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/dataclasses_json/api.py", line 50, in to_json
    return json.dumps(self.to_dict(encode_json=False),
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/dataclasses_json/api.py", line 86, in to_dict
    return _asdict(self, encode_json=encode_json)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/dataclasses_json/core.py", line 323, in _asdict
    value = _asdict(getattr(obj, field.name), encode_json=encode_json)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/dataclasses_json/core.py", line 338, in _asdict
    return copy.deepcopy(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle '_thread.RLock' object

May i know if i am doing anything wrong here?

[george-zubrienko]

Hi @bhupesh-simpledatalabs, this question is not really related to library functionality, so I'm going to move this to discussions instead. As to "am I doing something wrong", I'd say yes, you can't have a type that is linked to a distributed collection in a file format that is designed for single-node storage.