parallel json parsing

[dimatr]

On a 2.2 GB test this drops the JSONparser time from 3 min to 30 sec

[josiahseaman]

python -m pip install ray
Collecting ray
ERROR: Could not find a version that satisfies the requirement ray (from versions: none)
ERROR: No matching distribution found for ray

There's clearly a repo here: https://pypi.org/project/ray/ I've updated pip. Perhaps this is pypy only distro? Could you please give me more info on ray install compatibility and whether we'll be able to build it into the requirements.txt or some other automated solution?

[josiahseaman]

The issue appears to be that wheels are only built for Linux and MacOS, but I'm testing on a Windows machine. The lack of support makes sense really, but it should at least be noted this would be the first departure of Pantograph from being cross-platform compatible. Would you mind writing an OS switch and local import instead?

https://ray.readthedocs.io/en/latest/installation.html

[lomereiter]

Also consider using an OS-independent library (joblib?)

[subwaystation]

Do we really need support for Windows?
There is no Windows support for e.g. odgi, too.
We will have our docker pipeline, so as long as docker is available, the whole thing will work out.

[dimatr]

Please check now. The current change works on Linux and Windows

[subwaystation]

Reading in the data on a 28 core machine is much faster now! Thanks @dimatr .

Just one minor thing: How about a parameter where users can specify the number of cores to use?

[dimatr]

I will add one new parameter --parallel-cores with default os.cpu_count()

[dimatr]

should be ready now, please check