QPY dump and load in Rust

[gadial]

Summary

This PR adds implementations of the qpy module dump and load functions with Rust, making it possible to have a full QPY flow directly in Rust without Python.

Details and comments

This PR adds a top level QPYFile struct to the rust QPY framework, handling the general file metadata and the the circuits. A new file, interface.rs, handles the actual dump and load functions. We have python wrappers that are called from python's interface.py for supported versions.

Since we still don't have a pure rust QuantumCircuit, the rust functions dump_qpy and load_qpy still have Python remnants in their I/O. In particular, load_qpy returns python circuits (as Py<PyAny>) and dump_qpy accepts the QuantumCircuitData struct which is used as a transition layer from python's QuantumCircuit.

The circuit offset table, present in QPY17, is actually not required at all since all the size data for a given circuit is included in its serialization. Moreover, computing the circuit offset table is not trivial in the binrw framework. So to avoid complicating our data structures with a field that will become obsolete in QPY18, I added another struct, QPY17File. When converting QPYFile to QPY17File the circuits are serialized, the offset table is correctly calculated, and then QPY17File can also be written using the standard binrw macros.

[qiskit-bot]

One or more of the following people are relevant to this code:

• @Qiskit/terra-core
• @mtreinish

[qiskit-bot]

One or more of the following people are relevant to this code:

• @Qiskit/terra-core
• @mtreinish

[eliarbel]

Thanks for the PR Gadi. It's a good step forward which will get us much closer to being able to expose QPY in the C API. Before delving into the fine details of the PR, I wanted to discuss a few general points:

1. The original purpose of the circuit offset table is to allow concurrent processing during deserialization for better performance. For example, in the Rust QPY world, this could materialize as concurrently unpacking multiple circuits from an in-memory sliced buffer. Why do you want to remove it in QPY18? I think we should at least experiment a little with parallel unpacking to see how much speedup, if any, we can get from such a scheme, before we decide to drop it. Could you have a side experiment for this? I think it should be straight forward already with the Rust QPY on main.
2. It seems to me that with QPYFile and QPYV17File we somewhat abuse the general concept behind binrw, taking it from "map this byte stream to the layout described by this struct" (and vice versa) to "abstract and own QPY file handling entirely". The latter could be fine if we eventually decide to drop the offset table and always treat a given QPY file as linear storage. But if we end up supporting parallel file processing, the binrw abstraction at the file level is probably not the right way to go.
3. Since the file header is versioned in the QPY format, it's probably better if we have an explicit representation for it as a binrw-derived struct (similarly to how we have e.g. CircuitHeaderV12Pack). What do you think?

[gadial]

Thanks for the PR Gadi. It's a good step forward which will get us much closer to being able to expose QPY in the C API. Before delving into the fine details of the PR, I wanted to discuss a few general points:

1. The original purpose of the circuit offset table is to allow concurrent processing during deserialization for better performance. For example, in the Rust QPY world, this could materialize as concurrently unpacking multiple circuits from an in-memory sliced buffer. Why do you want to remove it in QPY18? I think we should at least experiment a little with parallel unpacking to see how much speedup, if any, we can get from such a scheme, before we decide to drop it. Could you have a side experiment for this? I think it should be straight forward already with the Rust QPY on main.
2. It seems to me that with QPYFile and QPYV17File we somewhat abuse the general concept behind binrw, taking it from "map this byte stream to the layout described by this struct" (and vice versa) to "abstract and own QPY file handling entirely". The latter could be fine if we eventually decide to drop the offset table and always treat a given QPY file as linear storage. But if we end up supporting parallel file processing, the binrw abstraction at the file level is probably not the right way to go.
3. Since the file header is versioned in the QPY format, it's probably better if we have an explicit representation for it as a binrw-derived struct (similarly to how we have e.g. CircuitHeaderV12Pack). What do you think?

1. Concurrent unpacking is definitely on the table - if I suggested to remove the offset table, it's because I didn't realize at the time that this was its purpose.
2. I agree, if we go for concurrent processing this will change.
3. My experience so far was that it's relatively simple to handle all versions in one struct, instead of having different structs for different versions, and use local ifs when required. This is because we had no "big" changes in QPY13-17. If we take the opportunity to use QPY18 for a large overhaul, we'll probably use different structs - but for now I don't think this is required.