How should we decompose benchmark circuits?

[jordandsullivan]

I've made a number of discoveries in implementing our benchmarks in this notebook for #2:

There is an issue with the way we count 2 qubit gates using Qiskit:

raw_gates = qiskit_circuit.count_ops()
num_raw_gates = raw_gates.get("cz", 0) + raw_gates.get("cx", 0) + raw_gates.get("cy", 0)

This works fine if a compiler is already compiling down to 2 qubit gates, but qiskit.transpile does NOT do this by default. It produces compiled circuits with Rxx and Ry gates for instance:

which is fine, we can add controlled rotations to the list of operations we check (although it would clearly be more convenient if Qiskit simply provided a utility to count all 2-qubit operations).

But where I reach a genuine condundrum is where Qiskit's default pass compiles to abstract multi-qubit unitaries:

Is it fair to compile the circuit down to 2 qubit gates beforerunning qiskit.transpile(circuit), even though Qiskit would not do this by default? I have already had to make the decision of choosing a rather arbitrary subset of PyTKET and Cirq's compiler passes to try to make a comparison (since neither has a default set that I could find), both of which include compiling down to 2 qubit gates.

Benchmarking these compilers is a bit tricky bc I don't want to write a full set of compiler passes for Cirq, Qiskit, and TKET just to compare them, but I have to make some choices in order to get data. For instance, PyTKET has a pass called FullPeepholeOptimise, which you might want to call a default, but it takes 200x as long, so is that a fair comparison?

[jordandsullivan]

@sonikaj I would appreciate your input here

[sonikaj]

Hi Jordan, try passing it a backend of the form:

from qiskit.providers.fake_provider import GenericBackendV2
backend = GenericBackendV2(num_qubits=num_qubits, basis_gates = ['rz', 'rx', 'ry', 'cx'])

That should force it to compile to this gate set before it creates the final circuit. I think this is fair for comparison because most of the hardware backends will only accept this kind of gate set, and it's what we are targeting as well.

[jordandsullivan]

These aren't final numbers, but here is an example of what I'm talking about if we decompose into the above target gate_set, these are the results (see my notebook):

@sonikaj could you weigh in again?

[sonikaj]

Yes, I would say that the decomposition is usually counted as part of compiling.

[jordandsullivan]

@sonikaj does it surprise you that there is essentially no reduction in gate counts for 4/6 of the benchmarks when we decompose the circuits into CNOT, H, Rx, Ry, Rz beforehand?

[sonikaj]

Is this including the CX gate pass? Without that, there is likely not much to reduce in these circuits. In that sense, they are really only benchmarks when you include the mapping pass.

[jordandsullivan]

No it doesn't include the Cx pass yet. Good to know.

[jordandsullivan]

In the current version, I have decided to include both decomposed versions of the files (as indicated in the filename) as well as circuits without any gate decomposition. To count the gates on the non-decomposed circuits, I am using custom functions to count the number of multi-qubit gates (rather than just 2 qubit gates): d0483d2#diff-4885f6e829f132f4fff686b2f3e6b90269b606006bf318760a0a6a8d3f8afcba

[willzeng]

I’d agree that for best comparison between toolchains the final target gate set should be fixed.

Failing that, definitely do need to force it down to single and two qubit gates for now.

For the medium term, we can have different benchmark tasks that measure performance against different hardware targets. For example neutral atoms and ion traps have some multi qubit gates and we’d want tasks that see if compilers are good at targeting them.