mqt.core.evaluation

Evaluating the json file generated by the benchmarking script.

Module Contents

compare(baseline_filepath: str | PathLike[str], feature_filepath: str | PathLike[str], factor: float = 0.1, sort: str = 'ratio', dd: bool = False, only_changed: bool = False, no_split: bool = False, algorithm: str | None = None, task: str | None = None, num_qubits: int | None = None) → None[source]

Compare the results of two benchmarking runs from the generated json file.

Parameters:

  • baseline_filepath – Path to the baseline json file.

  • feature_filepath – Path to the feature json file.

  • factor – How much a result has to change to be considered significant.

  • sort – Sort the table by this column. Valid options are “ratio” and “algorithm”.

  • dd – Whether to show the detailed DD benchmark results.

  • only_changed – Whether to only show results that changed significantly.

  • no_split – Whether to merge all results together in one table or to separate the results into benchmarks that improved, stayed the same, or worsened.

  • algorithm – Only show results for this algorithm.

  • task – Only show results for this task.

  • num_qubits – Only show results for this number of qubits. Can only be used if algorithm is also specified.

Raises:

  • ValueError – If factor is negative or sort is invalid or if num_qubits is specified while algorithm is not.

  • FileNotFoundError – If the baseline_filepath argument or the feature_filepath argument does not point to a valid file.

  • JSONDecodeError – If the baseline_filepath argument or the feature_filepath argument points to a file that is not a valid JSON file.

main() → None[source]

Main function for the command line interface.

This function is called when running the mqt-core-compare CLI command.

mqt-core-compare baseline.json feature.json [options]

In addition to the mandatory filepath arguments, it provides the following optional command line options:

  • --factor <float>: How much a result has to change to be considered significant.

  • --sort: Sort the table by this column. Valid options are ‘ratio’ and ‘algorithm’.

  • --dd: Whether to show the detailed DD benchmark results.

  • --only_changed: Whether to only show results that changed significantly.

  • --no_split: Whether to merge all results together in one table or to separate the results into benchmarks that improved, stayed the same, or worsened.

  • --algorithm <str>: Only show results for this algorithm.

  • --task <str>: Only show results for this task.

  • --num_qubits <int>: Only show results for this number of qubits. Can only be used if algorithm is also specified.