Benchmarking the DD Package¶
MQT Core provides a benchmark suite to evaluate the performance of the DD package. This can be especially helpful if you are working on the DD package and want to know how your changes affect its performance.
Generating results¶
In order to generate benchmark data, MQT Core provides the mqt-core-dd-eval CMake target (which is made available by passing -DBUILD_MQT_CORE_BENCHMARKS=ON to CMake). This target will run the benchmark suite and generate a JSON file containing the results. To this end, the target takes a single argument which is used as a component in the resulting JSON filename.
After running the target, you will see a results_<your_argument>.json file in your build directory that contains all the data collected during the benchmarking process. An exemplary results_<your_argument>.json file might look like this:
1import json
2from pathlib import Path
3
4filepath = Path("../test/python/results_baseline.json")
5
6with filepath.open(mode="r", encoding="utf-8") as f:
7 data = json.load(f)
8 json_formatted_str = json.dumps(data, indent=2)
9 print(json_formatted_str)
{
"BV": {
"Functionality": {
"1024": {
"dd": {
"active_memory_mib": 0.26538848876953125,
"compute_tables": {
"matrix_add": {
"col_ratio": 0.005506607929515419,
"collisions": 45,
"hit_ratio": 0.5,
"hits": 4086,
"inserts": 4041,
"load_factor": 0.0283203125,
"lookups": 8172,
"memory_MiB": 1.125,
"num_buckets": 0,
"num_entries": 0,
"peak_num_entries": "unused"
}
}
},
"runtime": 1.2
},
"256": {
"dd": {
"active_memory_mib": 0.06616973876953125,
"compute_tables": {
"matrix_add": {
"col_ratio": 0.008893280632411068,
"collisions": 18
}
},
"matrix": {
"unique_table": {
"total": {
"col_ratio": 0.00514176735528262,
"collisions": 2475
}
}
}
},
"runtime": 0.440665166
}
},
"Simulation": {
"256": {
"dd": {
"active_memory_mib": 0.021484375,
"matrix": {
"unique_table": {
"total": {
"col_ratio": 0.0029885811069223626,
"collisions": 1094
}
}
}
}
}
}
},
"GHZ": {
"Functionality": {
"512": {
"dd": {
"compute_tables": {
"matrix_add": {
"col_ratio": 0.011530912659470068,
"collisions": 47
}
}
},
"runtime": 0.2
}
}
}
}
To compare the performance of your newly proposed changes to the existing implementation, the benchmark script should be executed once based on the branch/commit you want to compare against and once in your new feature branch. Make sure to pass different arguments as different file names while running the target (e.g. baseline and feature).
Running the comparison¶
There are two ways to run the comparison. Either you can use the Python module mqt.core.evaluation or you can use the CLI.
Both ways are shown below.
In both cases, you need to have the evaluation extra of mqt-core (i.e., mqt.core[evaluation]) installed.
Using the Python package¶
The Python package provides a function compare() that can be used to compare two generate result files.
The function takes two arguments, the file path of the baseline json and the file path of the json results from your changes.
The function will then print a detailed comparison. An exemplary run is shown below.
1from mqt.core.evaluation import compare
2
3baseline_path = "../test/python/results_baseline.json"
4feature_path = "../test/python/results_feature.json"
5compare(baseline_path, feature_path)
Runtime:
Benchmarks that have improved:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|-----:|
| 1.2 | 0.579 | 0.482 | BV | Functionality | 1024 |
Benchmarks that have worsened:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|----:|
| 0.2 | 0.428 | 2.142 | GHZ | Functionality | 512 |
Benchmarks that have stayed the same:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|----:|
| 0.441 | 0.428 | 0.972 | BV | Functionality | 256 |
| nan | 0.56 | nan | BV | Simulation | 256 |
Note that the method offers several parameters to customize the comparison. See mqt.core.evaluation.compare().
An exemplary run adjusting the parameters is shown below.
1compare(baseline_path, feature_path, no_split=True)
Runtime:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|-----:|
| 1.2 | 0.579 | 0.482 | BV | Functionality | 1024 |
| 0.441 | 0.428 | 0.972 | BV | Functionality | 256 |
| 0.2 | 0.428 | 2.142 | GHZ | Functionality | 512 |
| nan | 0.56 | nan | BV | Simulation | 256 |
Using the CLI¶
In an even simpler fashion, the comparison can be run from the command line via the automatically installed CLI. Examples of such runs are shown below.
1! mqt-core-compare ../test/python/results_baseline.json ../test/python/results_feature.json --factor=0.2 --only_changed
Runtime:
Benchmarks that have improved:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|-----:|
| 1.2 | 0.579 | 0.482 | BV | Functionality | 1024 |
Benchmarks that have worsened:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|----:|
| 0.2 | 0.428 | 2.142 | GHZ | Functionality | 512 |
1! mqt-core-compare ../test/python/results_baseline.json ../test/python/results_feature.json --no_split --dd --task=functionality
Runtime:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|-----:|
| 1.2 | 0.579 | 0.482 | BV | Functionality | 1024 |
| 0.441 | 0.428 | 0.972 | BV | Functionality | 256 |
| 0.2 | 0.428 | 2.142 | GHZ | Functionality | 512 |
DD Package details:
| before | after | ratio | algo | task | n | component | metric |
|---------:|----------:|--------:|-------:|--------------:|-----:|--------------------------:|------------------:|
| 8172 | 0 | 0 | BV | Functionality | 1024 | compute_tables_matrix_add | lookups |
| 0.009 | 0.005 | 0.556 | BV | Functionality | 256 | compute_tables_matrix_add | col_ratio |
| 18 | 10 | 0.556 | BV | Functionality | 256 | compute_tables_matrix_add | collisions |
| 0.5 | 0.8 | 0.625 | BV | Functionality | 1024 | compute_tables_matrix_add | hit_ratio* |
| 2475 | 1873 | 0.757 | BV | Functionality | 256 | matrix_unique_table | total_collisions |
| 0.005 | 0.004 | 0.757 | BV | Functionality | 256 | matrix_unique_table | total_col_ratio |
| 0.012 | 0.009 | 0.787 | GHZ | Functionality | 512 | compute_tables_matrix_add | col_ratio |
| 47 | 37 | 0.787 | GHZ | Functionality | 512 | compute_tables_matrix_add | collisions |
| 4041 | 4033 | 0.998 | BV | Functionality | 1024 | compute_tables_matrix_add | inserts |
| 0.265 | 0.265 | 1 | BV | Functionality | 1024 | | active_memory_mib |
| 0.066 | 0.066 | 1 | BV | Functionality | 256 | | active_memory_mib |
| 0 | 0 | 1 | BV | Functionality | 1024 | compute_tables_matrix_add | num_entries |
| 0.028 | 0.029 | 1.009 | BV | Functionality | 1024 | compute_tables_matrix_add | load_factor |
| 0.006 | 0.006 | 1.178 | BV | Functionality | 1024 | compute_tables_matrix_add | col_ratio |
| 45 | 53 | 1.178 | BV | Functionality | 1024 | compute_tables_matrix_add | collisions |
| 4086 | 2000 | 2.043 | BV | Functionality | 1024 | compute_tables_matrix_add | hits* |
| 0 | 16384 | inf | BV | Functionality | 1024 | compute_tables_matrix_add | num_buckets |
| 1.125 | nan | nan | BV | Functionality | 1024 | compute_tables_matrix_add | memory_MiB |
| nan | 767 | nan | BV | Functionality | 1024 | compute_tables_matrix_add | peak_num_entries |
| nan | 16384 | nan | BV | Functionality | 256 | compute_tables_matrix_add | num_buckets |
1! mqt-core-compare ../test/python/results_baseline.json ../test/python/results_feature.json --dd --algorithm=bv --num_qubits=1024
Runtime:
Benchmarks that have improved:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|-----:|
| 1.2 | 0.579 | 0.482 | BV | Functionality | 1024 |
Benchmarks that have worsened:
| before | after | ratio | algo | task | n |
|----------|---------|---------|--------|--------|-----|
Benchmarks that have stayed the same:
| before | after | ratio | algo | task | n |
|----------|---------|---------|--------|--------|-----|
DD Package details:
Benchmarks that have improved:
| before | after | ratio | algo | task | n | component | metric |
|---------:|--------:|--------:|-------:|--------------:|-----:|--------------------------:|-----------:|
| 8172 | 0 | 0 | BV | Functionality | 1024 | compute_tables_matrix_add | lookups |
| 0.5 | 0.8 | 0.625 | BV | Functionality | 1024 | compute_tables_matrix_add | hit_ratio* |
Benchmarks that have worsened:
| before | after | ratio | algo | task | n | component | metric |
|---------:|----------:|--------:|-------:|--------------:|-----:|--------------------------:|------------:|
| 0 | 16384 | inf | BV | Functionality | 1024 | compute_tables_matrix_add | num_buckets |
| 4086 | 2000 | 2.043 | BV | Functionality | 1024 | compute_tables_matrix_add | hits* |
| 45 | 53 | 1.178 | BV | Functionality | 1024 | compute_tables_matrix_add | collisions |
| 0.006 | 0.006 | 1.178 | BV | Functionality | 1024 | compute_tables_matrix_add | col_ratio |
Benchmarks that have stayed the same:
| before | after | ratio | algo | task | n | component | metric |
|---------:|---------:|--------:|-------:|--------------:|-----:|--------------------------:|------------------:|
| 4041 | 4033 | 0.998 | BV | Functionality | 1024 | compute_tables_matrix_add | inserts |
| 0.265 | 0.265 | 1 | BV | Functionality | 1024 | | active_memory_mib |
| 0 | 0 | 1 | BV | Functionality | 1024 | compute_tables_matrix_add | num_entries |
| 0.028 | 0.029 | 1.009 | BV | Functionality | 1024 | compute_tables_matrix_add | load_factor |
| 1.125 | nan | nan | BV | Functionality | 1024 | compute_tables_matrix_add | memory_MiB |
| nan | 767 | nan | BV | Functionality | 1024 | compute_tables_matrix_add | peak_num_entries |