Benchmarking the DD Package¶
MQT Core provides a benchmark suite to evaluate the performance of the DD package. This can be especially helpful if you are working on the DD package and want to know how your changes affect its performance.
Generating results¶
In order to generate benchmark data, MQT Core provides the mqt-core-dd-eval
CMake target (which is made available by passing
-DBUILD_MQT_CORE_BENCHMARKS=ON to CMake). This target will run the benchmark
suite and generate a JSON file containing the results. To this end, the target
takes a single argument which is used as a component in the resulting JSON
filename.
After running the target, you will see a results_<your_argument>.json file in
your build directory that contains all the data collected during the
benchmarking process. An exemplary results_<your_argument>.json file might
look like this (taken from the eval directory):
1import json
2from pathlib import Path
3
4filepath = Path("../eval/results_baseline.json")
5
6with filepath.open(mode="r", encoding="utf-8") as f:
7 data = json.load(f)
8 json_formatted_str = json.dumps(data, indent=2)
9 print(json_formatted_str)
{
"BV": {
"Functionality": {
"1024": {
"dd": {
"active_memory_mib": 0.26538848876953125,
"compute_tables": {
"matrix_add": {
"col_ratio": 0.005506607929515419,
"collisions": 45,
"hit_ratio": 0.5,
"hits": 4086,
"inserts": 4041,
"load_factor": 0.0283203125,
"lookups": 8172,
"memory_MiB": 1.125,
"num_buckets": 0,
"num_entries": 0,
"peak_num_entries": "unused"
}
}
},
"runtime": 1.2
},
"256": {
"dd": {
"active_memory_mib": 0.06616973876953125,
"compute_tables": {
"matrix_add": {
"col_ratio": 0.008893280632411068,
"collisions": 18
}
},
"matrix": {
"unique_table": {
"total": {
"col_ratio": 0.00514176735528262,
"collisions": 2475
}
}
}
},
"runtime": 0.440665166
}
},
"Simulation": {
"256": {
"dd": {
"active_memory_mib": 0.021484375,
"matrix": {
"unique_table": {
"total": {
"col_ratio": 0.0029885811069223626,
"collisions": 1094
}
}
}
}
}
}
},
"GHZ": {
"Functionality": {
"512": {
"dd": {
"compute_tables": {
"matrix_add": {
"col_ratio": 0.011530912659470068,
"collisions": 47
}
}
},
"runtime": 0.2
}
}
}
}
To compare the performance of your newly proposed changes to the existing
implementation, the benchmark script should be executed once based on the
branch/commit you want to compare against and once in your new feature branch.
Make sure to pass different arguments as different file names while running the
target (e.g. baseline and feature).
Running the comparison¶
The MQT Core source code contains a script that can be used to compare the
results of two runs of the benchmark suite. The script is located in
eval/dd_evaluation.py. It uses PEP 723 inline script metadata to specify the
script’s dependencies.
The comparison can be run from the command line via:
1! ../eval/dd_evaluation.py ../eval/results_baseline.json ../eval/results_feature.json --factor=0.2 --only_changed
Runtime:
Benchmarks that have improved:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|-----:|
| 1.2 | 0.579 | 0.482 | BV | Functionality | 1024 |
Benchmarks that have worsened:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|----:|
| 0.2 | 0.428 | 2.142 | GHZ | Functionality | 512 |
1! ../eval/dd_evaluation.py ../eval/results_baseline.json ../eval/results_feature.json --no_split --dd --task=functionality
Runtime:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|-----:|
| 1.2 | 0.579 | 0.482 | BV | Functionality | 1024 |
| 0.441 | 0.428 | 0.972 | BV | Functionality | 256 |
| 0.2 | 0.428 | 2.142 | GHZ | Functionality | 512 |
DD Package details:
| before | after | ratio | algo | task | n | component | metric |
|---------:|----------:|--------:|-------:|--------------:|-----:|--------------------------:|------------------:|
| 8172 | 0 | 0 | BV | Functionality | 1024 | compute_tables_matrix_add | lookups |
| 0.009 | 0.005 | 0.556 | BV | Functionality | 256 | compute_tables_matrix_add | col_ratio |
| 18 | 10 | 0.556 | BV | Functionality | 256 | compute_tables_matrix_add | collisions |
| 0.5 | 0.8 | 0.625 | BV | Functionality | 1024 | compute_tables_matrix_add | hit_ratio* |
| 2475 | 1873 | 0.757 | BV | Functionality | 256 | matrix_unique_table | total_collisions |
| 0.005 | 0.004 | 0.757 | BV | Functionality | 256 | matrix_unique_table | total_col_ratio |
| 0.012 | 0.009 | 0.787 | GHZ | Functionality | 512 | compute_tables_matrix_add | col_ratio |
| 47 | 37 | 0.787 | GHZ | Functionality | 512 | compute_tables_matrix_add | collisions |
| 4041 | 4033 | 0.998 | BV | Functionality | 1024 | compute_tables_matrix_add | inserts |
| 0.265 | 0.265 | 1 | BV | Functionality | 1024 | | active_memory_mib |
| 0.066 | 0.066 | 1 | BV | Functionality | 256 | | active_memory_mib |
| 0 | 0 | 1 | BV | Functionality | 1024 | compute_tables_matrix_add | num_entries |
| 0.028 | 0.029 | 1.009 | BV | Functionality | 1024 | compute_tables_matrix_add | load_factor |
| 0.006 | 0.006 | 1.178 | BV | Functionality | 1024 | compute_tables_matrix_add | col_ratio |
| 45 | 53 | 1.178 | BV | Functionality | 1024 | compute_tables_matrix_add | collisions |
| 4086 | 2000 | 2.043 | BV | Functionality | 1024 | compute_tables_matrix_add | hits* |
| 0 | 16384 | inf | BV | Functionality | 1024 | compute_tables_matrix_add | num_buckets |
| 1.125 | nan | nan | BV | Functionality | 1024 | compute_tables_matrix_add | memory_MiB |
| nan | 767 | nan | BV | Functionality | 1024 | compute_tables_matrix_add | peak_num_entries |
| nan | 16384 | nan | BV | Functionality | 256 | compute_tables_matrix_add | num_buckets |
1! ../eval/dd_evaluation.py ../eval/results_baseline.json ../eval/results_feature.json --dd --algorithm=bv --num_qubits=1024
Runtime:
Benchmarks that have improved:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|-----:|
| 1.2 | 0.579 | 0.482 | BV | Functionality | 1024 |
Benchmarks that have worsened:
| before | after | ratio | algo | task | n |
|----------|---------|---------|--------|--------|-----|
Benchmarks that have stayed the same:
| before | after | ratio | algo | task | n |
|----------|---------|---------|--------|--------|-----|
DD Package details:
Benchmarks that have improved:
| before | after | ratio | algo | task | n | component | metric |
|---------:|--------:|--------:|-------:|--------------:|-----:|--------------------------:|-----------:|
| 8172 | 0 | 0 | BV | Functionality | 1024 | compute_tables_matrix_add | lookups |
| 0.5 | 0.8 | 0.625 | BV | Functionality | 1024 | compute_tables_matrix_add | hit_ratio* |
Benchmarks that have worsened:
| before | after | ratio | algo | task | n | component | metric |
|---------:|----------:|--------:|-------:|--------------:|-----:|--------------------------:|------------:|
| 0 | 16384 | inf | BV | Functionality | 1024 | compute_tables_matrix_add | num_buckets |
| 4086 | 2000 | 2.043 | BV | Functionality | 1024 | compute_tables_matrix_add | hits* |
| 45 | 53 | 1.178 | BV | Functionality | 1024 | compute_tables_matrix_add | collisions |
| 0.006 | 0.006 | 1.178 | BV | Functionality | 1024 | compute_tables_matrix_add | col_ratio |
Benchmarks that have stayed the same:
| before | after | ratio | algo | task | n | component | metric |
|---------:|---------:|--------:|-------:|--------------:|-----:|--------------------------:|------------------:|
| 4041 | 4033 | 0.998 | BV | Functionality | 1024 | compute_tables_matrix_add | inserts |
| 0.265 | 0.265 | 1 | BV | Functionality | 1024 | | active_memory_mib |
| 0 | 0 | 1 | BV | Functionality | 1024 | compute_tables_matrix_add | num_entries |
| 0.028 | 0.029 | 1.009 | BV | Functionality | 1024 | compute_tables_matrix_add | load_factor |
| 1.125 | nan | nan | BV | Functionality | 1024 | compute_tables_matrix_add | memory_MiB |
| nan | 767 | nan | BV | Functionality | 1024 | compute_tables_matrix_add | peak_num_entries |
This internally runs uv run --script --quiet on the respective benchmark
script (including automatically installing any missing dependencies).