Benchmarking the DD Package¶
MQT Core provides a benchmark suite to evaluate the performance of the DD package. This can be especially helpful if you are working on the DD package and want to know how your changes affect its performance.
Generating results¶
In order to generate benchmark data, MQT Core provides the mqt-core-dd-eval CMake target (which is made available by passing -DBUILD_MQT_CORE_BENCHMARKS=ON to CMake). This target will run the benchmark suite and generate a JSON file containing the results. To this end, the target takes a single argument which is used as a component in the resulting JSON filename.
After running the target, you will see a results_<your_argument>.json file in your build directory that contains all the data collected during the benchmarking process. An exemplary results_<your_argument>.json file might look like this (taken from the eval directory):
1import json
2from pathlib import Path
3
4filepath = Path("../eval/results_baseline.json")
5
6with filepath.open(mode="r", encoding="utf-8") as f:
7 data = json.load(f)
8 json_formatted_str = json.dumps(data, indent=2)
9 print(json_formatted_str)
{
"BV": {
"Functionality": {
"1024": {
"dd": {
"active_memory_mib": 0.26538848876953125,
"compute_tables": {
"matrix_add": {
"col_ratio": 0.005506607929515419,
"collisions": 45,
"hit_ratio": 0.5,
"hits": 4086,
"inserts": 4041,
"load_factor": 0.0283203125,
"lookups": 8172,
"memory_MiB": 1.125,
"num_buckets": 0,
"num_entries": 0,
"peak_num_entries": "unused"
}
}
},
"runtime": 1.2
},
"256": {
"dd": {
"active_memory_mib": 0.06616973876953125,
"compute_tables": {
"matrix_add": {
"col_ratio": 0.008893280632411068,
"collisions": 18
}
},
"matrix": {
"unique_table": {
"total": {
"col_ratio": 0.00514176735528262,
"collisions": 2475
}
}
}
},
"runtime": 0.440665166
}
},
"Simulation": {
"256": {
"dd": {
"active_memory_mib": 0.021484375,
"matrix": {
"unique_table": {
"total": {
"col_ratio": 0.0029885811069223626,
"collisions": 1094
}
}
}
}
}
}
},
"GHZ": {
"Functionality": {
"512": {
"dd": {
"compute_tables": {
"matrix_add": {
"col_ratio": 0.011530912659470068,
"collisions": 47
}
}
},
"runtime": 0.2
}
}
}
}
To compare the performance of your newly proposed changes to the existing implementation, the benchmark script should be executed once based on the branch/commit you want to compare against and once in your new feature branch. Make sure to pass different arguments as different file names while running the target (e.g. baseline and feature).
Running the comparison¶
The MQT Core source code contains a script that can be used to compare the results of two runs of the benchmark suite.
The script is located in eval/dd_evaluation.py.
It uses PEP 723 inline script metadata to specify the script’s dependencies.
The comparison can be run from the command line via:
1! ../eval/dd_evaluation.py ../eval/results_baseline.json ../eval/results_feature.json --factor=0.2 --only_changed
Runtime:
Benchmarks that have improved:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|-----:|
| 1.2 | 0.579 | 0.482 | BV | Functionality | 1024 |
Benchmarks that have worsened:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|----:|
| 0.2 | 0.428 | 2.142 | GHZ | Functionality | 512 |
1! ../eval/dd_evaluation.py ../eval/results_baseline.json ../eval/results_feature.json --no_split --dd --task=functionality
Runtime:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|-----:|
| 1.2 | 0.579 | 0.482 | BV | Functionality | 1024 |
| 0.441 | 0.428 | 0.972 | BV | Functionality | 256 |
| 0.2 | 0.428 | 2.142 | GHZ | Functionality | 512 |
DD Package details:
| before | after | ratio | algo | task | n | component | metric |
|---------:|----------:|--------:|-------:|--------------:|-----:|--------------------------:|------------------:|
| 8172 | 0 | 0 | BV | Functionality | 1024 | compute_tables_matrix_add | lookups |
| 0.009 | 0.005 | 0.556 | BV | Functionality | 256 | compute_tables_matrix_add | col_ratio |
| 18 | 10 | 0.556 | BV | Functionality | 256 | compute_tables_matrix_add | collisions |
| 0.5 | 0.8 | 0.625 | BV | Functionality | 1024 | compute_tables_matrix_add | hit_ratio* |
| 2475 | 1873 | 0.757 | BV | Functionality | 256 | matrix_unique_table | total_collisions |
| 0.005 | 0.004 | 0.757 | BV | Functionality | 256 | matrix_unique_table | total_col_ratio |
| 0.012 | 0.009 | 0.787 | GHZ | Functionality | 512 | compute_tables_matrix_add | col_ratio |
| 47 | 37 | 0.787 | GHZ | Functionality | 512 | compute_tables_matrix_add | collisions |
| 4041 | 4033 | 0.998 | BV | Functionality | 1024 | compute_tables_matrix_add | inserts |
| 0.265 | 0.265 | 1 | BV | Functionality | 1024 | | active_memory_mib |
| 0.066 | 0.066 | 1 | BV | Functionality | 256 | | active_memory_mib |
| 0 | 0 | 1 | BV | Functionality | 1024 | compute_tables_matrix_add | num_entries |
| 0.028 | 0.029 | 1.009 | BV | Functionality | 1024 | compute_tables_matrix_add | load_factor |
| 0.006 | 0.006 | 1.178 | BV | Functionality | 1024 | compute_tables_matrix_add | col_ratio |
| 45 | 53 | 1.178 | BV | Functionality | 1024 | compute_tables_matrix_add | collisions |
| 4086 | 2000 | 2.043 | BV | Functionality | 1024 | compute_tables_matrix_add | hits* |
| 0 | 16384 | inf | BV | Functionality | 1024 | compute_tables_matrix_add | num_buckets |
| 1.125 | nan | nan | BV | Functionality | 1024 | compute_tables_matrix_add | memory_MiB |
| nan | 767 | nan | BV | Functionality | 1024 | compute_tables_matrix_add | peak_num_entries |
| nan | 16384 | nan | BV | Functionality | 256 | compute_tables_matrix_add | num_buckets |
1! ../eval/dd_evaluation.py ../eval/results_baseline.json ../eval/results_feature.json --dd --algorithm=bv --num_qubits=1024
Runtime:
Benchmarks that have improved:
| before | after | ratio | algo | task | n |
|---------:|--------:|--------:|-------:|--------------:|-----:|
| 1.2 | 0.579 | 0.482 | BV | Functionality | 1024 |
Benchmarks that have worsened:
| before | after | ratio | algo | task | n |
|----------|---------|---------|--------|--------|-----|
Benchmarks that have stayed the same:
| before | after | ratio | algo | task | n |
|----------|---------|---------|--------|--------|-----|
DD Package details:
Benchmarks that have improved:
| before | after | ratio | algo | task | n | component | metric |
|---------:|--------:|--------:|-------:|--------------:|-----:|--------------------------:|-----------:|
| 8172 | 0 | 0 | BV | Functionality | 1024 | compute_tables_matrix_add | lookups |
| 0.5 | 0.8 | 0.625 | BV | Functionality | 1024 | compute_tables_matrix_add | hit_ratio* |
Benchmarks that have worsened:
| before | after | ratio | algo | task | n | component | metric |
|---------:|----------:|--------:|-------:|--------------:|-----:|--------------------------:|------------:|
| 0 | 16384 | inf | BV | Functionality | 1024 | compute_tables_matrix_add | num_buckets |
| 4086 | 2000 | 2.043 | BV | Functionality | 1024 | compute_tables_matrix_add | hits* |
| 45 | 53 | 1.178 | BV | Functionality | 1024 | compute_tables_matrix_add | collisions |
| 0.006 | 0.006 | 1.178 | BV | Functionality | 1024 | compute_tables_matrix_add | col_ratio |
Benchmarks that have stayed the same:
| before | after | ratio | algo | task | n | component | metric |
|---------:|---------:|--------:|-------:|--------------:|-----:|--------------------------:|------------------:|
| 4041 | 4033 | 0.998 | BV | Functionality | 1024 | compute_tables_matrix_add | inserts |
| 0.265 | 0.265 | 1 | BV | Functionality | 1024 | | active_memory_mib |
| 0 | 0 | 1 | BV | Functionality | 1024 | compute_tables_matrix_add | num_entries |
| 0.028 | 0.029 | 1.009 | BV | Functionality | 1024 | compute_tables_matrix_add | load_factor |
| 1.125 | nan | nan | BV | Functionality | 1024 | compute_tables_matrix_add | memory_MiB |
| nan | 767 | nan | BV | Functionality | 1024 | compute_tables_matrix_add | peak_num_entries |
This internally runs uv run --script --quiet on the respective benchmark script (including automatically installing any missing dependencies).