This page documents qsv's benchmarking infrastructure, results tracking, and performance tuning guidelines. It explains how the project measures performance, interprets results, and optimizes the toolkit for diverse data-wrangling workloads using standardized datasets and automated tooling.
For implementation details of specific performance features, see:
qsv uses a standardized benchmarking suite centered around scripts/benchmarks.sh to track performance across versions and hardware platforms. The infrastructure relies on hyperfine for statistical accuracy, utilizing warmup runs and multiple iterations to minimize noise scripts/benchmarks.sh33-40 scripts/benchmarks.sh72-76
The benchmarking process involves downloading a standardized 1M row NYC 311 dataset (approx. 520MB), preparing indices and stats caches, and executing a battery of commands scripts/benchmarks.sh17-23 scripts/benchmarks.sh66-70
Benchmark Data Orchestration
Sources: scripts/benchmarks.sh17-40 scripts/benchmarks.sh54-63 scripts/results/latest_run_info.tsv1-2
The system captures high-fidelity metrics for every command variation. These are stored in a structured format to allow for regression analysis scripts/results/benchmark_results.csv1-2
| Metric | Code Entity / Column | Description |
|---|---|---|
| Throughput | recs_per_sec | Calculated as 1,000,000 / mean. Key for normalization scripts/results/benchmark_results_display.csv1-2 |
| Central Tendency | mean, median | Average and middle execution times in seconds scripts/results/benchmark_results.csv1-2 |
| Variance | stddev | Standard deviation to identify measurement instability scripts/results/benchmark_results.csv1-2 |
| Resource Usage | user, system | CPU time spent in user space vs. kernel space scripts/results/benchmark_results.csv1-2 |
| Bounds | min, max | The fastest and slowest runs recorded scripts/results/benchmark_results.csv1-2 |
scripts/benchmarks.sh)The scripts/benchmarks.sh script is the primary entry point for performance measurement. It is designed to be portable across Unix-like systems and supports targeted benchmarking scripts/benchmarks.sh1-14
The script provides several operational modes through arguments scripts/benchmarks.sh42-126:
./benchmarks.sh sort runs only benchmarks containing the string "sort" scripts/benchmarks.sh4-5./benchmarks.sh setup installs dependencies like hyperfine and 7-Zip scripts/benchmarks.sh12 scripts/benchmarks.sh33-35The script "dogfoods" qsv by using a stable qsv_benchmarker_bin to process the results of the qsv_bin under test scripts/benchmarks.sh54-65 It validates that the benchmarker has necessary features enabled:
apply: For formatting results scripts/benchmarks.sh130-135luau: For aggregating multi-run data scripts/benchmarks.sh137-142to: For Excel spreadsheet generation scripts/benchmarks.sh144-152Sources: scripts/benchmarks.sh42-118 scripts/benchmarks.sh130-152
To achieve the results seen in official benchmarks, qsv should be compiled with CPU-specific optimizations.
target-cpu=native allows the compiler to use modern instructions (AVX2, SIMD) scripts/benchmarks.sh27-28polars, apply, geocode, luau, and to should be enabled for full capability scripts/benchmarks.sh26-28jemalloc or mimalloc to reduce contention scripts/results/run_info_history.tsv2-7The most significant performance gains in qsv come from leveraging its indexing and metadata caching subsystems. Indexing allows for nearly instantaneous record counts and faster random access, while the stats cache allows analytical commands to skip re-calculating univariate statistics.
Performance Gain Visualization
Sources: scripts/results/benchmark_results_display.csv16-18 scripts/results/benchmark_results_display.csv65-68
Based on the latest recorded metrics (Version 21.0.0 on Apple M2 Pro), the following throughputs are typical for core operations scripts/results/latest_results.csv2-50:
| Command Category | Specific Benchmark | Records/Sec |
|---|---|---|
| Instantaneous | count_index | ~90,909,091 scripts/results/latest_results.csv18 |
| High Speed | behead | ~766,871 scripts/results/latest_results.csv10 |
| Transformation | apply_op_similarity | ~530,223 scripts/results/latest_results.csv7 |
| Analysis | frequency_index | ~1,742,160 scripts/results/latest_results.csv68 |
| Complex | excel (conversion) | ~108,413 scripts/results/latest_results.csv37 |
qsv maintains a detailed history of every benchmark run in scripts/results/run_info_history.tsv. This allows developers to correlate performance changes with specific commits or environment changes scripts/results/run_info_history.tsv1-2
Each entry in the history log contains:
platform, cores, mem (total system memory) scripts/results/run_info_history.tsv1version, binary variant, and version_info (features and compiler version) scripts/results/run_info_history.tsv1-2elapsed_secs (total suite time), warmup_runs, and benchmark_runs scripts/results/run_info_history.tsv1qsv_env (captures any performance-impacting environment variables set during the run) scripts/results/run_info_history.tsv1-2Sources: scripts/results/run_info_history.tsv1-21 scripts/results/latest_run_info.tsv1-2
For developers looking to tune specific commands:
hyperfine using the benchmark dataset scripts/benchmarks.sh72-76samply or perf) to generate flamegraphs and identify hotspots.mimalloc or jemalloc) or increasing parallelism via Rayon../benchmarks.sh <command_name> and compare the mean and recs_per_sec against the historical values in scripts/results/benchmark_results.csv scripts/results/benchmark_results.csv1-10Sources: scripts/benchmarks.sh1-40 scripts/results/benchmark_results.csv1-10
Refresh this wiki
This wiki was recently refreshed. Please wait 7 days to refresh again.