Docs

Author: matthewmturner

CLAUDE.md

@@ -33,6 +33,55 @@ cargo run --features=flightsql -- serve-flightsql
 cargo run -- generate-tpch
 ```
 
+### Benchmarking
+
+Benchmarks measure query performance with detailed timing breakdowns:
+
+```bash
+# Serial benchmark (default, 10 iterations)
+cargo run -- -c "SELECT 1" --bench
+
+# Custom iteration count
+cargo run -- -c "SELECT 1" --bench -n 100
+
+# Concurrent benchmark (measures throughput under load)
+cargo run -- -c "SELECT 1" --bench --concurrent
+
+# With custom iterations and concurrency
+cargo run -- -c "SELECT 1" --bench -n 100 --concurrent
+
+# Save results to CSV
+cargo run -- -c "SELECT 1" --bench --save results.csv
+
+# Append to existing results
+cargo run -- -c "SELECT 2" --bench --concurrent --save results.csv --append
+
+# Warm up cache before benchmarking
+cargo run -- -c "SELECT * FROM t" --bench --run-before "CREATE TABLE t AS VALUES (1)"
+```
+
+**Benchmark Modes:**
+- **Serial** (default): Measures query performance in isolation
+  - Shows pure query execution time without contention
+  - Ideal for understanding baseline performance
+
+- **Concurrent** (`--concurrent`): Measures performance under load
+  - Runs iterations in parallel (concurrency = min(iterations, CPU cores))
+  - Shows throughput (queries/second) with multiple clients
+  - Reveals resource contention and bottlenecks
+  - Higher mean/median times are expected due to concurrent load
+
+**Output:**
+- Timing breakdown: logical planning, physical planning, execution, total
+- Statistics: min, max, mean, median for each phase
+- CSV format includes `concurrency_mode` column (serial or concurrent(N))
+
+**FlightSQL Benchmarks:**
+```bash
+# Benchmark FlightSQL server (requires --flightsql flag and server running)
+cargo run -- -c "SELECT 1" --bench --flightsql --concurrent
+```
+
 ### Testing
 
 Tests are organized by feature and component:

README.md

@@ -68,6 +68,12 @@ dft -f query.sql
 # Benchmark a query (with stats)
 dft -c "SELECT * FROM my_table" --bench
 
+# Concurrent benchmark (measures throughput under load)
+dft -c "SELECT * FROM my_table" --bench --concurrent
+
+# Save benchmark results to CSV
+dft -c "SELECT * FROM my_table" --bench --save results.csv
+
 # Start FlightSQL Server (requires `flightsql` feature)
 dft serve-flightsql
 
@@ -78,6 +84,39 @@ dft serve-http
 dft generate-tpch
 ```
 
+### Benchmarking
+
+`dft` includes built-in benchmarking to measure query performance with detailed timing breakdowns:
+
+```sh
+# Serial benchmark (default) - measures query performance in isolation
+dft -c "SELECT * FROM my_table" --bench
+
+# Concurrent benchmark - measures throughput under load
+dft -c "SELECT * FROM my_table" --bench --concurrent
+
+# Custom iteration count
+dft -c "SELECT * FROM my_table" --bench -n 100
+
+# Save results to CSV for analysis
+dft -c "SELECT * FROM my_table" --bench --save results.csv
+
+# Compare serial vs concurrent performance
+dft -c "SELECT * FROM my_table" --bench --save results.csv
+dft -c "SELECT * FROM my_table" --bench --concurrent --save results.csv --append
+```
+
+**Benchmark Output:**
+- Timing breakdown by phase: logical planning, physical planning, execution
+- Statistics: min, max, mean, median for each phase
+- Row counts validation across all runs
+- CSV export with `concurrency_mode` column for result comparison
+
+**Serial vs Concurrent:**
+- **Serial**: Pure query execution time without contention (baseline performance)
+- **Concurrent**: Throughput measurement with parallel execution (reveals bottlenecks and contention)
+- Concurrent mode uses adaptive concurrency: `min(iterations, CPU cores)`
+
 ### Setting Up Tables with DDL
 
 `dft` can automatically load table definitions at startup, giving you a persistent "database-like" experience.

docs/cli.md

@@ -64,31 +64,69 @@ basic_auth.password = "Pass"
 
 ## Benchmark Queries
 
-You can benchmark queries by adding the `--bench` parameter.  This will run the query a configurable number of times and output a breakdown of the queries execution time with summary statistics for each component of the query (logical planning, physical planning, execution time, and total time).
+You can benchmark queries by adding the `--bench` parameter. This will run the query a configurable number of times and output a breakdown of the query's execution time with summary statistics for each component (logical planning, physical planning, execution time, and total time).
 
-Optionally you can use the `--run-before` param to run a query before the benchmark is run.  This is useful in cases where you want to hit a temp table or write a file to disk that your benchmark query will use.
+### Benchmark Modes
 
-To save benchmark results to a file use the `--save` parameter with a file path.  Further, you can use the `--append` parameter to append to the file instead of overwriting it.
+**Serial Benchmark (default):**
+Measures query performance in isolation, running iterations one after another. This shows the pure query execution time without any contention or resource sharing overhead.
 
-The number of benchmark iterations is defined in your configuration (default is 10) and can be configured per benchmark run with `-n` parameter.
+**Concurrent Benchmark (`--concurrent`):**
+Measures query performance under load by running iterations in parallel. This reveals:
+- Throughput (queries per second) with multiple concurrent clients
+- Resource contention and bottlenecks
+- Performance degradation under concurrent load
 
+Concurrent mode uses adaptive concurrency: `min(iterations, CPU cores)` to avoid overwhelming the system.
+
+### Options
+
+- **`--bench`**: Enable benchmarking mode
+- **`--concurrent`**: Run iterations in parallel (for concurrent benchmarking)
+- **`-n <count>`**: Number of iterations (default: 10, configured in config file)
+- **`--run-before <query>`**: Run a setup query before benchmarking (useful for cache warming)
+- **`--save <file>`**: Save results to CSV file
+- **`--append`**: Append to existing results file instead of overwriting
+
+### Examples
 
 ```sh
+# Serial benchmark (default)
 dft -c "SELECT * FROM my_table" --bench
 
-# Run a configurable number of benchmark iterations
-dft -c "SELECT ..." --bench -n 5
+# Concurrent benchmark
+dft -c "SELECT * FROM my_table" --bench --concurrent
+
+# Custom iteration count
+dft -c "SELECT ..." --bench -n 100
+
+# Concurrent with custom iterations
+dft -c "SELECT ..." --bench -n 100 --concurrent
 
-# Save benchmark results to a file
+# Save benchmark results to CSV
 dft -c "SELECT ..." --bench --save results.csv
 
-# Append benchmark results to existing file
-dft -c "SELECT ..." --bench --save results.csv --append
+# Append results (compare serial vs concurrent)
+dft -c "SELECT ..." --bench --save results.csv
+dft -c "SELECT ..." --bench --concurrent --save results.csv --append
 
-# Run a setup query prior to running benchmark.  This can be useful to quickly iterate on various paramters
+# Run a setup query before benchmarking
 dft -c "SELECT ..." --bench --run-before="CREATE TEMP TABLE my_temp AS SELECT ..."
+
+# FlightSQL benchmark (concurrent)
+dft -c "SELECT ..." --bench --concurrent --flightsql
 ```
 
+### Output
+
+Benchmark output includes:
+- **Mode**: `serial` or `concurrent(N)` where N is the concurrency level
+- **Timing breakdown**: Logical planning, physical planning, execution (min/max/mean/median)
+- **Row counts**: Validation that all runs returned the same number of rows
+- **CSV format**: Results include a `concurrency_mode` column for comparison
+
+**Note**: Concurrent benchmarks typically show higher mean/median times due to resource contention - this is expected and reveals how the system performs under load.
+
 ## Analyze Queries
 
 The output from `EXPLAIN ANALYZE` provides a wealth of information on a queries execution - however, the amount of information and connecting the dots can be difficult and manual.  Further, there is detail in the `MetricSet`'s of the underlying `ExecutionPlan`'s that is lost in the output.