Skip to content

BENCHMARKS.md

Latest commit

 

History

History
140 lines (105 loc) · 4.36 KB

BENCHMARKS.md

File metadata and controls

140 lines (105 loc) · 4.36 KB

Beads Performance Benchmarks

This document describes the performance benchmarks available in the beads project and how to use them.

Running Benchmarks

All Dolt Benchmarks

go test -tags=bench -bench=. -benchmem ./internal/storage/dolt/...

Specific Benchmark

go test -tags=bench -bench=BenchmarkGetReadyWork_Large -benchmem ./internal/storage/dolt/...

With CPU Profiling

go test -tags=bench -bench=BenchmarkGetReadyWork_Large -cpuprofile=cpu.prof ./internal/storage/dolt/...
go tool pprof -http=:8080 cpu.prof

Benchmark Categories

Compaction Operations

  • BenchmarkGetTier1Candidates - Identify L1 compaction candidates
  • BenchmarkGetTier2Candidates - Identify L2 compaction candidates
  • BenchmarkCheckEligibility - Check if issue is eligible for compaction

Cycle Detection

Tests on graphs with different topologies (linear chains, trees, dense graphs):

  • BenchmarkCycleDetection_Linear_100/1000/5000 - Linear dependency chains
  • BenchmarkCycleDetection_Tree_100/1000 - Tree-structured dependencies
  • BenchmarkCycleDetection_Dense_100/1000 - Dense graphs

Ready Work / Filtering

  • BenchmarkGetReadyWork_Large - Filter unblocked issues (10K dataset)
  • BenchmarkGetReadyWork_XLarge - Filter unblocked issues (20K dataset)
  • BenchmarkGetReadyWork_FromJSONL - Ready work on imported database

Search Operations

  • BenchmarkSearchIssues_Large_NoFilter - Search all open issues (10K dataset)
  • BenchmarkSearchIssues_Large_ComplexFilter - Search with priority/status filters (10K dataset)

CRUD Operations

  • BenchmarkCreateIssue_Large - Create new issue in 10K database
  • BenchmarkUpdateIssue_Large - Update existing issue in 10K database
  • BenchmarkBulkCloseIssues - Close 100 issues sequentially (NEW)

Specialized Operations

  • BenchmarkLargeDescription - Handling 100KB+ issue descriptions (NEW)
  • BenchmarkSyncMerge - Simulate sync cycle with create/update operations (NEW)

Performance Targets

Typical Results (M2 Pro)

Operation Time Memory Notes
GetReadyWork (10K) 30ms 16.8MB Filters ~200 open issues
Search (10K, no filter) 12.5ms 6.3MB Returns all open issues
Cycle Detection (5000 linear) 70ms 15KB Detects transitive deps
Create Issue (10K db) 2.5ms 8.9KB Insert into index
Update Issue (10K db) 18ms 17KB Status change
Large Description (100KB) 3.3ms 874KB String handling overhead
Bulk Close (100 issues) 1.9s 1.2MB 100 sequential writes
Sync Merge (20 ops) 29ms 198KB Create 10 + update 10

Dataset Caching

Benchmark datasets are cached in /tmp/beads-bench-cache/:

  • large.db - 10,000 issues (16.6 MB)
  • xlarge.db - 20,000 issues (generated on demand)

Cached databases are reused across runs. To regenerate:

rm /tmp/beads-bench-cache/*.db

Adding New Benchmarks

Follow the pattern in sqlite_bench_test.go:

// BenchmarkMyTest benchmarks a specific operation
func BenchmarkMyTest(b *testing.B) {
	runBenchmark(b, setupLargeBenchDB, func(store *SQLiteStorage, ctx context.Context) error {
		// Your test code here
		return err
	})
}

Or for custom setup:

func BenchmarkMyTest(b *testing.B) {
	store, cleanup := setupLargeBenchDB(b)
	defer cleanup()
	ctx := context.Background()

	b.ResetTimer()
	b.ReportAllocs()

	for i := 0; i < b.N; i++ {
		// Your test code here
	}
}

CPU Profiling

The benchmark suite automatically enables CPU profiling on the first benchmark run:

CPU profiling enabled: bench-cpu-2025-12-07-174417.prof
View flamegraph: go tool pprof -http=:8080 bench-cpu-2025-12-07-174417.prof

This generates a flamegraph showing where time is spent across all benchmarks.

Performance Optimization Strategy

  1. Identify bottleneck - Run benchmarks to find slow operations
  2. Profile - Use CPU profiling to see which functions consume time
  3. Measure - Run baseline benchmark before optimization
  4. Optimize - Make targeted changes
  5. Verify - Re-run benchmark to measure improvement

Example:

# Baseline
go test -tags=bench -bench=BenchmarkGetReadyWork_Large -benchmem ./internal/storage/dolt/...

# Make changes...

# Measure improvement
go test -tags=bench -bench=BenchmarkGetReadyWork_Large -benchmem ./internal/storage/dolt/...
⚡