Add autotuner for benchmark parameter optimization with path-based configuration

[Copilot]

The benchmark framework measures QPS/latency/recall for given configurations but lacks automated parameter tuning. This adds a tool to sweep parameters and identify optimal configurations.

Implementation

New binary: diskann-tools/bin/autotuner

• Path-based configuration system that doesn't hardcode JSON structure, making it robust to benchmark framework changes
• Users specify parameters via JSON paths (e.g., "jobs.0.content.source.max_degree")
• Generates all parameter combinations, runs benchmarks, parses results
• Three optimization criteria: maximize QPS at target recall, minimize latency at target recall, maximize recall
• Outputs individual configs/results plus summary with best configuration

CLI commands:

# Generate example sweep config (supports --benchmark-type pq/disk for specific formats)
autotuner example --output sweep_config.json

# Run sweep
autotuner sweep \
  --base-config benchmark.json \
  --sweep-config sweep_config.json \
  --output-dir ./results \
  --criterion qps \
  --target-recall 0.95

Path-based configuration format:

{
  "parameters": [
    {"path": "jobs.0.content.source.max_degree", "values": [16, 32, 64]},
    {"path": "jobs.0.content.source.l_build", "values": [50, 75, 100]},
    {"path": "jobs.0.content.search_phase.runs.0.search_l", "values": [[10, 20, 30], [20, 40, 60]]}
  ]
}

Design benefits:

• Framework-agnostic: works with any benchmark JSON structure
• Future-proof: survives benchmark framework refactoring
• Flexible: can sweep over any parameter by specifying its JSON path
• Maintainable: no code changes needed when benchmark schema evolves

Changes

• Add diskann-tools/src/bin/autotuner.rs - main implementation with path-based logic and 6 tests
• Add diskann-tools/AUTOTUNER.md - comprehensive documentation of path-based approach
• Update diskann-benchmark/README.md - reference autotuner
• Update diskann-tools/Cargo.toml - add serde_json dependency

Addressing Review Feedback

The initial implementation used hardcoded JSON paths which would break if the benchmark framework changed. The tool has been refactored to use a path-based configuration system where users specify JSON paths in their sweep configuration. This makes the tool robust to changes in the benchmark framework - if the schema changes, users only need to update their sweep config, not the tool itself.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Create an autotuner tool on top of benchmark</issue_title>
<issue_description>## Is your feature request related to a problem? Please describe.
the repo contains a comprehensive benchmark framework that takes configuration as inputs and measures throughput (qps), latency etc. We want a tool that builds on this to sweep over a subset of parameters, for example R (graph degree), l_build, l_search, and quantization bytes where applicable, and identify the best configuration.

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Provide references (if applicable)

If your feature request is related to a published algorithm/idea, please provide links to
any relevant articles or webpages.

Additional context

Add any other context or screenshots about the feature request here.

</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Create an autotuner tool on top of benchmark #735

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[harsha-simhadri]

@copilot this design is not robust to changes to the benchmark framework. If the benchmark framework and params are changed, the approach in this PR can be stale and eventually flaky. Is there a different strategy that would help this stay in sync with the benchmark framework

[Copilot]

@copilot this design is not robust to changes to the benchmark framework. If the benchmark framework and params are changed, the approach in this PR can be stale and eventually flaky. Is there a diffe...

You're absolutely right. The current implementation hardcodes JSON paths like job.get_mut("content").get_mut("source") which is brittle.

I'll refactor to use a JSONPath-based override mechanism that:

1. Takes parameter overrides as path-value pairs (e.g., "$.jobs[*].content.source.max_degree": [16, 32])
2. Doesn't hardcode any JSON structure assumptions
3. Stays in sync with benchmark framework changes automatically

Working on the refactor now.

@copilot this design is not robust to changes to the benchmark framework. If the benchmark framework and params are changed, the approach in this PR can be stale and eventually flaky. Is there a diffe...

Refactored to use a path-based configuration system in commit ab17601.

The new design addresses your concern by:

• Using JSON paths instead of hardcoded structure (e.g., "jobs.0.content.source.max_degree")
• Providing generic set_json_path() that works with any JSON structure
• Making users specify what to sweep via paths in their config
• Staying in sync automatically - if benchmark schema changes, only the user's sweep config needs updating

Example sweep configuration:

{
  "parameters": [
    {"path": "jobs.0.content.source.max_degree", "values": [16, 32, 64]},
    {"path": "jobs.0.content.source.l_build", "values": [50, 75, 100]}
  ]
}

See updated AUTOTUNER.md for full documentation of the path-based approach.