Skip to content

fix: Remove evaluation metric key from schema which failed on some LLMs#105

Open
jsonbailey wants to merge 3 commits intomainfrom
jb/aic-1897/remove-keys-from-evaluation-structure
Open

fix: Remove evaluation metric key from schema which failed on some LLMs#105
jsonbailey wants to merge 3 commits intomainfrom
jb/aic-1897/remove-keys-from-evaluation-structure

Conversation

@jsonbailey
Copy link
Contributor

@jsonbailey jsonbailey commented Mar 11, 2026

Note

Medium Risk
Changes the structured output schema and parsing contract for judge evaluations, which can affect compatibility with existing providers/models and downstream consumers of the judge response shape. Test updates reduce risk but runtime behavior may differ if any integrations still return the old evaluations object.

Overview
Updates judge structured-output handling to use a fixed schema { "evaluation": { "score", "reasoning" } } rather than generating a dynamic evaluations object keyed by evaluation_metric_key.

Judge now always builds the schema (no key parameter), parses only the single evaluation payload, and marks evaluations as failed when that object is missing/invalid; tests are updated accordingly and dynamic schema helper methods are removed.

Written by Cursor Bugbot for commit 916df2a. This will update automatically on new commits. Configure here.

@jsonbailey jsonbailey requested a review from a team as a code owner March 11, 2026 22:40
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant