[Evaluation] Recover partial red team results when Foundry execution raises#45541
[Evaluation] Recover partial red team results when Foundry execution raises#45541slister1001 wants to merge 1 commit intoAzure:mainfrom
Conversation
…raises When orchestrator.execute() raises (e.g., ConnectTimeout on 1 of 50 objectives), attempt to recover partial results from the orchestrator before falling back to the empty-result error path. Previously, any single objective failure caused the entire risk category's results to be discarded (data_file set to empty string, 0 results returned). Now, completed objectives are processed through the normal FoundryResultProcessor pipeline and included in the final output. The error is demoted from ERROR to WARNING when partial results are available, since it is not a total failure. The original full-failure path is preserved when get_attack_results() returns empty. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Improve resilience of red team evaluation runs by attempting to recover and process partial Foundry attack results when orchestrator.execute() raises, instead of discarding the entire risk category.
Changes:
- On
orchestrator.execute()exception, triesorchestrator.get_attack_results()to recover partial results. - Downgrades logging from ERROR to WARNING when partial results are recovered.
- Preserves existing empty-result fallback behavior when no results can be recovered.
| # Attempt to recover partial results before giving up | ||
| partial_results = [] | ||
| try: | ||
| partial_results = orchestrator.get_attack_results() | ||
| except Exception: | ||
| pass | ||
|
|
||
| if partial_results: | ||
| self.logger.warning( | ||
| f"Partial failure executing attacks for {risk_value}: {e}. " | ||
| f"Recovered {len(partial_results)} partial results." | ||
| ) |
There was a problem hiding this comment.
partial_results is computed but not used to drive downstream processing. If FoundryResultProcessor later calls orchestrator.get_attack_results() again (or expects state set by a successful execute()), the recovered results may be lost and the post-exception path could still behave like a full failure. Consider wiring the recovered partial_results into the processing path (e.g., pass into the processor or stash on the orchestrator) so the recovery is deterministic.
| try: | ||
| partial_results = orchestrator.get_attack_results() | ||
| except Exception: | ||
| pass |
There was a problem hiding this comment.
Swallowing all exceptions from get_attack_results() makes diagnosing recovery failures difficult. Consider at least logging at DEBUG level (optionally with exc_info=True) when partial-results recovery fails, so operators can distinguish 'no partial results' from 'recovery call failed'.
| pass | |
| self.logger.debug( | |
| "Failed to recover partial attack results for %s", | |
| risk_value, | |
| exc_info=True, | |
| ) |
| self.logger.warning( | ||
| f"Partial failure executing attacks for {risk_value}: {e}. " | ||
| f"Recovered {len(partial_results)} partial results." | ||
| ) |
There was a problem hiding this comment.
In the partial-results path, the failure is only surfaced via logs and not recorded in the structured red_team_info data (unlike the full-failure fallback). If callers rely on structured output rather than logs, consider persisting a 'partial failure' indicator and the exception message alongside the recovered results so the outcome is observable without log access.
| ) | |
| ) | |
| # Surface partial failure in structured red_team_info | |
| if "Foundry" not in red_team_info: | |
| red_team_info["Foundry"] = {} | |
| existing_info = red_team_info["Foundry"].get(risk_value, {}) | |
| # Do not discard any existing structured data for this risk value | |
| existing_info.setdefault("status", "partial_failure") | |
| existing_info["partial_failure"] = True | |
| existing_info["error"] = str(e) | |
| red_team_info["Foundry"][risk_value] = existing_info |
github-actions
bot
added
the
Evaluation
When orchestrator.execute() raises (e.g., ConnectTimeout on 1 of 50 objectives), attempt to recover partial results from the orchestrator before falling back to the empty-result error path.
Previously, any single objective failure caused the entire risk category's results to be discarded (data_file set to empty string, 0 results returned). Now, completed objectives are processed through the normal FoundryResultProcessor pipeline and included in the final output.
The error is demoted from ERROR to WARNING when partial results are available, since it is not a total failure. The original full-failure path is preserved when get_attack_results() returns empty.
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines