Python API no longer retries requests which may have temporarily failed (e.g. 502)

[alahiff]

When metrics are sent a single POST to /api/metrics with status 502 results in an error visibile to users:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/home/alahiff/client/20260226/venv/lib/python3.11/site-packages/simvue/dispatch/queued.py", line 172, in run
    self._callback(_buffer, queue_label)
  File "/home/alahiff/client/20260226/venv/lib/python3.11/site-packages/simvue/run.py", line 501, in _dispatch_callback
    return _metrics.commit()
           ^^^^^^^^^^^^^^^^^
  File "/home/alahiff/client/20260226/venv/lib/python3.11/site-packages/simvue/api/objects/base.py", line 560, in commit
    _response = self._post_single(**self._staging)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alahiff/client/20260226/venv/lib/python3.11/site-packages/simvue/api/objects/metrics.py", line 143, in _post_single
    return super()._post_single(is_json=False, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alahiff/client/20260226/venv/lib/python3.11/site-packages/simvue/api/objects/base.py", line 652, in _post_single
    _json_response = get_json_from_response(
                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alahiff/client/20260226/venv/lib/python3.11/site-packages/simvue/api/request.py", line 277, in get_json_from_response
    raise RuntimeError(error_str)
RuntimeError: Creation of metric failed for url 'http://localhost:8000/api/metrics' with status 502

Errors such as this should definitely be retried.

[alahiff]

http.HTTPStatus.BAD_GATEWAY (502) is missing from RETRY_STATUS_CODES, but adding it has no effect anyway. As far as I can tell Tenacity is doing nothing.

[alahiff]

Note that fixing Tenacity doesn't help (e.g. using retry_if_result), as we then get this error before any of the retries:

RuntimeError: Cannot append item '{'values': {'resources/cpu.usage.percentage': 1.0, 'resources/cpu.usage.memory': 116.8828125}, 'time': 1.0539274215698242, 'timestamp': '2026-02-26T14:28:53.348861', 'step': 0}' to queue 'metrics_regular', termination called.

Even just adding time.sleep(5) to the beginning of the post function results in an error like this.