-
Notifications
You must be signed in to change notification settings - Fork 3
Description
Hello,
We have identified an issue with CloudbeesProvider that can lead to a non-recoverable state in production.
Issue description
When we initialize a CloudbeesProvider, and we provide a timeout value, the Rox.setup can timeout. When that happens, a TimeoutError is raised. But the Rox.setup set the state of Rox to Rox.State = RoxState.Set
Trying to initialize a new CloudbeesProvider (we never got the first instance because it raised) will result in another exception: AttributeError
This is because the Rox.setup will return early because of this:
def setup(api_key, rox_options=None):
with Rox.startupShutdownLock:
if Rox.State is not RoxState.Idle and Rox.State is not RoxState.Corrupted:
Logging.get_logger().warn('Rox was already initialized, skipping setup')
returnAs a result, this line fails because Rox.setup(api_key, rox_options) == None and you cannot call .result(timeout) on it.
How to reproduce
A simple pytest like this one showcases the issue:
@mock.patch("rox.core.core.Core.setup")
def test_cloudbees_provider_error(mock_rox_core_setup):
mock_rox_core_setup.return_value = Future()
with pytest.raises(TimeoutError):
CloudbeesProvider(api_key="api_key", timeout=0.1) # raise TimeoutError
# Even if CloudbeesProvider raised, the Rox is still in state Set
assert Rox.State == RoxState.Set
with pytest.raises(AttributeError):
CloudbeesProvider("api_key") # raises AttributeErrorSolution
There are several solution possible, but something as simple as:
try:
self.setup_result = Rox.setup(api_key, rox_options).result(timeout)
except TimeoutError:
Rox.State = RoxState.Corrupted
self.shutdown()
raisecan do the trick.
What do you think ?