Currents - Notice history

The system has recovered. We will continue to investigate the root cause and will update the description of the incident with the details when we have them.

Our backlog has recovered, and the service is back to normal.

The root cause appears to have been changes in our now reverted deployment. We will continue to investigate the cause and provide more details after our investigation.

We are still seeing issues as a result of the backlog, we are scaling up to address the backlog of tasks.

We are continuing to investigate the cause in the meantime.

We have reverted the most recent release, and the error rates have gone down.

We are still experiencing some slowdowns due to the task backlog that grew during the incident, but the system is mostly recovered.

Runs that were created during the outage will be marked as timed out, and you may see errors related to those runs in your local clients if they are still in progress.

New runs should not be affected, and should now work as expected.

We are still actively investigating the root cause.

This incident has been resolved

All systems are operational.

Impact Assessment

• customers run results were significantly delayed for ~5 hours

• the delays caused other issues with services that expected the results like webhook and integrations.

Root cause analysis

At ~8:20pm PST (3:20am UTC) we had an internal cleanup task run that removed old docker images used to deploy our data processing service. An oversight in how this was configured vs our processes resulted in the currently deployed service's docker image being removed.

Between ~4:00am PST (11:00am UTC) and ~9:00am PST (4pm UTC) the system was not able to successfully take any scaling action while it tried to deploy the deleted image. There were still a small number of instances still running and processing tasks, but not enough to deal with the load.

Once we re-deployed a newer image, we were able to quickly recover to normal operating status.