Currents - Notice history

This incident has been resolved

All systems are operational.

Impact Assessment

• customers run results were significantly delayed for ~5 hours

• the delays caused other issues with services that expected the results like webhook and integrations.

Root cause analysis

At ~8:20pm PST (3:20am UTC) we had an internal cleanup task run that removed old docker images used to deploy our data processing service. An oversight in how this was configured vs our processes resulted in the currently deployed service's docker image being removed.

Between ~4:00am PST (11:00am UTC) and ~9:00am PST (4pm UTC) the system was not able to successfully take any scaling action while it tried to deploy the deleted image. There were still a small number of instances still running and processing tasks, but not enough to deal with the load.

Once we re-deployed a newer image, we were able to quickly recover to normal operating status.

This incident has been resolved

All systems are operational.

Impact Assessment

• a few customers weren't able to create orchestrated runs for ~1.5h

• the orchestration didn't produce optimal results for ~4 hours

Root cause analysis

At ~12:15pm PST (7:15pm UTC) our database cluster performed a failover from primary node to one of its secondary nodes as part of applying a maintenance patch.

{"t":{"$date":"2024-08-27T19:12:21.629+00:00"},"s":"I",  "c":"ACCESS",   "id":20250,   "ctx":"conn126158","msg":"Authentication succeeded","attr":{"mechanism":"SCRAM-SHA-1","speculative":true,"principalName":"mms-automation","authenticationDatabase":"admin","remote":"192.168.253.167:50284","extraInfo":{}}}

{"t":{"$date":"2024-08-27T19:15:01.090+00:00"},"s":"I",  "c":"COMMAND",  "id":21579,   "ctx":"conn126158","msg":"Attempting to step down in response to replSetStepDown command"}

The patched server caused one of the queries to return a malformed response, which resulted in a cascading series of failures. As a result some customer were not able to create orchestrated runs.

After releasing a hot patch that restored system stability, we released a permanent fix that restored the full service capacity.

Update 2024-08-28 12:25AM PST

The root cause was further refined to an issue in MongoDB update.

The cluster underwent MongoDB Version Upgrade from v6.0.16 to v6.0.17. The plan for the upgrade completed at 08/27/2024, at 07:13:33 PM UTC which corresponds with the onset of the query issue.
​
MongoDB internal team identified aggregation issues introduced in version v6.0.17 and has scheduled a downgrade the cluster to v6.0.16.

MongoDB is progressively reverting the 6.0.17 upgrades back to 6.0.16, but I do not have a timeline for when your clusters will be done.
​

We sincerely apologize for the inconvenience. Please let me know if you have any further questions.

We have identified the root cause and released a hot patch that prevent erroneous HTTP responses. The orchestration isn't distributing the tests in the optimal order at the moment, working on restoring the service to full capacity.