Australian productivity tools maker Atlassian is having a pretty bad week.
An outage that started on April 4 – yes, over 10 days ago – continues to cause the company issues, impacting approximately 400 of its clients. Given the scale of the company, these clients could be some pretty heavy hitters.
In a blog post, Atlassian’s CTO Sri Viswanath has given some more details for what, exactly, is going on, after several days of quiet. As of April 14, around 55% of those affected had regained access to their data and services.
“Communications gap”
After explaining that there was a “communications gap” between tweets, and the fact that the wrong set of IDs were provided, Viswanath gets into the crux of the issue.
“[T]he script we used provided both the “mark for deletion” capability used in normal day-to-day operations (where recoverability is desirable), and the “permanently delete” capability that is required to permanently remove data when required for compliance reasons,” he wrote.
“The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 customers were improperly deleted.”
Update on cloud outage impacting ~400 customers. As part of scheduled maintenance our team ran a script to delete legacy data from a deprecated service. Instead of deleting the data the script erroneously deleted sites, and connected products, users, and 3rd party apps. (1/5)April 12, 2022
The fix is in
Atlassian, of course, has been working hard to rectify this mistake and restore customer data.
As of April 15, around 55% of customers have had this data restored, the company says, and automation is being used to speed up the rest of the recovery.
Atlassian maintains extensive backup and recovery systems, and there has been no reported data loss for customers that have been restored to date. This incident was not the result of a cyberattack and there has been no unauthorized access to customer data. (2/5)April 12, 2022
The fact that this wasn’t a cyberattack, and the company maintains extensive backups, means that the eventual data loss will likely be minimal. But it remains an issue for a company whose clients rely on it.
In the hyper-competitive cloud services space, a simple error like this – causing pain to paying clients – could force some to look elsewhere.
“We know that incidents like this can erode trust,” says Viswanath. “We are not meeting the high standards that we set for ourselves. This includes our communications efforts, which until now were entirely focused on reaching our impacted customers directly.”
