Open-source Git Repository GitLab has been facing major database connection issues on its platform for the last twelve hours or so. The Git platform is currently offline due to a data migration failure witnessed after accidental deletion of its production data. This is something that was unexpected and occurred during the regular database maintenance.
Though GitLab returned back online after its regular maintenance but users started experiencing load time and stability issues with it. This led the company to escalate the problem for emergency maintenance. This led the platform to be taken offline to examine the issue. There has, thus, been an outage since yesterday.
Database maintenance has been completed, https://t.co/r11UmmDLDE is back online
— GitLab.com Status (@gitlabstatus) January 31, 2017
We are performing emergency database maintenance, https://t.co/r11UmmDLDE will be taken offline
— GitLab.com Status (@gitlabstatus) January 31, 2017
Then, the company started tweeting out maintenance updates (which are now also available in this Google Doc) to make users aware of the problem with their database. It then started experiencing issues with the production database, which is used to handle the platform daily. It felt helpless in recovering the same and accidentally deleted the production data, containing a massive 300 GB of data. About 4.5GB of data was remaining when the delete command was cancelled, reports The Register.
we are experiencing issues with our production database and are working to recover
— GitLab.com Status (@gitlabstatus) February 1, 2017
We accidentally deleted production data and might have to restore from backup. Google Doc with live notes https://t.co/EVRbHzYlk8
— GitLab.com Status (@gitlabstatus) February 1, 2017
The issue with the production database might have been massive but the company has since been rigorously trying to recover the data from backups — with some luck at the moment. The developers using the platform needn’t panic as the company has already tweeted out saying that only the database, including issues and merge requests, was affected and the git repositories and wikis are safe and sound.
The incident affected the database (including issues and merge requests) but not the git repo’s (repositories and wikis).
— GitLab.com Status (@gitlabstatus) February 1, 2017
The situation is now under control and the company’s Twitter is updating the status regularly. The Google Doc sheds light on the “problems encountered” by the platform saying that none of GitLab’s deployed backup/replication techniques were not working reliably. This sounds like a major problem within the company’s infrastructure. But, it has managed to slowly recover 60 percent of its database.
You can track the recovery of the database: https://t.co/S4rR2BAns3
— GitLab.com Status (@gitlabstatus) February 1, 2017
In a statement received by The Tech Portal, a GitLab spokesperson says,
On Tuesday, GitLab experienced an outage for one of its products, the online service GitLab.com. This outage did not affect our Enterprise customers or the wide majority of our users.
As part of our ongoing recovery efforts, we are actively investigating a potential data loss. If confirmed, this data loss would affect less than 1% of our user base, and specifically peripheral metadata that was written during a 6-hour window.
We have been working around the clock to resume service on the affected product, and set up long-term measures to prevent this from happening again.
San Francisco-based GitLab, for those unaware, is an open-source platform which has received backing worth $25.6 million from the likes of Khosla Ventures, August Capital, and Y Combinator. The company currently has over 100 employees and competes with uber-popular platforms like GitHub and Atlassian. It has attracted investors to its platform, courtesy of its open-source nature and collaboration tool.
