Dear Customers and Partners!
The reliability and stability of GREEN-API have always been our top priority. When choosing a hosting provider, we focused first and foremost on reliability, even if the cost was significantly higher than other offers on the market. That is precisely why we deployed our core infrastructure on Yandex Cloud and relied on protection provided by multiple availability zones.
What happened showed that these measures were insufficient. We faced the loss of our core infrastructure, prolonged downtime, and the need to restore the service practically from scratch.
Below, I’ll explain what happened, why the recovery took so long, and what changes we’re planning.
What Happened
On October 8, a drone attack occurred at the Yandex data center in the Sasovo ru-central1-b zone. Part of our infrastructure was affected, but mission-critical servers continued to operate. We managed to quickly stabilize the service.
On October 11, there was a second drone attack on Yandex’s data center in Vladimir, in the ru-central1-a zone. This time, we lost most of our infrastructure. Microservices, databases, and Kubernetes cluster control servers were affected.
The loss of the control servers prevented us from quickly redistributing the load and bringing new capacity online. We had to redeploy the infrastructure necessary for the service to function.
How We Restored GREEN-API
Our DevOps engineers began the recovery process immediately after the incident on October 11. The first task was to find a provider capable of allocating the necessary server capacity.
This proved to be challenging. On that day, demand for servers had risen significantly, and many companies were closed for the holiday. We were unable to obtain the necessary resources immediately and in full.
On the dedicated servers, we began deploying separate Kubernetes clusters from scratch for WhatsApp, Telegram, and MAX. At the same time, we were restoring databases and configurations.
We were able to restore all the data necessary for the MAX instances to function from backups. Therefore, MAX users did not have to re-authorize their instances.
Unfortunately, we were unable to restore the data for the WhatsApp and Telegram instances from backups. That data was irretrievably lost.
The central database was also destroyed, but its backup was preserved. We were able to restore the database—with some data loss—within a single day.
Why the Recovery Took Three Days
The scale of the damage made it impossible to simply restart the servers or migrate individual components. We had to procure new computing resources, redeploy clusters for each messaging app, restore the central database, and verify that all components were working together properly.
The work was complicated by a shortage of server resources and the loss of some configuration files.
Additional time was required to restore the central database from a backup.
Typically, it takes us about two weeks to deploy and fully test a single Kubernetes cluster. This time, we deployed three clusters in three days.
Unfortunately, the allocated server capacity proved insufficient for stable cluster operation, and we observed significant service disruptions. Over the next two days, we managed to stabilize the GREEN-API service.
This was a huge amount of work for the team under challenging conditions. But we understand that what matters to our clients is the result—a working service. A three-day recovery period remains unacceptably long.
What We’re Changing
We invested a great deal of effort and resources into the transition to Kubernetes. However, the cluster architecture alone did not protect us from the simultaneous loss of critical components at a single provider.
Our main mistake was relying too heavily on the reliability of a single hosting provider. Deploying across different availability zones proved insufficient.
We now plan to transition to a geographically distributed infrastructure across independent providers with database mirroring. Our goal is to ensure we can continue operations or quickly restore them in the event of the loss of a single data center or provider.
This is a more complex and more expensive solution. However, this incident has shown that such changes are necessary for the reliable operation of GREEN-API.
Compensation for Customers and Partners
As compensation for the downtime, we are prepared to offer:
- To all partners whose servers were affected in Russia—a 50% discount on their October bill
- To all customers—one additional month of service free of charge
We understand that this compensation cannot fully offset the impact of the outage on your business. However, we believe it is essential to support our customers and partners who faced difficulties due to the downtime on our part.
Our Responsibility
You have entrusted GREEN-API with communications with your customers. We understand that service unavailability means a halt to your business processes, financial losses, and reputational risks.
We let you down. We apologize for the extended downtime and data loss.
Our priority now is to analyze the causes of this incident and modify our architecture to reduce the risk of such a situation recurring and to shorten recovery times in the event of future outages.
Thank you for your patience, support, and for staying with us.
Andrey Shamin
CEO and Founder of GREEN-API 