We were aware that external conditions can change and that the efficiency of our rank tracking system would significantly vary with that until adapted accordingly. So that is why we had a “spare”, backup system in place. However, we could not switch to it fast enough and, even when it was up and running, it couldn't deliver at the planned capacity. We didn’t take into account that its efficiency would be affected as well, as this was a first, so this is why it took us 5 days to fully recover.
To prevent this kind of setback in the future, we have now doubled our rank tracking capacity and improved the procedures, in order to make switching to our spare rank tracking system possible in a matter of minutes, reliably so. This enables us to stay nimble and adapt the systems to these changed external conditions, all the while still delivering on a par with our Service Level Agreement: fresh ranks for all keywords on both devices, daily by 8 AM.
As we could not recover all of the failed ranks between the 15th and the 20th, we are marking this timeframe on the ranks and Visibility Score graphs, so that you would always be aware of this data accuracy concern when reporting.
Posted Aug 21, 2019 - 19:01 EEST
Monitoring
The systems are now up and running at the optimal capacity, but we are thoroughly monitoring them for the following days. We are also further enhancing our high-availability system for rank tracking, to make sure that, under similar circumstances, we would not have this type of downtime anymore.
We have also started to recover any missing ranks.
Posted Aug 20, 2019 - 18:30 EEST
Identified
Our engineering team has identified that the sudden drop in the rate at which our systems can extract the SERPs has been caused by new external constraints. The systems are now progressively recovering their capacity and we are estimating that they will be back at full speed in the upcoming 24hrs. Following the full recovery, we will also start a process to recover any missing historical ranks.
Posted Aug 19, 2019 - 16:50 EEST
Update
The investigation is still ongoing, though progress has been made.
Posted Aug 18, 2019 - 14:30 EEST
Update
Our engineers have identified some issues, but we are still continuing to investigate the main cause.
Posted Aug 17, 2019 - 12:45 EEST
Investigating
A failure on the crawlers is generating delays in obtaining the daily ranks and calculating the corresponding visibility scores.
Posted Aug 16, 2019 - 18:00 EEST
This incident affected: Rank tracking service and Visibility.