RedisLabs network issues
Incident Report for Yuso
Postmortem

Due to some network connectivity issues coming from our message broker (Redis) provider Redis Labs (https://status.redislabs.com/incidents/h1gmlvw0g7v0),,) some of our redis workers could not process new messages, especially for the dispatch process.
Although this looked like the rides couldn’t be dispatched at all, the system was in fact working really slowly, the time for the worker to get a stable access to the Redis instances.
Since the issue didn’t last for long, the workers could process all the messages right after the network connection was stable again.

Posted Mar 23, 2020 - 15:23 CET

Resolved
20h45 - Call from our client Webtaxi
20h50 - A lot of notification from our error management system on redis connection
20h55 - Update on RedisLabs status page about network connectivity issues
21h00 - Dispatch restarted
21h05 - Reboot of all our Redis worker to avoid dead connections
Posted Mar 13, 2020 - 20:45 CET