October 24, 2022 | Detroit, Michigan
View More Details & Registration Information

The Sched app allows you to build your schedule but is not a substitute for your event registration. You must be registered for KubeCon + CloudNativeCon North America 2022 - Detroit, MI + Virtual and add this Co-Located event to your registration to participate in these sessions. If you have not registered but would like to join us, please go to the event registration page to purchase a registration.

Please note: This schedule is automatically displayed in Eastern Daylight Time (EDT), UTC -4. To see the schedule in your preferred timezone, please select from the drop-down menu to the right, above "Filter by Date."

The schedule is subject to change.
Back To Schedule
Monday, October 24 • 12:55pm - 1:05pm
⚡ Lightning Talk: Honey, I Broke the Things: Debugging Gray Failures in Production! - Radha Kumari, Slack

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
Migrations are one of the most challenging tasks we do as infrastructure engineers.
These are sometimes long, tedious and come with many technical challenges of their own.
At Slack, we switched from HAProxy to Envoy Proxy for all ingress traffic. Overall, this migration was a success, and did not cause any downtime, but even so, we ran into several interesting edge cases that caused minor problems, such as failing a small percentage of requests, or increasing latency for requests, or sometimes an unhappy bot.

Troubleshooting these sorts of 'gray' failures can be difficult, so this talk will discuss some of those facepalm moments: how they were detected, steps taken to investigate them, and how they were solved.

Takeaways from this talk include a specific set of approaches for debugging such problems with Envoy Proxy and other web proxies that we learnt via these events along with some engineering practices that eases the stress during a large migration.


Monday October 24, 2022 12:55pm - 1:05pm EDT
Room 310 B Huntington Place: 1 Washington Blvd, Detroit, MI 48226
  Lightning Talks