Andrii Raikov
Principal Software Engineer, Delivery Hero SE
Scalability and reliability are key factors, especially when handling millions of daily orders. Load testing is essential for ensuring smooth performance under growing demands and holiday spikes. However, manual analysis of results is time-consuming and often incomplete, overlooking critical factors like latency, resource usage and small deviations, which looks negligible from the quick glance. For example, memory leaks in the Golang program.
In this session, you'll learn how we delegated load test analysis by integrating AI into our workflow. By automating result evaluations, AI analyzes complex datasets, going beyond if-else conditions. Our system now delivers actionable insights, identifying issues like memory leaks and CPU inefficiencies, along with improvement suggestions based on benchmarks.
Discover how this AI-powered approach helps us meet resilience and performance goals, scaling from average load to 4x peak demand within minutes, while maintaining p95 under 150 milliseconds.
AI is a nice tool and helper for daily routine tasks. But the quality of the "help" directly depends on provided input and... model itself. Bear with me and look together how to transform AI to help with routine tasks.
In the fast-paced realm of global e-commerce, Delivery Hero stands as a titan, orchestrating over 10 million daily orders across 70 countries. As Andrii Raikov, Principal Engineer at Delivery Hero, outlined in his talk, the COVID-19 pandemic was a pivotal moment that exposed the company's scalability challenges. This period of rapid growth urged the company to adopt an ethos of "redesign for failure," with load testing becoming a cornerstone for ensuring systems could handle significantly increased loads without compromising on latency or error thresholds.
Initially, Delivery Hero's load test analysis was a manual and resource-intensive process. Engineers created extensive reports, as Andrii described: "Individual engineers compiled sprawling 12-page Google Docs filled with screenshots from various monitoring tools." This approach was clearly unsustainable as the infrastructure expanded. The transition to automation began with templated reporting using GoLand, allowing for dynamic data embedding and chart generation.
Raikov highlighted the simplicity of implementing AI: "Using AI is easy—three lines of code is almost all implementation we have." Yet, the real challenge lay in crafting effective prompts and managing the AI's verbosity and hallucinations. Raikov emphasized, "AI is talkative," underscoring the importance of precise prompt structuring to extract valuable insights.
To enhance the AI's utility, Raikov introduced a structured approach to prompt engineering. This involved three components: defining explicit requirements, detailing production and load test data, and specifying desired output formats. This meticulous approach ensured that the AI provided clear, actionable insights, such as identifying outlier CPU usage indicative of potential load-balancing issues.
Raikov concluded with an optimistic view of AI's potential in load testing, balanced by an awareness of its limitations. He invited attendees to engage with these challenges: "Integrating AI for load testing analysis is 'super easy' at a surface level, but genuine value depends on precise problem formulation and continual prompt iteration."
Principal Software Engineer, Delivery Hero SE
17 years software experience, Go specialist at Delivery Hero, builds high-load scalable applications
Stay connected, share your thoughts, and be part of the community.
