Scalability and reliability are key factors, especially when handling millions of daily orders. Load testing is essential for ensuring smooth performance under growing demands and holiday spikes. However, manual analysis of results is time-consuming and often incomplete, overlooking critical factors like latency, resource usage and small deviations, which looks negligible from the quick glance. For example, memory leaks in the Golang program. In this session, you'll learn how we delegated load test analysis by integrating AI into our workflow. By automating result evaluations, AI analyzes complex datasets, going beyond if-else conditions. Our system now delivers actionable insights, identifying issues like memory leaks and CPU inefficiencies, along with improvement suggestions based on benchmarks. Discover how this AI-powered approach helps us meet resilience and performance goals, scaling from average load to 4x peak demand within minutes, while maintaining p95 under 150 milliseconds. AI is a nice tool and helper for daily routine tasks. But the quality of the "help" directly depends on provided input and... model itself. Bear with me and look together how to transform AI to help with routine tasks.
Revolutionizing Load Testing at Scale
In the fast-paced realm of global e-commerce, Delivery Hero stands as a titan, orchestrating over 10 million daily orders across 70 countries. As Andrii Raikov, Principal Engineer at Delivery Hero, outlined in his talk, the COVID-19 pandemic was a pivotal moment that exposed the company's scalability challenges. This period of rapid growth urged the company to adopt an ethos of "redesign for failure," with load testing becoming a cornerstone for ensuring systems could handle significantly increased loads without compromising on latency or error thresholds.
From Manual to Automated Load Testing
Initially, Delivery Hero's load test analysis was a manual and resource-intensive process. Engineers created extensive reports, as Andrii described: "Individual engineers compiled sprawling 12-page Google Docs filled with screenshots from various monitoring tools." This approach was clearly unsustainable as the infrastructure expanded. The transition to automation began with templated reporting using GoLand, allowing for dynamic data embedding and chart generation.
Integrating AI: Challenges and Considerations
Raikov highlighted the simplicity of implementing AI: "Using AI is easy—three lines of code is almost all implementation we have." Yet, the real challenge lay in crafting effective prompts and managing the AI's verbosity and hallucinations. Raikov emphasized, "AI is talkative," underscoring the importance of precise prompt structuring to extract valuable insights.
Structuring Prompts for Maximum Efficacy
To enhance the AI's utility, Raikov introduced a structured approach to prompt engineering. This involved three components: defining explicit requirements, detailing production and load test data, and specifying desired output formats. This meticulous approach ensured that the AI provided clear, actionable insights, such as identifying outlier CPU usage indicative of potential load-balancing issues.
Continuous Improvement and AI's Role in Load Testing
Raikov concluded with an optimistic view of AI's potential in load testing, balanced by an awareness of its limitations. He invited attendees to engage with these challenges: "Integrating AI for load testing analysis is 'super easy' at a surface level, but genuine value depends on precise problem formulation and continual prompt iteration."
About The Speaker
Principal Software Engineer, Delivery Hero SE
Andrii has a total of 17 years of software engineering experience and has been very passionate about Go for the past 6 years. Over the past 5 years at Delivery Hero, he has adeptly harnessed Golang, excelling in constructing high-load applications, leveraging Golang's concurrency capabilities to engineer scalable solutions that optimise performance and handle substantial data processing tasks.
THE WEEKLY DIGEST
Subscribe
Sign up to be notified when we post.
Subscribe