Would Agentic Coders ever be production-grade? Learning's from Intercom

Navigating the AI Product Development Landscape

In his talk, "Would Agentic Coders Ever be Production Grade? Learnings from Intercom’s AI Agent Journey," Des Traynor, co-founder and Chief Strategy Officer of Intercom, explores the complex journey of developing and deploying their AI-powered customer support agent, Fin. This journey underscores the evolving demands and opportunities for AI agents in production environments.

Building AI Products vs. Traditional Software

Des highlights the fundamental differences between developing AI products and traditional software. He emphasizes, "AI building product in the age of AI is extremely different." While impressive demonstrations are achievable in controlled settings, the real challenge lies in how these systems perform in the unpredictable world of customer support. Early versions of Fin, built shortly after the launch of ChatGPT, performed well in demos but struggled with real-world "torture tests." It was only with the advancement of models like GPT-4 that Intercom could "safeguard" Fin for real-world deployment.

The Ongoing Lifecycle and Probabilistic Readiness

Des elaborates on the dynamic nature of AI software development, describing it as a continuous cycle of brainstorming, testing, and iteration. He states, "the AI software development lifecycle is a lot more like brainstorm about all the things that might be possible… hope to God it works in the wild." Fin’s journey from resolving 25% of customer contacts at launch to approximately 58-59% today illustrates the importance of persistent iteration and live data testing. Readiness involves robust testing, user engagement, and impactful outcomes, rather than reaching a fixed point.

Customization, Edge Cases, and Customer Empowerment

Customization is crucial for catering to diverse customer needs and preferences. Des stresses that complexity should not be offloaded onto users. Instead, Intercom offers structured options for settings like tone or answer length, leaving free-form customization for niche requirements. This approach helps navigate the spectrum of customer needs while minimizing the risks of inappropriate responses.

AI and Human Support: Job Changes and Organizational Impact

Des provides a nuanced perspective on AI’s impact on human roles within organizations. While Fin automates routine queries, it also transforms the nature of human support roles, which now focus on complex issues. This shift opens up opportunities for more specialized positions, such as principal product specialists or automation overseers. Des draws parallels to past technological transformations, suggesting that while some roles may be displaced, new, more strategic roles will emerge.

Agent Operations, Under-the-Hood Innovations, and AI Coding Tools

Intercom employs a multi-layered approach to agent architecture, with proprietary subsystems handling tasks like retrieval and disambiguation. The choice of underlying large language models (LLMs), such as partnerships with Anthropic, provides incremental improvements rather than transformative changes compared to these architectural innovations. Des acknowledges the productivity gains from AI-assisted coding but emphasizes the importance of clean, cohesive codebases for maximizing these tools’ benefits.

About The Speakers

Des Traynor

Chief Strategy Officer & Co-founder, Intercom

Co-founder and Chief Strategy Officer at Intercom, leading R&D and shaping product strategy, with a background in UX, teaching, and startup investing.

Ben Galbraith

Head of Product, Tessl

Ben Galbraith leads product & design at Tessl with a track record across Google, Segment, and multiple successful startups, now based in London