Building a Retrievable Codebase Memory Model

Enhancing AI Code Review with Context

Nimrod Kor begins by addressing a fundamental challenge in AI-driven code reviews: the lack of contextual understanding. "Something felt a little bit missing," Kor remarks, highlighting the robotic nature of early iterations of Vaz's review agents. They could summarize changes and generate standard comments but couldn't grasp the full context of the code. Kor emphasizes that understanding the 'why,' 'where,' and 'how' is essential for effective reviews.

Hauser illustrates this point with the classic B/13 optical illusion, underscoring the adage "context is king." In code review, this translates to understanding the code, its place in the system, and its impact on users. To infuse deeper context into their platform, Kor and Hauser set out to generate "flow context"—summaries that describe the function of modules or code regions, enhancing the LLM prompts for improved summarization and review.

Iterative Development and Challenges

Initially, the team adopted a lean, manual proof-of-concept (POC) approach to test the value of additional context. Despite frequent iterations, they found that context did not always change the output, which led to careful evaluations at every step. Once they confirmed the value of context, they sought to automate its generation.

They first considered feeding entire repositories to LLMs but soon rejected this due to issues with cost, speed, and scalability, as well as the "lost in the middle" problem. Kor explains that even models with large context windows are not viable due to performance and cost constraints, and stale data undermines the goal.

Advanced Techniques for Context Generation

Their initial automated approach involved summarizing modules using TF-IDF to identify unique elements. However, this method struggled with common patterns, leading to inconsistent results. Recognizing this, they shifted to a more graph-based analysis using Tree-sitter to build "connection graphs" that represent code entities and relationships. By employing the Luvain algorithm, they identified strongly connected components to extract representative files or regions.

Despite technical success, this method failed when changes affected less prominent areas not included in summaries. Learning from this, they adopted a dynamic strategy that identifies not just changed files, but also related documents within the connection graph, even if untouched by the pull request. This "graph theory magic" allows them to assemble the minimal context needed for precise LLM comprehension.

Continuous Evaluation and Adaptation

Kor and Hauser stress the importance of evaluation throughout their journey. They employ various benchmarking techniques, such as having LLMs judge summaries and comparing outputs with and without context. Hauser warns against "falling in love with a single example," as effective solutions in one codebase may not work elsewhere.

The key takeaway from Kor and Hauser's experience is to avoid rushing into building without validating impact, to remain open to iteration, and to prioritize systematic evaluation. By combining precomputation, dynamic retrieval, and rigorous benchmarking, they aim to create truly context-rich, retrievable codebase memory for AI code reviews.