Building Notion AI: Lessons Learned and Myths Busted

The Birth of Notion AI

The journey of Notion AI began in October 2022 when the team gained early access to GPT-4. Simon Last described this as a turning point, stating, "Playing with GPT-4 was the trigger for me. Oh my God, this thing is actually really useful now." This realization led to the immediate development of Notion's first AI product, an AI writing assistant launched in February 2023. The assistant can write, edit, and insert text, offering various pre-packaged prompts for improving writing.

This initial foray into AI was not just about incorporating a trendy technology but about fundamentally transforming how users interact with digital content. The AI writing assistant was designed to be intuitive, allowing users to seamlessly integrate it into their existing workflows. This integration was made possible by understanding the core needs of users—efficiency, accuracy, and ease of use. Simon Last emphasized that the AI's role was to complement human creativity, not replace it, by taking over repetitive tasks and allowing users to focus on more strategic activities.

Key AI Products Introduced by Notion

Notion introduced three main AI products: AI Writing Assistant, AI Autofill, and Q&A. The AI Writing Assistant allows users to write and edit text with ease, using custom prompts. Simon Last explained, "There were prepackaged actions and then also you could just type whatever you wanted." The AI Autofill feature enables users to fill out database columns using AI-generated prompts, useful for summarizing or translating content. The Q&A product, launched in November 2023, indexes all of Notion, using embeddings to facilitate a chat bot where users can ask questions. Simon Last noted, "We built an embedding index over all of notion, and then you could ask questions and it's a chat bot."

Each of these products serves a distinct but complementary purpose in the Notion ecosystem. The AI Writing Assistant boosts productivity by streamlining content creation, while AI Autofill automates data entry processes, reducing manual workload. The Q&A feature leverages advanced natural language processing to provide accurate and contextually relevant answers, making information retrieval faster and more intuitive. Together, these products exemplify Notion's holistic approach to integrating AI across its platform, enhancing its utility and user experience.

Organizational Structure and Team Dynamics

The AI development at Notion started with a small, agile "tiger team," which Simon Last believes is crucial for rapid innovation. "It's good to have a small group of people that can move really fast," he stated. As the AI efforts expanded, the team grew to about 20 people, organized into subgroups focusing on indexing, UX, and modeling. Despite challenges in democratizing AI across teams, Simon Last emphasized the importance of enabling more teams to work with AI.

This approach not only fosters innovation but also promotes a culture of collaboration and knowledge sharing. By embedding AI specialists within various teams, Notion ensures that AI expertise permeates the organization, leading to more cohesive and integrated product development. Simon Last highlighted the importance of maintaining a balance between centralized AI expertise and distributed innovation, allowing all teams to leverage AI while benefiting from a shared foundation of knowledge and resources.

Evaluation and Testing of AI Capabilities

Evaluating AI products is challenging, with Simon Last highlighting the need for a repeatable evaluation system. Notion focuses on robust logging and dataset creation to track and address failures. Simon Last explained the importance of deterministic evaluations: "For the situations that you test, you need to make sure that those work and they don't regress." This approach allows Notion to continuously improve their AI capabilities.

The evaluation process involves rigorous testing, using both synthetic and real-world data to simulate various scenarios. This iterative method ensures that AI models are not only accurate but also resilient to changes and capable of adapting to new data inputs. By prioritizing empirical testing and data-driven insights, Notion can refine its AI models, enhancing their reliability and performance over time.

The Role of Fine-Tuning and Model Selection

Simon Last shared insights into the complexities of fine-tuning models, noting that it often complicates the development process. "You're making your job like a hundred times harder," he remarked. Instead, Notion prefers in-context learning and leveraging the latest models to maintain product stability while adapting to new technological advancements.

Rather than relying heavily on custom-trained models, Notion opts to use the most advanced models available, integrating them into its products in a way that aligns with user needs. This strategy allows Notion to stay at the forefront of AI innovation without the overhead of extensive model training and maintenance. Simon Last emphasized the importance of flexibility and adaptability in AI development, recognizing that the landscape is rapidly evolving.

Building Trust in AI

Trust is a critical aspect of AI adoption. Notion employs strategies such as user verification and transparent actions to build trust. Simon Last stated, "We show you this little pop up and the default is no, but you can opt in to sharing data with us." Citations and visualizations in the Q&A product also help users verify answers, fostering trust in the AI's outputs.

Transparency is key to trust, and Notion is committed to clear communication with its users about how their data is used and protected. By providing users with the option to opt-in for data sharing, Notion respects user privacy while still gathering valuable insights for improvement. Additionally, visual cues and citations in AI-generated content allow users to assess the reliability of the information provided, enhancing their confidence in the tool's accuracy.

The Future Vision for Notion AI

Looking ahead, Simon Last envisions AI automating tedious tasks, allowing humans to focus on higher-level work. He sees AI as a primitive, integral to Notion's mission of enabling custom software creation. "Our goal as a company is to try to break the pattern of these like rigid vertical SaaS tools," he shared. This vision positions Notion AI to significantly impact knowledge work, elevating productivity and innovation.

As AI continues to evolve, Notion aims to harness its potential to transform how individuals and organizations manage information. By automating routine processes, Notion empowers users to allocate their time and energy towards more strategic, creative pursuits. This shift not only enhances productivity but also fosters a more dynamic and adaptable work environment, where AI serves as a powerful ally in achieving business goals.