RAG beats Fine-Tuning in learning your code base

**Guy Gur-Ari:** [00:00:00] To deeply understand the code base, no matter what you're using. So you could be you could be trying to call a function. And so you're, you want to get a completion at your cursor that just fills in the fields for that function correctly, pulling them in from all kinds of places in your code, or you could be.

Again, diving into a new repository, right? And so you want to understand what is this repository about? Give me the high level structure. How did the components talk to each other? What's going on here? I don't want to bother reading documentation, which often is missing or out of date. I just want to know what the current state of the code is or maybe I'm looking for a component that I know I wrote six months ago, but I can't figure out where it is.

I just want to ask chat, tell me where this thing is and it should go in. And find it for me. 

**Simon Maple:** You're listening to the AI native dev brought to you by Tessl.[00:01:00] 

**Guy Podjarny:** Hello everyone. Welcome back to the AI Native Dev. Today we have a really fun episode. We were going to dig into what does it mean to build a coding assistant, a coding analysis, and we'll have a better explanation of what it is here, to make developers more and more productive as they build with AI.

And to do that, we have Guy Gur-Ari, who is the co founder of Augment. Guy, thanks for joining us here on the podcast. 

**Guy Gur-Ari:** Yeah, very glad to be here. Thanks for inviting me on. 

**Guy Podjarny:** And we have a slightly unique episode here in which we have two half hosts here where we have Dion. Hey, Dion.

**Dion Almaer:** Hey, good to be here. It's really fun to be with my favorite founders who happened to both be Israeli guys. 

**Guy Podjarny:** Yeah, so Guy and I are both Guy yeah, actually there's a bit of a story behind that, right? We were both in the same group many years ago, not to say so we don't sound too old.

But many years ago and there were four guys named Guy. And so none of us were named Guy. At least for me, I stayed with Guypo on it as the hashtag, but [00:02:00] Guy you managed to retain just the first name piece. 

**Guy Gur-Ari:** Yes I was Guygur back then, but that somehow did not stick maybe because I didn't try to propagate it much. 

**Guy Podjarny:** Yeah. You like the vowel, Guypo at least had that extra O, but you were the Guy G R clearly all a bunch of nerds, it was all about the username that that we happen to have on the Unix system, but it's Solaris, the sort of system Cool.

So we're going to dig into indeed, what is to augment code, if you will, code development on it, maybe to get us started can you tell us guy just a little bit about Augment and what you do and then we'll go from there. 

**Guy Gur-Ari:** Yeah Augment is an AI coding assistant that gives you the whole code base at your fingertips.

Essentially, every way you interact with Augment, we try to make it seem like the models really understand your full code base. And so what that means is that then you can go and make changes to the code, write new code, and do that all with confidence, without having to retain the whole code base in your head.

So just to give, I think we'll go more deeply into it, but just [00:03:00] to give one example where we see this help a lot if I need to go into a new part of the code base, for example, that I've never touched before, with Augment, I can get up to speed and start making changes. With confidence, usually within a few hours to a day versus something that would take me sometimes a week or longer just to get up to speed on what's happening in that part of the code base or in a new repository I'm looking at.

And so that's maybe a way to summarize. It's for us, it's about the context, making the context from the code base available to the model leads to much better results as far as we can tell. 

**Guy Podjarny:** Cool. Super interesting. And yeah, we're going to dig a lot more into that. And I guess maybe veering a little bit backwards from augment you've had one of the earlier exposures to LLM still in your role in Google. Can you tell us a little bit about the I guess how you found out about LLMs, you started building them. And I guess maybe when is it that you learn to appreciate their power.

**Guy Gur-Ari:** Yeah. So I joined Google, this was six years Very excited about machine learning. I thought this was [00:04:00] a scientific revolution that was happening and a once in a lifetime thing. And I was very excited to be part of it. Back then, the discussion was really centered a lot more from my perspective on vision models, convolutional neural networks were all the rage.

And so I was doing a lot of research work on that. And then when GPT 3 came out for me that was a pivotal moment because it became clear that there's something very special about language models based on transformers where suddenly instead of having to fine tune the model to do something that you like, you can prompt it.

You can do few short prompting, which really cut down on the amount of time you had to spend with the model. To get something out of it. And so me and the team at Google, we really pivoted our work toward language models. And we started evaluating them. We started being more involved in training the large language models.

I was involved in some of the training of the large models at Google [00:05:00] And kind of through that work I became convinced that yes, this is it. This is really the new heart I would say of the AI revolution is these generative models that are very general and can generate a lot of text for you.

**Guy Podjarny:** And I guess how quickly did it become apparent that not only are they powerful, these LLMs, but that they can actually help with code. They can help with software development. 

**Guy Gur-Ari:** Yeah. So I think that certainly took some time. I think that to me and to folks I at least interacted with that was not obvious from the beginning and I think maybe the reason why is that there was something about reasoning heavy tasks that seemed more remote or more out of reach or harder. There are many people who've done very good work on reasoning tasks for many years trying to do a lot of sometimes you might call it feature engineering. Sometimes you might call it find architectures to solve hard reasoning tasks, a lot of [00:06:00] symbolic work, semantic analysis on code and on math and so on.

And still that work is, I would say a lot of it is still relevant, but I think it was the big surprise, certainly to me. And I think also to many others is the degree to which you can apply scale of compute and scale of data and solve hard reasoning tasks. And so code is an example of that.

Other examples of that are math and science. I think probably most people were surprised by how good you can make these models using very general techniques, the same techniques that you use to get them to write poetry to get them to edit law documents and so on and help them solve reasoning tasks.

And I think code was a part of that. I think one of the earliest indications that there was something interesting happening was the Codex model. And then at Google, we had our own code models where we saw some success. But the rate of progress on code has been surprising.

**Guy Podjarny:** But also these other models, they were not built on language, right? Like they were built [00:07:00] on, more, I don't know what the term is it the neuro symbolic or is it like, I don't know, like I'm probably abusing the terms, but they were understanding the code, bringing it down into some sort of symbolic representation of the code and then analyzing on those.

And the LLMs is I, my kind of layman data science understanding here. Like they were very different. They jumped into completing treating code as text. Is that a correct kind of representation here? 

**Guy Gur-Ari:** That's exactly right. This is part of, so people I think widely in the community thought that yes, you need to do something special to deal with code and also to deal with math and other hard reasoning tasks.

Neurosymbolic approaches were that was one big approach. There were also purely symbolic approaches. And yes, the surprise is that when you take a large language models, and you just use the same general techniques, the same architecture, the same methods of training that you use on other text domains, you get excellent results if you scale it up.

Yes. So that was I agree with that description. Yeah. 

**Dion Almaer:** And you worked on the research that went into the Minerva [00:08:00] paper around reasoning? Is that correct? 

**Guy Gur-Ari:** Yes. So we worked on another hard reasoning domain, solving math and science questions. And for us, that was the eureka moment in that domain, which was, yes, we, the work involved again, scaling up the data downloading all of the archive papers, all the research papers we could find.

Scanning the internet for all the papers that had something to do with mathematics, which at Google, all the infrastructure makes that straightforward to do and then train the largest model we could put our hands on back then it was the Paul model from Google, train it on all that data and see what it can do in terms of math.

And we were blown away by the quality you can extract from these very general models. train in a very general way, we got these models to pass university entrance exams based on these techniques. And so for me, that was the point where I thought, Oh if we can do this on reasoning, you can do this probably on code [00:09:00] and on many other domains.

It's not that reasoning is this special thing. The general techniques are really powerful enough to address those problems as well. 

**Guy Podjarny:** Very cool. And I realized Dion that I I skipped the part in which I explain your connection to Augment on it. So listeners may be aware that Dion has recently joined.

And you're a field CTO and you're doing a whole bunch of good things over here. And before that you were at Augment and you were leading a lot of the product over there. So you end up being here knowing all sorts of perspectives about this domain. 

**Dion Almaer:** Exactly. And when Guy was talking about, at the high level, the importance of context and the like, I just want to highlight that the proof is in the pudding with all of these things. When I first met with the Augment team and seeing the product even back then, and it's just gotten, obviously increasingly better over time. The feel of using that product compared to other coding assistants that are out there is just a very different experience.

And I noticed that as of today, Friday, January 17th, you can self serve and actually try it out [00:10:00] yourself. And I just really recommend downloading it, running on a real repo and doing a real task. And that's where you get to feel the difference, right? It's the analogy that I always give is there were lots of search engines and then Google came along and it felt a little bit different.

And that was my early experience with Augment. And that's where I get to meet up with Guy and Igor, the other co founder, and then the amazing research team that they'd built to pull this off the. Yeah, that we'll get into more detail. Yeah, it's a pleasure to be here with both Guys.

**Guy Podjarny:** So I guess, I think we've been attempting this topic for long enough on it. Let's dig into the context, give context on context. So what is this special sauce? Like when you talk about context matters, you talk about that being something that, provides different level of magic when you use it.

Tell us a bit more about what that means and maybe how it differs than maybe the normal way in which LLMs might operate, LLM systems might operate. 

**Guy Gur-Ari:** Yeah, so I'd say there are a few things that go into it. One is just basic [00:11:00] ability to deeply understand the code base no matter what you're using. So you could be trying to call a function and so you want to get a completion at your cursor that just fills in the fields for that function correctly, pulling them in from all kinds of places in your code.

Or you could be, again diving into a new repository, right? And so you want to understand what is this repository about? Give me the high level structure. How did the components talk to each other? What's going on here? I don't want to bother reading documentation, which often is missing or out of date. I just want to know what the current state of the code is, or maybe I'm looking for a component that I know I wrote six months ago, but I can't figure out where it is.

I just want to ask chat, tell me where this thing is and it should go in. And find it for me. So all these things require context. All these things require analyzing the full code base and being able to do that in a way to correctly answer questions and write, generate code based on it. [00:12:00] You have to have the whole code base in there.

And we know that the model context windows today, and probably in the foreseeable future are not large enough to fit real large scale, complex code bases in there. And so we have to use other techniques in order to make the models. aware of all that context. So one side of it is yes. How can we pull in the correct context from the repository to answer the particular query to address the particular completion?

I'd say the other side of it that we found also very important is in the interactivity. So that means, first of all, it has to be really fast. If it's not fast, it quickly gets annoying, even if the quality is good. And so there's a lot of engineering and research effort that goes into that. And there's also a real time aspect to it.

If I'm typing away a function in one file and I switched to my unit test file and I want to write the test for that. I [00:13:00] don't want to have to tell the model. I just wrote this function. Please write the test for me. It should know that function is something I just wrote. It should follow me along as I'm editing.

And so the code based awareness should also be aware of what I'm doing. It should be aware of the real time nature of the work. Getting that all to work together. And folks who have worked on RAD, they know that you need to indexing step and that indexing step can be challenging and can take a while getting all that to work together and getting it to work within hundreds of milliseconds and getting that whole thing to come together as a kind of delightful user experience.

This is where the challenges like get the quality, right? Get the speed, right? Get the real time aspect of it. And all these things have to come together. On any possible task a developer might want to do in their code base. 

**Guy Podjarny:** Yeah. Yeah. So I guess this is super interesting and a few clarifying questions.

One is does the context also make you a better developer or just a better developer in this environment? Is it? I can [00:14:00] understand or it's easy to understand how, if I'm a super skilled developer and I go into a brand new organization, I've never written code in their code base, then I would be at a certain level, maybe after five years working in this code base on it.

I will know how to write better code in this company. I wouldn't be a better developer. Maybe mine is what I've learned in this five years. But I would be better tuned in better tuned in my, with my surrounding, but is the same, is that the way to think about this context analysis? It's more about understanding, like being a better developer in this organization, or are there elements of it that you feel have the promise of actually just being a better developer, almost like the context of the world?

My natural thinking is like the context of the world gets encoded into the LLMs. That's what they bring with it. And you're talking about understanding the org. Is that right? 

**Guy Gur-Ari:** Yes, that's right. So I would say at this point we're starting with the code based awareness, which is what I've been describing.

There are a lot of other sources of information for that organization that can be relevant. We can talk about [00:15:00] them too. But yes, it is about aligning them all in some sense to the organization's knowledge everything else that's about the world. It usually is already baked into the model.

I would say that the way I've seen this make me a better developer is by making my learning process faster. And so often the model will notice or pick up on certain useful patterns or tricks or just good things that people have done around the code base and will surface them through these suggestions.

This process I think happens naturally, right? You talk to your co workers, you read other people's code, you learn. I would say that the context awareness accelerates that process. And of course, in any organization, there are things that are specific to that organization, but then there are also many useful patterns, right?

Whenever we work at a company, we learn things that we then take with us to our next role. I would say this accelerates that process, but it's less about, yes, this will, the [00:16:00] context awareness will help me learn what the latest methods are in my field. It's less about that. 

**Dion Almaer:** And I definitely saw that with customers, right?

Where I remember one developer using Augment was like, I heard that JavaScript had added sets and I kept thinking like I want to use this sometime and I never did. I never got around to it because it wasn't, it's outside of when I was actually doing things. But then all of a sudden as I'm writing code that sets apply, it's suggesting that.

And yes, that's baked into the overall piece, but also a lot of your co workers are pretty smart and do interesting things. And so it does pick up on those things as well and bring that across. So I definitely agree with Guy on just like the whole kind of accelerating your learning. And I definitely felt that even things I thought I had some expertise in, like TypeScript and the like, it was just teaching me things that felt like if I was pair programming with.

Colleagues and in this case, it's like a mesh of the intelligence of all of my colleagues all at once to help out. 

**Guy Podjarny:** [00:17:00] Yeah, it's interesting It's I guess the picture that comes to mind is more about focus if you know everything in the world and you might have there's all the craft of software development, but then there's a focusing element of looking at the aspects of the craft of software development that matter more in this specific organization in a specific setting and so maybe you now understand or the LLM or the system is now able to understand that is the type of specialization that you might want to bring to fore.

It does though, very much beg is yeah, in every organization, you take a bunch of things that you want to take with you to the next organization. And there's a whole pile of things that you learn to not do in the next organization. How do existing code base, existing enterprise scale, 20 year old code bases on it, have a lot of stuff in it that you absolutely do not want new codes to replicate.

How do you clean this code? How do you deal with that? 

**Guy Gur-Ari:** Yeah. And so that is the kind of thing where, to me, it falls under the bucket of lightweight steering. It's a bit like self [00:18:00] driving, where there, there are actually a lot of analogies here to self driving where solving the full thing autonomously will take a while.

I believe we'll get there eventually, but it will take a while. It will take years, probably, is my sense. And so something like how can we figure out automatically which parts of the code base are more relevant to the kind of thing we're doing now and which parts are relevant, but we actually want to avoid them. Understanding that automatically will take time.

I think a thing that can help with that is to go beyond the code base, because often that knowledge is not stored in the code base, right? It's stored in Slack conversations, it's stored in Google Docs and all kinds of design documents and so on. So certainly the more context we can show the model.

The better it will get at that but also with these kinds of questions, some lightweight steering helps. And so with Augment, you can give the model general guidelines that are always in effect, where you can tell [00:19:00] it when you're doing work, for example in this directory, or in this module, or in this component, pay less attention to this area of the code, because that's old.

And of course, that's not foolproof, just like everything with AI, it will keep making mistakes but it, at least steers the model more toward that direction. So I think currently it's some combination of showing them all as much context as we can, plus human steering to translate into just better generated code that follows the organization's patterns.

**Guy Podjarny:** And a bit of a crazy thought jumped to mind that I wonder if you can take like I think in the world of whatever in the tech world, some engineering organizations are perceived as better at their craft than others. There might be whatever, like Netflix used to be very much at the top of that.

It's Hey, I wonder if you can have Netflix sell the context of their code. So say, Hey, this is what good code looks like. And then I think there will be quite a few buyers to say, have your auto completion work to that tier. 

**Dion Almaer:** Yeah, even that would be a little 

**Guy Podjarny:** [00:20:00] bit too wild and nobody would ever do that in sharing that data.

**Dion Almaer:** But even within the organization, right? You have that particular teams where it's or even people where I just want to be like, Hey, any PR from Wendy, like I prioritize that more. And there was this, one of our, the favorite, customers who publicly spoken about this Guy, Merrick from Webflow, who's like a principal engineer and like deeply understands front end and runs the front end craft there, right?

He's able to steer things within the company, right? And for him, it was a huge aha moment. Like we talk a lot about like junior devs and what it means for them and learning code bases. And then as someone like him as a principal engineer, he's able to steer it. And it's wow, this is I'm virtually pair programming with all of the new developers of the company, right?

And like he can be going in there and okay, when you're using React now, this is how we want you to do things. We're using hooks. We're doing it in this certain way. And so given the other humans, the [00:21:00] ability steer what the system is doing, I think is huge. 

**Guy Podjarny:** And this is if I reflect this back a little bit, what is fairly common, although quite painstakingly takes a lot of effort, and so people don't do that as much, has been to try and build all sorts of linting or other rules that might inspect post writing the code.

That hey, that code that is written matches our standard, matches our new norms. You can do all sorts of things like that. And those are old practices. Again they're not done as often because they're just hard. It's just effort. Also, it's very annoying to write code and then afterwards get these, 50 aesthetic differences that changes that you need to do.

And so maybe a way to think, it sounds like a way to think about this is, can you reverse that? Can you take what you would have written in linting and in others, think about where they come later in assessment to actually inject them into the writing process, at least to the extent that the writing process was done by the coding system, by Augment in this case to build those out. So [00:22:00] it's interesting. It's very how again, maybe a silly question from a data center's perspective, but you talk about RAG, I think people understand fine. You chunk up the code in a super, super smart fashion on it, and you make the right code available at the right time.

So the system can make decisions. The counter to that is to train on actual customer code so that everything is just in the model. Why choose RAG and maybe I'll even preempt. They know like part of it is customers don't like you training on their code, but technologically, let's say people did allow you to train on their code.

There are technical reason to prefer this, some real time fetching on it over actually taking that code and lumping it in. So the model knows both the world and yourself. 

**Guy Gur-Ari:** Yeah there are a few reasons and early on we actually did some experiments comparing the two approaches. We found that RAG across the board gave us better results and there were a few reasons for that.

So I think one mental model I've seen that people have about fine tuning, and this could [00:23:00] be fine tuning it on code, or fine tuning on your documents, or fine tuning it on anything, is that I think sometimes people imagine that fine tuning is a bit like a human reading the text and learning from it.

Because when we read a book, there's a lot we get out of that book, even if we just read it once. The models, when you fine tune them on a corpus of data, let's say code, they will scan through it. They might scan through it multiple times. They don't learn things as deeply as humans do when they just read it once.

They will pick up on common patterns very easily. For example, if there's a certain way you like to write your doc strings, and most of your functions have doc strings like that, and then you fine tune on it, the model will be very good at generating the exact format of docstring that you have, because that's a very common pattern.

But fine tuning the model on your API does not mean that the model can now correctly call your API, because that's not what fine tuning does. Fine [00:24:00] tuning just gets the model to try to predict the next token at a time. It doesn't teach it these solving these higher level reasoning tasks of, oh, I was trained on this function and therefore I know how to call this function.

Just as a simple example. With RAG, we can actually get to that. We can actually train the model to understand that when we show it a function, it needs to do a good job when we try to call that function. So quality wise, we get better results, but then there are also operational differences. This goes back to the real time nature that I discussed and to expand on that a little bit again, in a real large organization every developer will work on their own branch and every developer's version of the code base will be slightly different and will be slightly different with my new differences that they just worked on a minute ago and to give them the best possible experience we need the models to adapt to every developer's branch at the state is at currently that if you want to get that to work [00:25:00] well with fine tuning, that would be difficult. That would require more than normal fine tuning. 

**Guy Podjarny:** It's like on the fly training of some sorts, which doesn't quite work. 

**Guy Gur-Ari:** Maybe you have to do something like LoRa, something like parameter efficient fine tuning, where you give every developer their own small adaptation of the model, but then you're sacrificing quality based on that because you don't have as many weights to adapt to the task. So I would say the more you think about if we saw large quality gains from fine tuning, I'd say this, the story would be different, but because we didn't see those.

And operationally, it's a lot harder to achieve this per user customization. We felt that RAG is just a much better fit for what we were trying to do. 

**Guy Podjarny:** Yeah. Interesting. Yeah. We had Simon Last on the podcast, co founder of Notion talking about Notion AI and give a different lens on how fine tuning is tricky. It was generally disillusioned, if you will, by fine tuning. It's really interesting what you talk about. Almost like they become a bit of a drop in the ocean, [00:26:00] right? And it's a way that I'm interpreting what you're saying is you have a lot of information. You just added more information.

It has a certain amount of priority over what you've learned before, but it's not a lot of priority versus RAG in which it gets a lot of priority. It was pointing out a different problem with fine tuning, which is it's very hard to debug. You have this magic box, you ask it to do something and hopefully if you gave it a whole bunch of good examples on it, you keep your fingers crossed and you hope that the magic box spits out the right answer when you ask it, and maybe it does it statistically more, but when it doesn't, you're at a bit of a loss.

It's like, why didn't you get that correctly? Which I found very insightful in that conversation. He was comparing it to agentic processes and agentic processes, it's easy for me to see how fine tuning is much harder to debug than agentic process because agentic process, it's still a process as a developer.

It's easy to relate to say, Hey, this step in this workflow phase failed. There's some, sequence of steps or some terminal output that I can look at and say, Hey, here, it failed. When you're using the product and using RAG, can you debug, is this something that [00:27:00] is within the realm of someone using Augment, maybe even not the developer, but the owner, whatever it is, the admin in the organization that owns it. Are they able to debug when the wrong data gets pulled in at the wrong time? Or is that more, hey, the product should take care of this. This should be outside of like the customer's needs.

**Guy Gur-Ari:** Yeah we feel like this is on us to solve. And so we love all feedback that we get from customers, especially when people say, Hey, in this request, I had this completion. I had this chat. It got it wrong. And so then we will dig in and we will. The first question typically will be was this a retrieval failure?

Was this a generation failure? And so it's a bit like the agents where you can go and figure out which step failed. You can go and check that did the retrieval pull in the right thing? And if it didn't, then the failure is there. And if it did, then the failure is the next step where the model couldn't figure out what to do with that information.

That's the most basic triage of a RAG system. [00:28:00] And so we have all the tooling to do that. Yeah we don't generally ask the users to do that. I have not heard that idea before. That's an interesting one. We haven't gotten requests for that, but generally we try to make users lives as easy as possible.

**Guy Podjarny:** You still need to provide the magic on the exactly. Yeah. So I guess the the market question though, when you look forward, it, it seems very clear to me and also, the product shows it. That indexing your code base and extracting it with RAG and a lot of other smarts produces good results.

However, you're not the only one in the market that claims to be indexing the database and bring in those RAG results. And, we hear it from a lot of other players. And I guess I wonder, like, how do you see the future of this? Do you think? The approaches in it, thinking about your own specifically are sufficiently different in how you would approach RAG.

How would we do that? That really there would be one path that would be very hard to mimic and this might win it and would be a breakout [00:29:00] level. Like Dion, you mentioned Google as the example, it was massively hard to mimic what Google has done. And yeah, everybody did search. Some did it correctly.

Or is this a momentary next 2, 3, 5 years? You have those opportunities, but over time we should expect RAG is fundamentally a good enough search of your code base and retrieval of it. And that really you see that as an entry point and you would build powers elsewhere to be able to build out from them.

What do you think we should expect as an industry? 

**Guy Gur-Ari:** Yeah, to me it's definitely more of an entry point. I think it's an area where we have an edge right now, as far as we can see from what we're testing and from what customers are telling us the feedback we're getting on even just this week, I know we are in an eval in a customer where we're being compared and the thing I'm hearing is yes, Augment is in a different league in terms of the productivity we get out of it, thanks to the context awareness. So these are things we are hearing regularly, but my guess is that, yes, this is [00:30:00] probably temporary because just the speed of progress, the rate of progress in this field, how fast things are moving, how fast models are getting faster. Honestly, I'll be pretty surprised if in a year or two RAG is the thing that differentiates I think it will be other things to give one example of something that we are preparing to roll out soon.

This is the sort of the other side of RAG. So the side of RAG we talked about is as the input to the models to give you better predictions, right? But the other side of it, which is as important as the ability of the models to actually go in and take actions across the code base, right? So we're not talking about, or your cursor is right here.

Give me a few lines of code anymore. And also this is users expectations at this point. Please make the changes for me across the code base. And so that's now a different prediction task where you need to understand based on the task in front of me, not just what do I pull in to do a good job, but where do I go in [00:31:00] this large complex code base to make these changes.

So that's an example, I would say of a component and a feature will be rolling out soon. Which again requires a whole lot of research to get to work and is not something you can get with off the shelf retrievers or even as far as I know, the best retrievers you can get right now.

And so I believe there will be many of these things, many extra pieces, many extra models that will combine into one system that provides the best possible user experience. We haven't figured out what all these things are yet, but there are clearly these other components that are needed to automate.

To get the most value we can out of AI for code. 

**Guy Podjarny:** Cool. Super interesting. I think you've veered into models on it. So I think that's an interesting question. So you're not training the model, you're doing RAG. Again, simplistically put, you can do RAG on top of any model. And pull it out.

Maybe tell us a little bit about what is your approach to models? And and why are you building your own models? And I think [00:32:00] you have a bit of a history over here, learning. So just be interesting to hear how your perspective has evolved over time. 

**Guy Gur-Ari:** Yeah, so first at a high level, it's a mix.

We are also actually training models because it depends on the feature and what model is driving it. But for most of the features we do end up having to both train models and do the retrieval for it all to work together. So there's actually a substantial amount of training that happens.

The thing we don't do, though, is we don't train models from scratch. We don't do pre training. And that was a bet we made very early on. When we started open source models were nowhere near what they are now. The best, even the best open source code models were quite bad compared to what we have now, but we made the bet that they will catch up and they will become excellent.

And the reason we made that bet is because. I guess one reason is at Google, I saw what goes into pre [00:33:00] training models. And at that point, there was almost a playbook. You had to go get data, you have to clean it, deduplicate it. You need a bunch of GPUs and need to train on them.

And you need to watch out for GPU failures. And there's a whole list of things you have to do. And it's very challenging, but there is a playbook. And if you follow the playbook, you will get a good model out of it. And then the game becomes, okay, how can I squeeze more out of the model? How can I improve a little bit more on data quality?

And all these things matter a lot. But I'd say based on that the bet was open source can do that. It's happened much faster than I expected and to a greater degree than I expected. 

**Guy Podjarny:** That seems to be a recurring pattern in the world of AI. It's always faster than we think. 

**Guy Gur-Ari:** Absolutely. So open source is like that as well.

We've fairly quickly got access to extremely good base models for code. So these are not models that you can take and put in your product as is and they will drive your completions. So just to give an [00:34:00] example or completions or any other feature to give an example. Besides RAG, there are other things that are important.

If I'm getting a completion around the cursor, the model needs to take into account what's before the cursor, what's after the cursor. It needs to give me a completion that stops at the right point. This is not something that any off the shelf model that I've seen can do well currently. And so there are things like that, that we need to train the models to do well which go beyond RAG.

RAG is of course, like a big piece of it. But the thing is the better, the base model you start with, the better the pre trained model you start with, you can put that extra training on top and you can get a much better model at the end. And so our approach is that we keep up with the best open source models.

They still are coming up regularly. And it's very nice to see that after the general purpose models. The next thing you get are the code specific models. This still seems to be happening, which is great for us. And so we will do a lot of training on top of open [00:35:00] source models, but we don't do our own pre training and then we will do RAG.

**Guy Podjarny:** Yeah. Yeah. I'm curious this is a bit of a philosophical question, I think to both of you, Guy and Dion. I guess in the world of true open source, because we say open source models, I think like probably open weights models have oftentimes been been described now like this, but in open source, there's a way to contribute back.

So as an ecosystem, you can talk about some piece of software that is open source that becomes better and better. Because once it gets Kubernetes or Linux or something like that because a bunch of interested parties that are all sharing the use of this, they come in and they contribute back.

In these open weights models, as far as I understand, it's very hard to contribute back. You can report bugs, you can do things, but, it's not actually open source. You don't typically, or ever, actually see the data set just available to you and say, hey, I'm going to correct this data or add to this data. Do you maybe two in one types of questions? Like one is what would make these open source models [00:36:00] win in the long run? At the moment feels a little bit like they're at the goodwill of the business strategy of certain organizations that are publishing them and hosting it.

And then related to that is if you do want to engage and tweak bug fix or contribute back to the open weights models is that possible? 

**Guy Gur-Ari:** Yeah, this is a good question. So I'd say what will make them win? So to me, this is more, of course it's competitive in some sense, but the way I think about it is more as an ecosystem of models.

We have different models driving our different features. There's a whole spectrum in terms of the size of the model, the latency you get, and how do you match that to the appropriate feature? And how specific is the feature? How general is it? And sometimes we use closed sources, closed models.

Sometimes we use open weights models. I think it's a good distinction. Open source versus open weights, because indeed we don't, for almost all these models, we don't have access to the full training recipe. So I would say in terms of who wins, yes the reality is [00:37:00] that overall staying at the frontier or being close to the frontier requires more and more compute resources, and so more capital over time.

We keep seeing optimizations, which is natural because once something works and the quality is good, usually you go back and you find like a whole list of things you can improve to do better, and that includes training. And so we are regularly seeing small models coming out that are performing as well as models that were 10x larger before.

And so that's still often takes us back to the regime where. Oh, this task that used to be out of reach for many groups because they didn't have a lot of compute. Now you can get that with a smaller model. We saw that again and again. It happened with Llama, I would say. It happened with Quen more recently.

It's still happening. There seems to be still a lot of room for optimization. But overall, yes, I think it's true that to stay close to the frontier, it's going to require more and more capital resources. On the smaller side of the [00:38:00] models I don't really see a reason why folks can't stay on the frontier.

Because if your latency requirements are fairly strict then for a while I believe we will stay on smaller models. And so then it also is less effort to train them. Also the smaller models don't benefit as you can throw a hundred X more data at them, but it's not like they can learn all that data anyway.

**Guy Podjarny:** So all the compute resources are, but to challenge that a little bit, guys, like all of those are the reasons why if I'm building a model and then I'm opening it or sharing the weights of it, it is less expensive for me to do it, but it's still expensive for me to do it.

Like I, I'm still opening it. And so I'm curious about the sort of the second part of the question, which is how do you contribute? How do you benefit from a community using an open weights model? And maybe it's a question even like specifically at Augment, like, how do you contribute back to those models when you decide to do it?

I'm not asking how often you're doing it. That's a sort of a business choice, but when you do it what are even your [00:39:00] options? 

**Guy Gur-Ari:** Yeah, the options. I'd say we think we'd love to contribute back because we've benefited a lot from the open weights models, but it it's fairly limited. Our options are very limited.

I think it's more of a research. To me, it's more of a research problem. Once you're handed the weights, you can push them forward by continuing to fine tune and making the model better on more specific tasks that you care about. So you can do that. But going in, fixing things, let's say I found that the model makes some mistake, right?

And it's a recurring pattern. And whenever I show it certain kinds of inputs, it makes that mistake. If this was an open source program, I could go, I could fix it. I could submit a pull request, right? That's easy. But with models, we don't have that level of control over the models. At the end there's still a black box and there's research certainly on interpretability.

And I think some of that research is aimed at can we go in there and figure out which neurons [00:40:00] control which inputs and which outputs? And when there is a problem, can we go change those particular neurons and fix them? That research is still at an earlier stage than just our ability to train a black box model.

The best path I see for us to be able to contribute back is if that research pans out and we get a set of tools where we can start submitting patches to models. It'd be great if we had the weights, we could fix a problem and then we go and submit a patch. It's not a source patch, it's some patch to the weights.

But just apply this patch and your model will get slightly better on these problems. That would be analogous to what we see with open source. We don't have that ability right now because we're limited on research. 

**Guy Podjarny:** Yeah, indeed. Dion, do you have any views on this? 

**Dion Almaer:** Yeah, I think I'm very happy that we don't call non open source models, even when some companies try and sneak that into their releases, but there's tons of value in the open weight models.

So I don't want to dissuade that from happening. And you do see things where people just more into the [00:41:00] are able into the overall ecosystem of contributing various tools around these things because they feel like in general there's more openness and availability there.

So there's nuance, right? There's a bunch of open source projects where I know that if I put a PR in, it's not going to be accepted. It's open source, but it doesn't actually matter in that case, right? So I think there is a bunch of open nuance to all of these things. But it's been nice to see the direction that we're going in, just the raw competition in this space is the thing that matters, I think, more than anything else. So we're not locked in. 

**Guy Podjarny:** Yeah. Yeah. We definitely benefit as a community from open weights models, including Augment, like a good example of that. And so I'm happy for doing it. There is open research collaboration on it, but that ironically also happens with the closed models on it. I don't know if it's more or less common amidst open weights models than the closed ones, but that's just interesting. Like I, I find myself pondering the Especially when we do call it open source, [00:42:00] which is a natural I'm not fussed about the fact that people, whatever, use the sort of the slightly inaccurate term for it.

But I more think if you call it open source, the reason we keep saying open source will win is because the community will come together. And this is actually a place in which the community cannot fully come together. They can encourage, they can support an organization. But eventually there's some amount of if you're using Code Llama or M3 or whatever you're at the mercy a bit of. The continued strategy of Meta to publish the next version after they spend whatever many million dollars on yeah. 

**Guy Gur-Ari:** Maybe to add one comment to this, let's say there are open source models.

There are models that are completely open source. You have access to the training scripts. You can see their training logs in 1DB. You see all their training data and how they collected it. It's all there, right? They're fully open source, but still, if I find a problem with that model and I want to fix it, the only thing I can do is go back to the training data [00:43:00] and change it.

Assuming I understand this is the same problem you mentioned before with fine tuning, if you see a problem, you have to go back to the data and change it to fine tune and it's all very indirect. And then I have to spend all the time and effort to train the model again and check if I fix the problem. So even if I have access to everything, I still am very limited in what I can contribute.

It's a huge project to submit like a bug fix to the model. So that to me is really what it boils down to. It's wonderful when people, and we benefit from that as well. It's wonderful when people open source everything, very useful. But we're really gated on our ability to go and patch and fix these models.

**Guy Podjarny:** Yeah. It's a good point. Open weights versus sources one delta, but there's also just the sheer scale. And interoperability that get in the way as well. So yeah, this conversation has been great on it. And we're flying through time over here in conversations.

It clearly that Augment is a very powerful product today and it goes in and it helps the human developer write code in the IDE. And today I believe it does all the [00:44:00] completion. It also does chat based generation on it. And a lot of these patterns are emerging amidst coding assistance and amidst users of coding assistance to learn what to expect.

Let's cast our eyes a little bit to the future. And I guess for starters do you think people will still need an IDE completion solution on it in call it five years time, like when you think about Indeed, the journey towards more autonomous coding. 

**Guy Gur-Ari:** Yeah, that's an interesting question.

Five years in this space is like eternity. So everything I say about a century, 

**Guy Podjarny:** roughly a century. Yeah, it's it 

**Guy Gur-Ari:** feels like that. And so this is completely speculative. Let's say it like this. I think on, it depends on the task you're trying to solve, right? If you're trying to solve something that's on the simpler side.

Let's say in five years some percent of issues coming into workspace, are they going to be automatically fixed and patched by models? Yes, I think that's essentially a guarantee. But now the question is of these issues that are coming in, what is it going to be? Is it going to be like [00:45:00] 20%?

50%? Probably depends on the project. Is there going to be still a good amount of issues that developers will have to go in and dig in and fix themselves. I would say again, I really like the self driving analogy because, it's not that it's a simpler task, but at least the action space is a lot simpler, right?

You have your steering wheel, you have your pedals. We still have not solved full self driving, but we're getting a lot of value along the way from more and more automation. Coding is a richer task in terms of at least the action space. The reasoning is different. There's definitely different kinds of reasoning that go into it.

What I expect, my guess is that, yes, I think completions will still be relevant from what I'm seeing so far, the better the tools become, it's not that I end up just doing the same work that I did before faster using these tools. And so using less completions or less next edit or whatever it is I'm using, what ends up [00:46:00] happening is that I actually end up accomplishing a lot more.

And I ended up using, yeah, I use some amount of time. Let's say in chat, I use some of the amount of time and other features. I still use completions a lot because it still happens a lot that I need to go in and make changes manually. And when I do that, sometimes it's a lot of code. And when I do that, I really want to have completions there with me and I want them to be context aware.

And I think that's especially true in the larger, more complex code bases, because those are the ones that are going to get automated the slowest, because those are the hardest to deal with. So five years is a long time but I still expect these tools to be around for a while, even as we add more and more automation and let people write more and more code using natural language, which is like very clearly the trend.

I expect these tools to all live together. 

**Guy Podjarny:** I guess though, if I let me try two attempts at trying slightly longer timelines. So if you're a software developer today a junior to mid level software developer, [00:47:00] what skills do you need to build out to be successful in your career, given evolution, given the anticipated growth in ability of solutions such as Augment. 

**Guy Gur-Ari:** Yes. So I think I would point to two things that seem to me to be essential and will continue to be essential for the foreseeable future. First, I think for the foreseeable future, these models will keep making a lot of mistakes. The software systems that we have are already incredibly complex.

And the more AI automation we have, they will just get more and more complex. And they mess up. The AI systems mess up a lot. And when that happens, there's no replacement to having a deep understanding of how the system works. I don't think that's, I don't think that's going to go away. And so in my experience, I can use augment to crank out hundreds, thousands of lines of code.

Sometimes it all works great. It all works fine for a while and then something goes wrong. And when something goes wrong, I [00:48:00] still need to have the understanding of how the system works. What is it built on? Full stack understanding often of what things are for me to be able to go in and fix things when Augment can't fix it for me.

So I don't think that depth of knowledge, of course, has always been important in programming. I don't think that's going to go away. On the totally other end of the spectrum. It has become very important. I'd say to understand these models and understand how to prompt them to get the most value out of them.

We are still at a point where the thing cannot read your mind. And so there's certainly a lot of tricks into it's a whole skill set. How do you get the most value out of these models at this point? And of course, us and everyone else are constantly trying to make that less of a challenge and make it as seamless as possible.

But still the models, the more you learn about the models and how they work and how to get the more value out of them. That skillset will, I believe for now just become more valuable over time. So these are like two very extreme ends of the spectrum that I [00:49:00] think a developer entering the field would benefit from.

**Guy Podjarny:** Yeah. We might be nitpicking a little bit on timelines, but I think both of those things, while I a hundred percent agree with them today. I guess the question is, what's their longevity? Like the models and the tools around them should really get better at the prompting. Like they're never going to read your mind, they better understand the context, they better understand the surrounding.

And so it should be easier to prompt. And so it's almost if you invest right now in prompt engineering, a lot of these things you're going to need to unlearn. This has already been proven, right? If you were really proficient at I know in image models, it's easy, right? If you got into your diffusion models and into your stability, And then you got into midjourney and just use the English, today.

But, and I think on the other side, like the systems, wouldn't Augment be dramatically smarter in five years time and it's understanding of the ecosystem, like how many people on the team would really have a better understanding. 

**Dion Almaer:** But don't you need to break down systems and know how to elaborate and.

That's always going to be true, right? 

**Guy Podjarny:** It's just you're going to understand, not so [00:50:00] much remember the whole system, but rather know how to engage with some presented knowledge of the system. 

**Dion Almaer:** Exactly. I guess 

**Guy Podjarny:** The extreme question that I have that I, like maybe literally a bunch of us parents would be facing is.

Would you recommend to your kid right now to go do a CS degree? 

**Guy Gur-Ari:** That's a hard question. I think my answer to that question changes at this point, almost on a monthly basis at this point, I am not sure. I am not sure. It's really hard to predict because on the one hand yes, I agree with everything you said.

It will, of course, get easier. That is what I'm working on every day, right? It's just 

**Guy Podjarny:** we're in this industry and it's a good thing. More people will create more things more easily. It's a good thing. It just has impacts on the profession. 

**Guy Gur-Ari:** That's right. Yes, exactly. It's all good for society. Also, just on the comment on prompt engineering, just to clarify, yes, I don't think people should go and study to become prompt engineers.

I guess maybe to summarize that point, what I'm trying to say is just I think what happens a lot is [00:51:00] people will try the models and get something out of them. Sometimes it's good, sometimes it's not good. And they will assume that, yeah, this is the value we can get from the model. There is more out there.

That's really my message. Push it, try to push it in all kinds of directions to learn what the limits are. I think often people will try it out and they'll say, okay, this is it. This is where the level of quality is. And yes, that will become easier over time, but you can get a lot out of these models if you try to push them on the question of the CS degree so first of all I do believe in humanity and I think getting a CS degree for the purpose of learning deep things about the world is still a valuable thing to do.

I think in terms of a career choice. I have to say I'm on the optimistic side here because I think implicit in these discussions is this thought that there is there's this assumption, maybe implicit assumption that there is a finite amount of things we were going to want to build. 

And with AI, we're going to automate it, and there's [00:52:00] going to be nothing left to build. If I look back at the history of programming, every time a new programming language came out, the whole point of it was to automate away the chores and the toil of the previous language, make it easier to write software, right?

But at the same time, the amount of software we write and the complexity has just skyrocketed. It's not that the new languages have made us like, allow us to just do our job in five minutes and go to the beach. No we could build more and bigger things. So this is complete speculation, but my belief is that what we will see is just far more advanced, complex and useful systems that will come out of AI and that there will still be room for humans to, to steer these systems.

And yes, that I do believe you will still need a deep understanding of what happens. Now, I think one data point that will make me rethink that is if we manage to actually solve self driving end to end and completely get rid of that task fully then maybe, yes, maybe I [00:53:00] can see how other real world tasks can succumb to that as well.

And maybe it will happen. 

**Guy Podjarny:** But overall, For example would you want someone Would you invest in, whatever, becoming a taxi driver right now or a professional driver? 

**Dion Almaer:** I think the spectrum of developers is so different than driver, though. There's certain developers who I would trust to contribute to the Chromium or the Linux kernel.

It's like a small subset compared to the ones I would trust to build a web app. They're not the same things, right? There's very different levels of things that are going on. So what does my kid want to do? What layer of abstraction do they want to live in for this, right? What kind of CS degree are they actually getting, right?

There's many different types now, right? So I think there's, I definitely would love my kids to make that choice, if depending on what they wanted to do, And, I want to democratize this for them and I want them to be focused on solving real problems. And so being able to live at higher level abstractions, to build bigger things, [00:54:00] like that sounds great to me.

It doesn't seem like the worst degree to get. 

**Guy Podjarny:** Yeah. Yeah. I'm in a just know that I'm in the optimistic camp in terms of there will be things to build. And I think being a software creator is something that we will need in a decade, in two decades, in five decades. Or whatever solution creator on it with software.

I have a lot less conviction around the computer science degree being able to keep up with the pace of change. There I'm like a very very much a short seller. I am not optimistic about that being the correct training. Even today, I'd say it's dubious. Yeah, if you're gonna go and do a PhD in machine learning, that's somewhere in computer science and math is probably a good place to be if you're going to go build even contribute to the Linux kernel or the equivalent, future crazy world of AI power development.

Even then I'm not certain that a computer science degree would give you the tools that you need to be doing that type of work in a decade or two. So I guess we won't know, but it's a very real [00:55:00] question, right? Cause our kids will be grappling with that. 

**Guy Gur-Ari:** It's a real question. I think it's trying to predict both like the technology advances and the job in some sense. You're trying to predict the job market situation which depends on the technology, depends on how much stuff we'll need to build the competition maybe if I stay on the optimistic side, I would expect CS degrees to evolve. Let's say, because if there is room for people to create software and it does require a deep understanding of systems, then I would hope that the degrees do evolve to match that.

But yes we're going to find out. 

**Guy Podjarny:** We'll find out. Guy, Dion, this has been excellent, the conversation on it. I feel like we can continue it on, but I think we're starting to Really stretch the attention span of the listeners for a single episode, maybe Guy, just one bit, if people want to check out Augment or try out, what's like a one call to action you'd give them.

**Guy Gur-Ari:** Yeah, go to augmentcode. com click get augment and start coding reach out to us with any feedback, good or bad. If you want to talk to us if you have feature requests, if you see things that you see you would love [00:56:00] to have in the product. If you just want to chat about AI for code reach out, we'll be happy to chat.

We love talking to our users. 

**Guy Podjarny:** Sounds good. Awesome. This was great. Thanks. Thanks again for coming on to both of you. And thanks everybody for tuning in and I hope you join us for the next one. 

**Simon Maple:** Cheers. Cheers.