2024 Year in Review: AI Innovations and Predictions

**Simon Maple:** [00:00:00] You're listening to the AI Native Dev, brought to you by Tessl. Hello again and welcome to the AI Native Dev my name is Simon Maple 

**Guy Podjarny:** And I'm Guy Podjarny and happy new year. Happy new year to everyone. First episode of the year. 

**Simon Maple:** I know it's scary, right? 

**Guy Podjarny:** Actually. A thousand to go. 

**Simon Maple:** There was a, there was an episode that would drop today. Mason Baker.

So that's true. Officially second episode of the year. The first live episode. The beauty of recording. Yeah, absolutely. That's true. That's true. Yeah. That was a hard episode to record because we kept having to say. Next year, this year, last year, it's a tricky one because it goes out the wrong side.

Yeah, especially in startup terms. Absolutely. Yeah. So yeah, welcome, welcome to the 2024 review episode where we're going to be talking about various I guess changes that have happened in the AI space as well as some of the roundups of our episodes that we had here on the podcast from some of the [00:01:00] amazing speakers actually that we chatted with over the second half of last year, because of course, we only started this podcast on July 9th, 2024. But in the short six months we did 33 episodes. 

**Guy Podjarny:** I think that's the definition of prolific. Yeah. Although we did get at least one podcaster gave us advice to do three episodes a week.

So relatively speaking we're tame here. 

**Simon Maple:** Yeah. Consider this. We started slow then. So of those 33. Which do you think is the most popular episode of all of those, 

**Guy Podjarny:** the most popular, not the best, not the most popular. 

I would say maybe the first one with the desk, not the first.

So the second one with Des, I think that one would be a candidate. I guess my second guess would be maybe Tamar from Glean. Yeah. Yeah, those will probably be.

What would your third be?. Be what? The way third, neither of those hit you wrong. The last one notion the notion of last Simon Last. 

**Simon Maple:** Yeah, that was, and I think because it was such good real world advice with Notion AI.

I think it [00:02:00] struck a chord. So that was, and it was very like 

**Guy Podjarny:** a kind of, no, no fuzzing around on it, like straight up learnings from how to build Notion AI, which is also like a really far along tool on it. So it's not that surprising, but yeah. 

**Simon Maple:** And I'm sure we'll quote many of these episodes.

So many quotes. 

**Guy Podjarny:** It's so like probably the hardest part of thinking about this episode or this sort of summary view is there's so many great guests that we had and feel so privileged about it. What was some of your favorites then? I know you just said it's hard to choose. I'm going to make you choose Guy.

I think probably our summary episodes when we get together. Yeah. But yeah, those are really fun. Those are really fun. Definitely would have to include the Des Traynor episode that was early on. First of all, he's just such a, has such amazing soundbites, and so it just has these great quotes that stayed with me that I think I'll probably, I continue at this point, they became part of my vocabulary.

So that was really good. Really liked the episode with Armon. I think he just, he really organized, he's such an organized thinker. He's the founder of HashiCorp and he thinks about infrastructure as code and modern infra. And in [00:03:00] really precise ways, every question had a, Hey, here's how to think about it.

Here's the picture. And then here's how AI fits into it, which I loved. Always love talking to Tamar from Glean. Maybe that's why I thought that's in Tamar's the popular episodes. And she's just such a brilliant product thinker. So I love talking to her and how she thinks about, we talked a lot about evaluation at LLM Judge.

Yeah. And probably like the one that was most like interactive and fun for me was the one with Caleb Sima, where I got to talk about security, where I know a thing or two as well.

But really, like I've enjoyed some of these, like it feels again, feels spoiled with riches here of all these amazing people that want to come and share their learning on the podcast. How about yourself? What were yours? 

**Simon Maple:** Yeah. Yeah. Similar actually. There's so many amazing people that we talked to.

It's lovely to have an excuse to ask to, Hey, can we just chat for an hour? I think. Although I talked to Amir most weeks spending time with Amir at Shabbat to talk about the categorization of AI tools. And of course he's an investor and really thinking about the various areas in which there is going to be growth and understanding which tools and which startups are doing some real [00:04:00] innovation in that space.

That was a great episode. And I feel like yeah, 

**Guy Podjarny:** Another order from chaos. 

**Simon Maple:** Absolutely. Absolutely. Yeah. And also Amara Graham who's a DevRel at Camunda, really interesting the way they're using AI for to allow their users to essentially gain access to ask its AI chatbots to essentially look through into the Cominda docs to grab answers and things like that.

Great, yeah. Really good usage of real world AI with their users. A couple of others. James Ward. It's always great. I've known James for many years, but it's great to actually have an excuse to say, Hey, James, let's hang out for an hour. And so that was really good. James had just moved over to Amazon.

Yeah. Q developer. So that was an interesting one. And I'm gonna be cheeky with the session today that was released today. We recorded in 2024. So I'm gonna put Macey yeah, Macey down in the list of 2024 for the, yeah, in the list. That session was a really hands-on. We kinda went through a number of tips with prompt engineering and how you can get the most out of your LLM in one of the cheapest ways, which is just altering your prompts.

So that was a really fun very natural [00:05:00] chatting session. Yeah. So I love that. I love that. 

**Guy Podjarny:** Just also Macey is just awesome she's great. Yeah. It's hard to, yeah, there are so many conversations. I will say we, we started the podcast thinking a little bit, maybe idealistically that we're really focused a bit more purely about tools that help you develop software with AI.

And we did find ourselves leaning or including a fair bit about how to use AI in your products and build products on top of AI because the two topics are so intertwined. And and I think that's interesting. That's the learning in its own way. Just, how intertwined. The thinking about AI as a part of your product, as a part of your tool stack.

And whether it is that you're building a product for someone else running on top of ai or whether it is that you're using AI to develop software and where does it work reliably and yeah, where does it not yeah. 

**Simon Maple:** Yeah. Interesting. So that's the favorite speakers what about the least favorite speakers then of 2024?

**Guy Podjarny:** I was going to say the joint episodes or the Monthly ones. 

**Simon Maple:** Oh, come on. You always told me you look forward to this. It's like a love hate relationship. Yeah. Absolutely. Yeah. So let's look back on 2024 then some of the [00:06:00] themes obviously this is the AI Native Dev Podcast, we should start talking about AI native.

How do you feel like the general impression acceptance? How have people been thinking about AI native going forward versus the today, whereby people are more thinking about that AI assistive mode, thinking about how they can use AI in their current workflows seen much change there.

**Guy Podjarny:** Yeah, I think so maybe like thinking a little bit about these sort of six months or maybe even like full year, not just on the podcast. I think earlier in that journey, so the beginning of the year, middle of the year, a lot of it was just about which tools, what can I even do with AI and DevTools?

And I think a lot of the time we spent on the podcast was indeed classifying with Amir the tools to begin with and then talking to all these kind of brilliant AI DevTool builders that are trying to figure out what's important. What is a coding assistant? What is a test creator? What does it mean to do doc generation with AI?

How can you use AI in DevOps? And I think the first section of the year was very much around understanding these tools. I think a lot of them are still quite nebulous. They're still okay. I focus on now Qodo, [00:07:00] right? But before Codium, when we interviewed them talk about creating tests, but really their suite is broader.

They're trying to figure out what is it that they do. They might anchor in tests, but they also do all sorts of code completions and creations in it. Similarly, I think a source graph, similar type of Might have come from kind of code search or from code completion, but they also do some of the tests.

So I think a lot of the kind of first part of our, maybe this is like Q3 really in the conversations, before the end of the year, a lot of it was just an understanding. What are the categories? What are the chunks of tools and really seeing these blurry lines between those definitions?

I think even today with a lot of these conversations, it's still hard. And I think that's true even for the vendors themselves to define where those lines are. If you're a developer and you're saying should I use AI for say doc creation. It's still not clear what would motivate you to pick a dedicated tool, sweep or whatever or versus, should it be a part of a suite that understood your code base?

And I think [00:08:00] that question remains open in the definition. 

**Simon Maple:** Because there'll be an amount where someone's going to say, look, my core, the core tool I need is this, whether it's code generation, code completion, et cetera. And do you know what? I want to effectively. use that same tool because I think it's just good enough.

It's not exciting, but it's just good enough for maybe docs or tests or something like that. And then actually this other space, whether it's performance or a little bit of ops or some monitoring or something else that I need, I'm going to use these dedicated tools. And it's, I guess it's whether or not we will see some of those categories elevated in terms of these primary categories that people want dedicated tools for and which ones are more the categories whereby it's good enough to use some of these other tools across multiple ways.

And I guess as a founder who's owned companies that have broadened, for example, you are the founder of Snyk, but when you used to work at Snyk as a CEO and, really looking at which products to expand, how much do you think a company is almost hurts [00:09:00] by expanding too fast versus gets that value in focusing on one specific area. Cause that's what is challenging every single AI dev tool today. 

**Guy Podjarny:** I think the primary difference. So I think some of it is that, and there's no single answer to that, right? You have to really, it almost always, the answer is come back to the user, come back to the what is a user looking for?

What is a developer interested in when they're looking at a tool? And so for instance, for security at Snyk, it's okay, do you really want a different tool to whatever secure their open source libraries and their code suck? No, they want one. And so there was like some urgency to expand. I think the challenge with AI is that the users don't know what they want yet.

Like the tech itself is not entirely clear what it is that it can or cannot deliver. So there's a question over there. And then subsequently users don't really know what they can and what they want to get out of these tools. They want perfection. They want something that just does everything.

Except they don't really because they still want to do the fun parts. But also they don't know what to expect of the tools and it's okay that they don't because the whole thing is brand new and the tool developers themselves they're also, they're [00:10:00] tempted by what they can do versus some definitions.

And so I guess I would think we're going to talk about predictions at the end of the year at the end of the episode. But I think a lot of it comes down to, we're still learning what are the right categories for AI DevTools, what's the right center. And we talked about this in one of the monthlies in which you do think about, is it really revolving around a magic box that would understand your code and its surrounding and based on that, it would offer all sorts of tools around it. That's one approach and some platforms like augment, I think is an example of that. Github is trying to do that. So Hey, we'll do everything. We'll understand your code. We'll go from there. Versus the sort of the specialized as well. Testing is its own thing.

Docs are their own thing. They're big, complicated domains. You need a full on product on it. We're going to be the AI powered version of that does it very well. So I think we'll still see a lot of those definitions. I think what we also see is that there's a lot of a growing consensus around maybe this sort of gap between building a demo of those products [00:11:00] and building something real. And with that maybe I'm biased. Clearly we talk about spec centric development a fair bit. But maybe an understanding that we need to get to the next step. Like step one is this magic. Hey, I type in this prompt, I get an application is magic.

I still feel it's magical. I live this and still feel it's magical. How it produces these applications out of thin air. A lot of people over at Christmas, we're creating a lot of these sort of fun apps and all that. And you can create those with bolt.New or with Claude's artifacts or things like that.

And that's amazing. And then you ask them to do something like whatever, make a website responsive. This is an example a friend told me and it becomes responsive. And then you ask it to do something else and it doesn't make that responsive on the website. There's no place in which you can define the application.

There's no anchoring in truth. There's nothing to build on and so it's magical. It's magical. And then it just reaches its limit and it's done. You can't really do much more beyond it. And I think there's appreciation that you need something better. You need some spec, you need some grounding in truth, you need some definitions to be able to go further with it.

I see a lot of acceptance of that. And you hear that also [00:12:00] in maybe in like aspirations, you hear, here on the podcast. You heard Jason Warner talk about that as a destination. Simon Last in that last episode, I think that I had in 2024 talked about how we asked him, what is the sort of the future?

What do you want to get from AI? And he talked about a spec centric. And I just define my requirements and some tests and some examples and build the application based on that. And even I heard the Cursor folks talk on the Lex Friedman podcast about how they think that's an aspiration.

So I do think that alongside the mess that is current AI dev tools that we will continue seeing. Powerful tools, a mess for a reason because it's new. We're also seeing an acceptance that there's another layer that was missing around that spec centricity. We're still not entirely sure how to get there. We ourselves at Tessl are working on that as well.

But I think there's acceptance that we need that to be able to build bigger on the AI Dev. 

**Simon Maple:** Yeah. Very interesting. One episode that we, Not end of the year on, but I think we were a couple away from the end of the year [00:13:00] was a mashup episode around evals. And we did that because it was a familiar topic across a number of different episodes where people were not saying the same thing necessarily, but that they all, mentioned eval as one of the key activities or evals is one of the key activities that they do from the very early stages, you mentioned Des Traynor in fact from some of the early episodes like that, even we're hearing this more and more, even towards the end of 2024 and beyond.

Why is this? Why is this grown in its importance for people running evals across models and across models as well as across versions of models? 

**Guy Podjarny:** Yeah in versions of their software, right? I don't know that it's it's not new because of the models. It's just new because as an ecosystem, we're learning.

How to build products on this sort of craziness that is these LLMs, but also the determinism as well, because we don't know how it's going. It just, it doesn't like, so I love the Des quote of, you don't know if your product works, it's not a very sophisticated quote but you basically don't know.

It's get used to it. You don't know if your product works. And really from that point onwards, which, fine. Des went on to talk about how they do [00:14:00] some of the regression tests and they have their torture tests, which another term that stayed with me of the deep test when you're making something as a big change.

so much. And so the evaluation all the way to the Simon Last episode when he talked about how at Notion there's a lot of, vibe checks, there's a lot of, he has an idea for doing something. And with AI, you can do it in 10 different ways and they're very different.

How do you do it? And to think about how do you build regression tests that include a bunch of these problematic cases that you need to include to be able to that you haven't regressed, right? That you don't fall back. But fundamentally, them Tamar at glean, everybody building on top of AI comes to appreciate this sort of hard truth that you don't know if your product works.

And that's, I think that's just really hard. No. How do you know? When is it that you can continue? How can you as a user rely on it? I think in the Caleb episode we had a good conversation talking about if you have a product that can find vulnerabilities for you and nine times out of 10, it tells [00:15:00] you you have no vulnerabilities.

And in the 10th, it tells you, you have a vulnerability or even the other way around, nine times out of 10, you'll catch a vulnerability in the 10th one that doesn't. What type of assurance do you have around the thing that has been scanned that you can say anything about it, right? If every 10th user, every hundredth user that encounters the website or every time in a hundred time you tweak the website, it breaks.

How do you deal with software that is unpredictable like that? Is it's tricky, right? It's something that I'm not even entirely sure. How is it that as a society we learned to do it. That's why I think we revert today to these things that you can eyeball and you can quickly see that it was not correct.

Or that it's not that big a deal. Okay. Like a blog post or an SDR email, like you do a thousand of these. It's okay if some statistical portion of them failed. But I think this notion of how do you build a robust enterprise grade product Tamar talked about that a lot.

How do you build that on top of something that is inherently unpredictable these LLMs? It's really tough. And we feel it at Tessl, right? Like we [00:16:00] have a lot of these assessments of Hey we can create software. We can create specs, we can do those.

And sometimes it's frustrating. It's Hey, I just ran it and it worked and now I'm running it and it doesn't work. Running it and it doesn't work. Oh, I run it again. It works. It's like an extreme version of it works on my machine, right? And so it's interesting. Yeah. I don't know that solve problem.

We'll continue to have conversations here to talk about best practices, but I think these evaluations, they try to make it as methodical as possible to say I have whatever it is that I defined a sufficient level of confidence that this product works, that I did not break it or that this new capability works and I can bring it to my users.

**Simon Maple:** Yeah, let's switch a little bit and talk about models because obviously, everyone's always trying to outdo each other with on the benchmarks and things like that. And there's a number of players that are trying to release models as fast as they can so that they're not too far behind.

But also they want to be able to show some improvement as well. So we've seen a number of models rolling out. A number of people on the podcast last year and you can see it across Twitter and things like that. People love [00:17:00] Claude, particularly around code generation. OpenAI used to be that kind of model up was the go to earlier, probably in the year. And it was there was one big jump with Claude 3.5 Sonnet, I think it was. Yeah. Yeah. Is this I guess, from the point of view of these models piggybacking or jumping over each other, we looking forward to something similar continuing to happen.

How do you see the model growth over the last year? 

**Guy Podjarny:** It's a really interesting question, right? Like they have evolved. I think from the beginning of the year to the end of the year, you can say that at the beginning of the year, it felt like there was a decent possibility that there's going to be one or two breakouts that are just going to get some form of gap that is going to be very hard to catch up to.

And I feel like by the end of the year, You look now, you look at the last six months, you look at how many times Gemini or DeepSeek or the o1 or indeed Anthropic and a bunch of their in Sonnet and others jump to the top of the benchmarks. And the benchmarks are [00:18:00] one thing, but also, there's a lot of vibe check in the community and definitely Anthropic is at the top of that list, I think at the moment.

. In terms of infatuation and I think just how many times they take the lead or they approach the lead and that table changes. I feel like today, it's a reasonable assumption that there's no breakouts really. Like once one of these top companies comes along, it's not that, anybody off the street can come along and create something that competes with them.

Maybe in certain countries you can copy the IP and kind of train your model on it for a few millions. There's some suspicions and some models on that. But. Fundamentally, like amidst these higher tier it's just very hard to break out of the pack. And so I think it's a good thing.

It's from a consumer. I think that's a good, it's a good thing. And we want to we benefit from that competition driving all of these vendors to create new things. But I think it also means that more and more products are becoming multi model. They don't want to bet on one tool. And I don't [00:19:00] think we have clear cut definitions of which model is better at what, but I do think that those would evolve some fundamental choices about how these models are built, make some models a bit more say, lyrical versus others, more analytical in code generation. They also have biases. And so I think as a builder of those tools, we increasingly you need to think as we are here and many are to think about how do you alternate the models?

Definitely allow the user to choose and sometimes even mix and match. And I think we'll see more and more of that. 

**Simon Maple:** Yeah, did you expect to see a V5 GPT in 2024? 

**Guy Podjarny:** There's some interesting rumor mill, right? So yeah yeah, there was a GPT five. So these are like rumors and they're not substantiated but you hear them all over the place that that the big models, the Llama 4, the GPT 5, the equivalent in Gemini have hit some form of scaling limits, that they've, run through some sort of big scale, they've cost them some substantial amount of money and and it hit some [00:20:00] wall and they have to go back to the drawing board.

And today you've gone from people talking about from like Sam Altman and the others talking about GPT 5 to talking about reasoning being the destination and talking about that evolution. And I think that's really interesting and might imply that the future for The likes of poolside and Jason Warner is is more compelling, right?

**Simon Maple:** Absolutely. And I think this is, you know, we'll see, if this is something that ends up favoring those specific type models versus the more generics. And I think we haven't yet been able to have that question answered as yet. 

**Guy Podjarny:** And that's what Jason said, right? He said fine, in the world of infinite resources.

Of course, the generic models work and I can scale indefinitely, but we don't live in that world. We live in reality. We do have constraints. And that was his bet. That said, I do want to say that there's a certain amount of delays also that we've seen and some amount of I don't know, like promises that have not yet been delivered, which we can touch on here.

And for instance, poolside, as far as I can tell, is not yet out, right? When we spoke, I think the expectation was that you'll be out in the fall. And, I [00:21:00] think they work with customers substantially, so I'm sure they're making a lot of progress, but they're not public yet. Augment took forever to go public.

So you do see a lot of delay in bringing product to market. 

**Simon Maple:** Very interesting. Really interesting question from Mike Burton here. Do you think that the models are getting better at gaming the benchmarks? Gaming the, surely gaming the benchmarks never happens in the industry, does it? 

**Guy Podjarny:** As a side comment, one of the things that most surprised me as we started building with LLMs is how much they cheat.

At the beginning we were talking and everybody on the team was saying Hey, we'll build with the LLMs. We can't inform them when we give them the spec. We can't give them the test as well because they'll cheat. They'll build something that just answers. I said, no, that's a theoretical problem.

It's not a real problem. Like these LLMs are not trying to cheat. They're trying to, lo and behold, if you give them the test, if they succeed at the beginning, they're good. But if it's a hard problem, we see literally if the input is this, return that straight up cheaters. So really interesting.

If the input is strawberry, there are three answers in the end. 

It's so it's really it's interesting to see that. [00:22:00] I'm sure they're getting the benchmark, right? Like we're, first of all, we're a bunch of humans working in competitive environments of it. You get given a test, you learn how to pass the test and do that.

I think there was like a question, are they optimizing for the benchmarks? For sure. For sure. Wouldn't you? You'd be foolish not to. Are they gaming the benchmark? That's an interesting question. What's gaming? It's I don't think they're cheating. As in, I don't think in the model there is an if or a manually modified weights, that the people have changed just for the benchmark's work.

Maybe that's the case, but I doubt it. I think what they do is they give an excessive amount of focus. On those benchmarks. Yeah, I do think that exists in every industry, right? Like when you have, in security, there used to be an app. There is one called Web goat and there's a few other sort of examples of very vulnerable apps and security scanners oftentimes get assessed that how many of these vulnerabilities do they find?

How many false positives? And of course, the rules are tuned to do that. I do think that, people assess these models A lot based on like community sentiment as well and not just the benchmarks. [00:23:00] So some mathematical problems like, the reasoning models that are attempting now to solve the arc test.

Those may be the benchmarks are still good, the sort of the small percentages now of whether you pass some engineering tests or like the sweet bench or something like that. I think those today pale in comparison to what the sentiment is. It's the audience that catches not just, what is this model capable of, but what is this model actually delivering in the real world? 

**Simon Maple:** Yeah. Actually that leads us, I was going to move on to something else, but that comment actually leads really nicely to this one actually from where we didn't capture the name from LinkedIn. So sorry about that.

It's the second part of this that I like. It's great to be able to play and explore. But we still need to be solving actual useful problems though, safely and ethically. And I think this is where it leads on from that demo to actual real business use.

And I think this is not something that we can reliably do today without. Very close assistance or attended flows. But yeah, something I wanted to, I

**Guy Podjarny:** think it's absolutely correct. And it [00:24:00] comes back. I posted this model earlier in the year. In that talks about A I tools and talk about how much change is required to use them and how much do you need to trust them to work to provide value.

And most of the tools are in that bottom left. And I think it's because what this person here is pointing out, which is, for today to be useful, they can just be assistant. They can give you like small things that maybe complete your sentence, right? Complete your code line. And to the extent that they do that, Correctly more often than not then it's useful cause you can eyeball them.

But I think what we still lack is yeah, of course, the sort of the business analysis and such is, it's not there, although we can debate what Agentic and Devin are trying to do, but also the ability to trust that they will get it right. It's just really hard.

**Simon Maple:** And this is interesting because one of the things, one of the pain points of 2024 is obviously the marketing side of AI, right? And there's been a lot of hype that, I've heard everything from AI is taking our jobs. AI is going to replace junior developers. Devin can do all these amazing things just by asking you some short prompts and things [00:25:00] like this.

And actually what we have learned and, unfortunately AI is probably getting more slammed than most technologies because of the overhype. And it's natural for people to get excited and for people to want to go to market with these great promises. But I think we've come to recognize much more about the capabilities of not just how to use AI to get the most of it, but just to truly understand the capabilities of what we can really get out of it.

Yeah. 2024, what's your impression been over the hype, particularly with things like, maybe it's Devin or maybe it's, 

**Guy Podjarny:** yeah, I think Devin is probably the sort of the the flagship a little bit for that. I think, first of all, it's worth remembering that with LLMs, with AI as a whole, but LLMs specifically, you roll the dice, right?

And so it's actually quite easy to create a very compelling demo, definitely a video of a demo. 'cause you might have tried it 20 times and it failed 19 the 20, but in that 20th it did something quite amazing. Maybe it's not quite unreliable every other time. It works. That's still 50 percent failure.

And so it's far from production grade [00:26:00] product, right? But it's very easy to create a demo that is compelling with it. 

**Simon Maple:** It's pretty comparable to some of my production codes. 

**Guy Podjarny:** Yeah. There's a reason you're at DevRel. Yeah. But I think it's it is very much kind of human, right? You try things out.

But you don't just ship the first code you wrote. You try it and you build those out. And I think that's true in AI as a whole. It's true in the AI SDR space where there's a lot of conversation now about how it's really, it's not delivering on the promise. 

It's true in basically any place that is trying to scale AI. . I think there are a few exceptions. In code world yeah probably Devin is the poster child of it, just because they made so much noise, I think back in February or March. 

**Simon Maple:** Yeah. 

**Guy Podjarny:** When they published that video that really implied, fully autonomous development here. I just give it that task. It got torn apart, I think a few weeks later even online and even though they've been building for a while, leading up to that, the only GA to the product at the very end of 2024 and everybody you talked to about it, like everybody's quite enchanted with it or by it.

But when they use it, they say it [00:27:00] doesn't actually work. Now, of course, there are people that probably do find it to be working, but most people, everybody I talk to and from our sort of attempts, it's really quite hard to get it to work to the level that it's not satisfying, like it's not doesn't make sense to you.

There's a lot of promise in it, so I think that promise remains, but I think there's been a growing appreciation of that delta between demo and promise, and I guess the upsetting part about it is with that hinder trust even more than it should almost like that trough of disillusionment of, Hey, I thought it can do all these wondrous things and it didn't in 2024.

I'm not even going to try it in 2025. 

**Simon Maple:** Yeah. And I guess, if we look back at last year, when we actually talked about the blog post that you wrote in one of our episodes around the trust and change matrix, what we effectively see is there are those that have the lowest change to our workflows that require least trust for us to actually use or to give it to adopt it and to [00:28:00] use it.

Those seem to be the tools which people are getting on board with without necessarily providing with too much hype or too much expectation because they're sitting there using it whereas it's not hurting, but it's going to be something that it's a cloud over some of the more further out organizations and companies and tools and projects that are thinking about the next few years in the future as to what this could look like at some stage because it's not there today.

Yeah. Yeah. Have you seen the last year in terms of the, I guess the adoption of those different styles of products? 

**Guy Podjarny:** I definitely think that the coding assistants are getting a lot of adoption, right? You have substantial cohorts of developers that say, I will not go back to developing without those like big amounts enough that you, I think it's silly to think that they do not provide value.

They provide value or at least they provide the perception of value to a lot of users. Clearly Cursor is the one everybody's enamored with right now. We love it. We use it. But generally, coding assistants are now, [00:29:00] a core part of the evolution, maybe of IntelliSense and and just auto completion but substantially and people are used to using it.

I do think that there's a lot of criticism of them. Some of it is probably just luddite as Hey, this is a change. No, it can't ever be as smart as I am. But I think a lot of it is also legit about the fact that even those tools and that's true for all of the ones, the tools that are getting adoption today in indeed the AI world as a whole in specific dev tools are the ones you can eyeball and you can say, Hey, this is correct.

So that's great. However, those tools, by definition, these types of changes, they think small, they think local. And for instance, they duplicate code by, huge measures, right? If you want to create a CSV parser and you're coding it up, you might stop and say, you know what, it doesn't make sense for me to create a CSV parser inside this whatever, like web page that I have over here.

I'm gonna go off, I'm gonna create a library, I will create it over there but the LLMs don't tend to have that incentive, they tend to think locally they think about the local problem solution. So they will write to you a [00:30:00] CSV parser right then and there inside and they might even do that successfully But now you will have seven of those and then you start seeing some tweets and maybe this comes back to that spec centracity which is you start seeing people complaining about just the prevalence of code. If LLMs, if these coding systems help you write twice as much code, because, which is, let's assume that's a big productivity boost. Who's maintaining that code, right?

How are you assessing the quality of that code? When are they deleting code? All of those things are are real concerns. I don't think they are forever concerns, but I think the adoption and the success today is very local and for it to become broader. We're going to need to get better.

Yeah, I'd be very interested, as we get into 2025, I think I'd be very interested to see what tools like Devin, for instance are used for, because I think technologically they're fascinating and it's interesting to test the boundaries of how much decision we can power or sort of, like how many decisions we can delegate to the LLMs.

However, for it to be the thing that [00:31:00] builds a piece of software for you, you need to provide some confidence of delivery, some assurance that it would work. And the next time you're asking me to make a tweak, it wouldn't break. Like Simon Last was mentioning on his podcast.

And like many, like it's not hard to see that example. If you pull me up 500 bucks to try it out. So again big potential, but I think big questions about how to cross the chasm. My bets in 2025 we wouldn't fully solve that chasm on this agentic stuff. I think there will be more directions towards it, but some paths that we're taking, they overly prioritize that magical start and they don't follow through to say what's the destination?

They just basically solve a lot of immediate problems without systems. 

**Simon Maple:** You mentioned 2025 a little bit there and looking forward let's get you to say something. Let's get you to say something on record that we can hold you accountable for in the future Guy. 2025 predictions then there's been a little bit in and around day zero coding, which was talked a little bit. First of all, [00:32:00] let's talk about the prediction of beyond day zero.

**Guy Podjarny:** I think so I guess as I started on here, I feel and I expect that there will be growing awareness to this problem of, I don't want to call it AI slop because it's actually good code that gets generated.

Yeah. But just, how do you deal with these quantities? So you can think about today coding systems, PR coding systems, they generate code, generating code, and pull requests and code that got merged is a poor but existing, like a measure of productivity. So you can think of those as productivity increases.

They write more code faster. Code is debt. Code is is a liability that you then need to maintain. I think 

**Simon Maple:** Even the best code. Yeah, exactly. 

**Guy Podjarny:** Code rots. Code over time. Yeah. It it becomes a code that is great today if you leave it untouched is probably for most types of code. Most types of systems will become terrible in a matter of years or less and become a security liability, become maintenance liability.

And so really, the question is or my prediction for 2025 is [00:33:00] we now have enough code that has been generated like that, that I think we'll start to feel in 2025. The problems of it. There's a lot of code that basically nobody in your company wrote, even if you didn't let anybody go or nobody resigned just because they were generated with these LLMs.

There was a lot of duplicate code on it. So you now have a lot of breadth. You have a lot of small apps that were quickly created with these LLMs, but have no maintenance kind of plan to them. Maybe they're managed correctly, maybe not. And so I think we're going to encounter that in 2025.

I expect that would lead to on one hand on pressure on these tools to think about maintenance, which may not be delivered this capabilities in 2025 because I think people are still mostly looking for that initial magic. But some amount of that, which I think will go towards back centricity. And I think in enterprise, you will be seeking a lot of second order tooling that maybe are guardrails on top of what gets generated.

Maybe look a little bit more holistically at the path. But I think we're going to go from that sort of [00:34:00] initial infatuation with the magical AI to saying okay, let's hold off a moment here. Continue to adopt this. We want to adopt this. We see the benefit. But what do we need to be successful with this long run?

So you'll see policies coming to play. That's what I would expect on the 

**Simon Maple:** yeah. And the more I guess, as we were talking about, AI tools and AI projects being great at working in a demo. But how do we actually get it to work day to day for us? 

Because we're still very stuck in day zero because we're dealing with it too much as the what can it do? Let's have a play with it and so forth. Yeah, that's very interesting. 

**Guy Podjarny:** And today, the sort of the hold back in some enterprises is just fear of pace. It's Oh, wow. I'm freaking out here.

I've got like 100 developers on the team, 1000 developers on the team. They're using Cursor, they're using Copilot, they're using any of these tools. And, they didn't ask me, like they were just using it. They're clamoring, no, I can't do anything else. I have to adopt it. And so you can see the sort of the fear, the protection, the response of enterprises is more around, hey, this is happening faster than I've had time to analyze.

And I think [00:35:00] what you find in 2025 is, okay, now companies have had time to analyze. They may have responded and some sort of them would allow. These things they've seen the good and the bad. And so I hope that we will also see and expect at least, especially in the second half of the year better policy definitions, better best practices.

And then of course, the technologies to be able to support them. 

**Simon Maple:** Yeah. And actually thinking about it, just as we're talking now, I remember a year ago, we were having plenty of conversations about whether companies are adopting AI based on various security policies, does it break security policies?

I haven't heard anyone talk about whether or whether they shouldn't pull in an AI tool based on because of the security reason for a long time, is that just something we're becoming numb to now? Or is this just a, more in the flow of, people looking at this and then in the usual process, 

**Guy Podjarny:** I think those conversations are still happening.

And it's just maybe we're a little bit onto the next generation of conversation. Yeah. Next level. But I think the conversation initially is more again, this disbelieving the magic. So it's no, it produces crap code. [00:36:00] And on average it produces average code. And average code is crap code, so it does have vulnerabilities and can it be better? Yes. Will it be better? Yes. Will it be perfect? No. But at the moment it's producing that. And so I think a lot of the conversation was still around adoption. Do I even want my developers producing code in that fashion. I think what we're going to get to now when we're talking is more about, okay, so let's say I accepted that it will produce code for me and I even added some form of assessment that I think that code is good enough is fit for use for my environment.

And I said, it's allowed where it is allowed. Now we produce the whole pile of code. How do I deal with that code? And I think whenever you have abundance, you start having this sort of management, the discovery problem within that abundance. So if we find ourselves producing 10 times as much code as we did before with AI, generally you might say positive, right?

We're producing kind of more functionality, more productivity, but also creating a very real problem. 

Yeah. 

And I think that's the problem where Tools today are not really thinking because I think that's not where [00:37:00] the users are. They're still trying to stretch the capabilities. Don't get me wrong, I don't think 2025 I think is still very much a discovery of what can this even do.

But I think we'll start seeing some of those policies. 

**Simon Maple:** Let's move on to models now. In terms of what's going to happen in 2025 based on model choice, who's going to pick what models, which model is going to reign supreme. And I've got to be a little bit quiet here, Guy, because we know we work in the same building as OpenAI

so I've got, they're literally just the floor below. So I'll whisper this one Guy, so Guy, will there be a sole leader? Will it be Open AI? Yeah. 

**Guy Podjarny:** Yeah. I think I alluded to that before. It's I don't think there's going to be a single leader. It's interesting. Like why would there be now you positive, how would there be, it feels like each of these companies, it's 

**Simon Maple:** not what we would want either.

**Guy Podjarny:** To be able to be the far leader, you need either substantially smarter people in a domain than you have in other places. I don't think any of these companies can claim that like Google, Anthropic, Meta, OpenAI, they all have brilliant people working [00:38:00] on AI. And I don't think any one team, and there's a fair bit of mobility between them.

I don't think any one team is like heads and shoulders above the other, or you need some form of like core unique data. And once again, I don't think any of them have any of that. And so I'd be very curious to see poolside and the dedicated models come out and see if that produces anything.

But I basically think we will see a continuation throughout the year of one upping one another. I love Mike's Burton's question from before on the benchmarks, because what I'd love to see I'm debating if this is a wish or a prediction. I guess I think it's a prediction that by mid or late in the year, we will have more crowdsourced benchmarks that are based on real world scenarios versus just the sort of the technically crafted benchmarks that are designed based on understanding of the limitations of the technology as they stand today. 

I definitely hope that's the case, but I also would predict that would be the case. [00:39:00] If you're building on AI right now, I think your best bet is to not commit to one. But rather think about how to build those out.

People that do commit, they'll probably be based on business reasons. It's like with the clouds. 

Yeah 

**Guy Podjarny:** The clouds do have advantages over one another. Generally, they're all fit for purpose. If you have a good commercial or other business reasons to use them. 

**Simon Maple:** Yeah. And I think we'll see vendors like ourselves and many others who want to consume models.

It's a necessity to be able to say that I'm going to offer whichever models you would like to use and we'll try and build that into the and act with those as to the best of our ability, right? 

**Guy Podjarny:** Just to one slightly spicy one, which I'm more curious about. I wonder if I can, I don't think I can make a prediction about it.

Is the relationship between GitHub and OpenAI? I think that would be interesting because we basically saw GitHub add Anthropic to their sort of option Copilot, but I'm pretty sure the default is still GPT and and OpenAI's models. It's interesting. It's a very complicated relationship between Microsoft and GitHub and OpenAI.

So I'm [00:40:00] curious to see if GitHub migrates further away. If they launch their own models, they might have data or if they continued the alliance 

**Simon Maple:** And I know it's not something you wanna make a prediction on, but what would your prediction be? 

**Guy Podjarny:** Yeah. I don't know.

It's really it's really hard. There's just a lot of behind the scenes. I don't think these are tech driven decisions. I think these are business driven decisions. 

**Simon Maple:** Adoption. We'll briefly talk about adoption before we think about what's happening with the show in 2025. There's also a question from GM that I want to get to as well.

So we'll try and rush through these last few bits of adoption. Tooling adoption first. You mentioned, right now it's the high adopting kind of tools with low trust, low change that are getting massively adopted. Do you see that continuing straight through 2025?

Any reason for that to massively switch across to something bigger, something wilder? Or is that the iterative transition? Yeah, it's a good question. 

**Guy Podjarny:** I think I I guess my prediction would be one, yeah, we'll continue to see the growth of adoption of coding assistance. I think that's already been proven.

And it's just a trend. And the tools will get [00:41:00] better, but also people's conviction around when to use them and what not to will grow. So I expect that's the easy one. I do think that there will be specific use cases in which AI tools will prove themselves a little bit more.

I think the most best candidates to lead that chart are migrations from specific versions of like frameworks to others from whatever spring two to spring three. And from specific ecosystems to other ecosystems, because these are environments in which you can define what correct is.

And there are quite a few, both vendors in those spaces and customers, that are looking for that need for modernization purposes. And so I think there will be a set of those types of cases. I think the generic ones, hey, just complete this pull request for me. Complete the fulfill this ticket for me.

I think we'll see more adoption of them in 2025 than here, but I think that we'll remain at the edges because it's just too generic. It's hard to successfully pick them. The one use case that I'm really interested in seeing where it goes is the [00:42:00] enterprise modernization use case.

There's a lot of hope right now in enterprises that they can take their big, COBOL or old style Java. And modernize it to new style Java apps typically and definitely LLMs can help carry a bunch of the load over there. What I'm curious about that feels like a bit of a dissonance is the reason that these apps have not moved is because they're very fragile and LLMs are basically the most unpredictable piece of software you've used.

And so can you really use these list predictable technologies as powerful as they are to apply them to the most fragile parts of your system? I think the promise is there, my guess is 2025 would be an experimentation year on that front as opposed to success. 

**Simon Maple:** And actually, let me pull up a a question here as well from, I think this is Min actually.

So hello, Min. Talking about developers not needing to learn code anymore and people jumping straight into things like Cursor and really talking natural language versus programming languages. And actually the way Min here [00:43:00] describes getting that understanding is something that I mentioned Martin Thompson, I quote Martin Thompson quite a lot as well about the mechanical sympathy, understanding what's going on, beneath the covers, beneath the hood to actually get the most out of that kind of a tool.

Yeah what will, in terms of the developer role in 2025, how will that change? Do you feel? 

**Guy Podjarny:** Yeah, I don't think in 2025 it will change. I think in 2025 most of these solutions and even the ones that are spec centric we'll still rely on the ability to understand codes to build anything of note.

That said, do I want my kid to go and invest substantial amounts of their time learning how to code today? I would say no. No, I do think that a bunch of the other capabilities here in computer science, when you talk about software architecture, when you talk about system designs, I think those would be alive and well.

I think those would continue into the future. There will be a variety of them. And it's interesting to think about the apprentice to master journey in there, because today, the way you cut your teeth around architecture is by [00:44:00] doing coding for pieces and learning and seeing. So I think there's new paths to be carved.

But I think those skills will remain very relevant. I don't expect those changes to be at the 2025 range. I do think in 2025 people, like developers should invest their time at getting comfortable with LLMs and with AI tools, because I think learning when to use them, when not to, building intuition around it, understanding what they can do, that would be a very important capability throughout the rest of their career and in 2025 specifically.

**Simon Maple:** Yeah, I totally agree. We started this podcast talking a little bit about spec driven development. Let's roll up as our last prediction in terms of how spec driven development going to change in 2025, what are we expecting to see there? 

**Guy Podjarny:** Yeah I think we can talk to Tessl, right? We definitely want to get a product out there.

We're trying to figure out all sorts of questions around what the spec centric development look like. So I fully expect to see sort of product, at least in beta form come out. It definitely in 2025 as early as we can make it, I think what we see a lot [00:45:00] is we see inklings of specs.

We see elements of anchoring of some definition in some of the GitHub tools and some of Cursor mentions. And I guess I think a lot of those anchoring in truths, a lot of those, trustworthy pieces amidst your or like anchors amidst your generation. I think those will grow.

And yeah, we work in AI native dev community to help share those opinions. So if you're working on things of that nature, we'd love to hear from you and give you a stage and share some of those learnings. 

**Simon Maple:** Yeah. Amazing. We'll finish. I mentioned a GM and we only have a few minutes left, but I'd love to, he's probably been waiting or she'd been waiting very patiently for this.

Are agents going to be a commodity or how they differ from company to company? Interesting. Interesting question. 

**Guy Podjarny:** Agents is a topic in its own, right? They're probably one we should have a full episode on. I think the definition of agents is another thing that has changed over the year where people at the beginning of the year, I think anything that was a loop really that sort of said, hey, I'm going to ask the LLM if this was correct or not, and [00:46:00] if not, I'll come back was deemed an agent today. A lot of people say no, that's just the loop. That's not nothing too specific. And really it's only an agent. If you can really delegate substantial amounts and give big tools and capabilities.

So I do think there was some questions around agentic development. I think what we will find out in 2025 is the scope of the delegation that we can allow to do. And we talked about this a little bit also with Des around sort of autonomy. What is autonomous? Do you autonomously write the next sentence?

. We don't consider that autonomous 'cause it's a small task. And it feels like we only talk about autonomy when the task is big enough to be processed. And so I think agentic is a little bit like that when we talk about delegating to an agent. As long as these are relatively small and contained tasks, then we don't think of them as agentic.

I think there will still be loops. And I think what we would learn through the year It's just the pieces, the types of smaller capabilities that we can trust the agentic logic to get right. And I do think those will grow. The notion of delegation into agentic, take some open [00:47:00] ended thing, like pick the right way to lay out my shoe store which they can get an answer, but it's very hard to assess whether that answer is correct or not.

I think those will still be in discovery mode in 2025. The gap between demo initial promise and an actual working reliable product in this world is still very substantial. And I think in most domains, it's going to take more than this year to bridge that gap. There will be a few breakouts.

There will be a few kind of categories of products that come through and a few specific use cases. I think in dev that will, but for the most part, I think 2025 is still going to be a lot of a year of exploration for AI and AI dev. 

**Simon Maple:** So my takeaway then AI is not just a set of if statements and agents.

I'm just fully, so yeah, with that knowledge, I think we can wrap this up. 

**Guy Podjarny:** So yeah, looking forward to 2025, you want to say maybe a few words about just a quickly about what will happen on the podcast in 2025? 

**Simon Maple:** Oh, absolutely. Yeah. Yeah. So this existing format, [00:48:00] of course, with the monthly roundups and, us talking to people way smarter than me.

**Guy Podjarny:** I'll leave you not gonna bring some sort of smarter people than myself.

**Simon Maple:** Yeah, we'll continue to do that. And we'd love to actually ask the audience if there are people or topics that that you'd like to hear from and about yeah, talk to us podcast@tessl.io and tell us, who you'd want us to chat with, what you want us to talk about.

Agents being a great idea of that topic. Apologies for that. And going forward as well we're looking to actually add a couple new hosts who actually had some cameo roles last year. We squeezed them in a little bit. Yeah. We squeezed them in Ben Galbraith and Dion Almey who, also with us here at Tessl.

We have an amazing team. Actually, I'm really enjoying working with folks here. Yeah, they're gonna be giving us some amazing new sessions hot off the press. We're gonna be talking about what is new that week. And talking about what it means to the industry, what it means to how we use AI generally and AI dev tools.[00:49:00] 

So I'm looking forward to that. 

**Guy Podjarny:** Really looking forward to those. And Ben and Dion have done this before in the Ajax era. That old as am I. And and they've brought this in. But again, I think we're still in exploring mode and understanding a little bit of what's going on and a little bit of what of that is just snake oil.

And one of that is something that you should pay attention to is very valuable and Ben and Dion are amidst the best to do that. 

**Simon Maple:** Yeah. So Guy, it's been a pleasure running this episode and last year's podcast with you and looking forward to many more in 2025. 

**Guy Podjarny:** Indeed. It's been a blast, Simon, and looking forward to a great year ahead.

And we really do want to hear from you. So just emphasizing Simon's ask, we have podcast@tessl.io. This is a start for us. It's a path. It's meant to be a service for you to bring smart people that are learning here and bring their learnings and their opinions to it. So do tell us formats, comments, guests you want to see on it.

We truly listen and we try to apply to them. So we want to hear from you and you'll hear me and Simon blabber on as well about our opinions on it. So if there's questions you want, like [00:50:00] the great questions you had today we'd love to hear those as well. So 

**Simon Maple:** Amazing. So happy new year to all make sure you subscribe to hear for more sessions in 2025 and look forward to the next episode.

Tune in for more. Thanks very much. 

Thanks for tuning in. Join us next time on the AI Native Dev, brought to you by Tessl.