Datadog CEO Olivier Pomel on AI, Trust, and Observability

Olivier Pomel: [00:00:00] If I look back at the cloud migration the one thing that the companies that defined the cloud at the time, so mostly Amazon did extremely well, was that they handled the security aspect extremely well. That was a big fear of everyone. Hey, I'm going to share my data or my compute with just about everybody else,

wont they be able to see what I have? And they've been extremely strong about it from day one. I think it would be good for the same to happen with the, the everything that relates to, to AI models today. So we can take those main fears off the table and let everybody else build with confidence.

Simon Maple: You're listening to the AI native Dev brought to you by Tessl.

Guy Podjarny: Hello everyone. Welcome back to the AI native Dev. Today we have a very special guest that I'm really excited to have on. We have Olivier Pomel, who is the co-founder and CEO of Datadog. If you happen to have not heard of it, which would be [00:01:00] very odd at the audience listening to this to this podcast it's a pretty amazing company, a whole growth story.

We're not gonna, we have so many things to talk about. We're not gonna cover that too much. But it was founded in 2010. Now, today it's valued as, like the market moves, but I think well north of $35 billion, it has over $3 billion in revenue. Or you can make that in 2025, 29,000 customers.

All sorts of massive numbers. And I think importantly, it is the leader, the pretty obvious leader in the world of observability, really formed and shaped the category on it with with a long series of products that help explore new domains and new areas in a very efficient fashion.

So Olivier, thanks a lot for coming onto the show. Tell me if I, did I misrepresent or miss anything important in the, that quick intro?

Olivier Pomel: It's all good. That's definitely me. Definitely Datadog. And thank you for having me. Super happy to be here.

Guy Podjarny: Cool. Cool. Olivier, I think there's, a million things we'll talk about, but really we're going to probably sink into AI and observability.

I think a few people are more qualified than you to share both a perspective of [00:02:00] where we are and where we're headed. Maybe we just start off a little bit with just some taxonomy here, right? And some definition. So when you think about observability, like how do you define this world of observability and what it contains?

Olivier Pomel: Yeah. So observability, by the way, it's the worst category name to pronounce and I do that in earnings calls, many hundreds of times. So first of all, I'm not in love with the word because it's, it seems somewhat reluctive and passive whereas a lot of what happens is you know, very much not passive but the observability is the modern version of what used to be called a monitoring before.

And I think what it represents is, and obviously the idea of understanding everything you need to understand to to see, how an app is performing, what is actually happening inside of it, whether it's doing what it's supposed to be doing. It's an evolution of what used to be different categories.

In the, like 5, 10 years ago used to have infrastructure monitoring, used to have application performance monitoring, used to have log management, used to have network monitoring. These were all like little microcosms of of little [00:03:00] categories. And I think in part, thanks to the work we've done, those have converged into a bigger category, which is called observability, that's very much end to end, goes from what runs on the servers, when what goes over the network, but also what are the end users of these applications doing and, what is it doing for the business?

Guy Podjarny: Yeah. Yeah. Definitely have been a part of that journey with a variety of solutions during that time.

It was interesting to see it coalesce today that scope is expected, is, is table stakes when you think about building an observability solution. Yeah, and I like the point around it not being as passive as it implies. We're gonna dig into that a little bit when we talk about AI and agents.

So that's the definition of observability. And I guess at a high level, what do you see as the sort of the key opportunities for AI within this world? We're gonna start a bit meta and then we'll drill in.

Olivier Pomel: Yes. So it turns out there are big opportunity is that every single layer of the stack, with AI the first one I would say, I would call it maybe the most straightforward and boring one is that there's [00:04:00] just more applications being built with AI and those applications look a little bit different in terms of the stack they use, so more GPUs for example.

A lot more data, things like that. And, and we see that today already. We see companies that are building AI that are consuming a lot more infrastructure and building with a lot more data. So that's the, we call it the level zero of the opportunity.

Guy Podjarny: And these are applications built that are powered by AI?

There is AI in the functionality of the application.

Olivier Pomel: Yes. And even more these applications that are implementing AI largely. Say if you're going to consume a lot of GPUs today, you are probably a model builder or you're a company that is powering the agents that others are gonna be using.

The second level on top of that is now there's a lot more applications that are built on top of models. And this is a different way of building the application. The application is not as deterministic. The model does a lot more. And the way you manage, you build, you manage, and you understand what's happening is fairly different.

So I would say that's another new version of observability. That's the second level of that [00:05:00] opportunity. And then the third level is okay, great. Now we have all of these AI technology, all these new building blocks and all these new smarts. What can we automate? What can we build so that the engineers don't have to go and solve issues themselves, but the machine can do a lot of that for them, can detect a lot more and can solve a lot more for them.

And so we are, heavily investing in all three basically the, like serving the communities that are building the AI today, serving the communities that are building on AI and then ourselves implementing with AI to, to do a lot of automation.

Guy Podjarny: Got it. And this is the actual audience for these things is in reverse order, right?

There are probably a thousand companies that could benefit from AI powered observability, if you will for every application that is built using AI on top of the models and probably a thousand of those for every model builder of this. Is that how you think about it as well?

Olivier Pomel: Yes. Yes. There, at the end of the day we think just everybody will want automation, right? Every single company in the world will want automation. That's the dream, [00:06:00] right? The dream is you never have to wake up in the middle of the night ever again. You fix an issue, the issue's been fixed for you.

Yeah. And maybe you check it when you are, when you're up the following day.

Guy Podjarny: Got it.

Olivier Pomel: Then I would say most companies will build on top of AI models and they will consume these models and they will have to understand, how they work. And a smaller fraction of companies will build the models themselves.

And operate vast fleets of GPUs and AI infrastructure. And there's a inverted pyramid there in terms of the the number of companies that the, these apply to. Yeah, it just so happens that today the biggest needs are with the companies that are building the models because they're upstream of, of everybody else in terms of the consumption of AI.

Guy Podjarny: Yeah. And they're using, they also became big operational consumers very quickly. And every time also, like they need the uptime, they need the, the inference, stability, and then eventually the they need the training and large quantities as well.

Olivier Pomel: Yes, exactly.

Guy Podjarny: So I think still focusing a sec on the first, on the, that third category on [00:07:00] improving the products using AI. So improving, I guess the large swath of Datadog's products of it. You've launched a bunch of things already in this domain. So again, just starting from setting the stage here, what have been the sort of the biggest needle movers or biggest opportunities you tap into from leveraging AI to better serve the broader set of customers, even those building kind of a classic applications, if you will?

Olivier Pomel: Yeah, so we have a, just a few things we built like a few products with names on them that we, we've pushed out. One, historically we've had a product for anomaly detection that's called Watchdog. And the idea there is basically, it watches absolutely everything that's going on and it tells you, when there's been a, a spike in errors for a specific service or a, a step change in the, the request rate for another service,

things like that, which, which really help. Obviously you can't be watching millions or billions of time series all the time and that's the point of it. So that's what Watchdog does. A lot of it is I would call it, machine learning 1.0, in that it's not using financial models largely.

[00:08:00] It's using statistical models that run very cheaply and that can run all the data all the time. And that's been there for quite a few years.

Guy Podjarny: Yeah. If you ask by the way the ChatGPT it's gonna tell you it's good old fashioned AI, which I found entertaining.

Olivier Pomel: Thank you ChatGPT, by the way, ChatGPT also says that I'm a snowboarding champion, which, I don't know.

I've never been on the snowboarding.

Guy Podjarny: Okay, interesting. Yeah. Some hallucinations to play with over there.

Olivier Pomel: Yes. The, there's another product we've released more recently, which is called Bits AI. And that's the more agentic version of what we do, which is Bits runs investigations, it facilitates response to outages and things like that.

And you can interact with it and it can run on its own. And so this one is much newer, obviously it's built on the newer junior generation of LLMs and all of the other even newer models that have come up more recently. And this one is we are heavily developing as being the future of automation for our customers with applications, initially for triage, in oncall situations or outage [00:09:00] situations or fixing errors, that are detecting on the code, on the production, or helping optimize cost of performance or, helping identify root cause of incidents.

So these are the kind of use cases that are implemented with Bits and there's a couple more products. One is LLM observability, this one is built to help our operators of applications built on LLMs understand how they're behaving. And we can dig more into that. And the last one is we've announced also a, a financial model for time series called Toto named after the Wizard of Oz.

Guy Podjarny: The dog in the Wizard of Oz over there. Which is interesting character to to name it after.

Olivier Pomel: Exactly. Exactly. And there's a, there's clever acronym, and the idea behind Toto is build basically the best possible time series prediction model.

And we can also talk a bit about it. I think it's it's fairly interesting. So our very first model and obviously there's a lot more we're building on top of it. So these are, I would say, four of the main directions product or the products we've crystallized all directions into.

Guy Podjarny: Yeah. Cool. Okay. [00:10:00] So thanks for that. Set the scene on it. Maybe let's start poking holes a bit at the opportunities and the challenges involved in them. I guess the first thing I want to talk about a little bit is trusting the insights from these tools.

So you mentioned, Bits and in general there's a lot of maybe an intrinsic understanding that is easy to grok around the fact that observability did deals with vast amounts of data and AI is amazing at data. It can look at all these things and now it can read the lines and kinda make sense of them and give you whatever it is, the root cause analysis or even choose when to wake you up at night.

But as we've just discussed with some snowboarding examples it also doesn't always get it right. I guess how do you first maybe what's your sort of sense of the current accuracy level of detecting these things and giving you the correct answer? And what's the appetite for experimentation in this domain amidst users?

Are they accepting, shall we say, of, of mistakes? Do they do they expect perfection?

Olivier Pomel: Yeah, I think so I'll start with one of the biggest learnings we've had when we started Datadog [00:11:00] was that there, there's one lie customers will tell you, which is if your system has a smarts to detect issues I prefer you to give me the false positives and then I'll decide whether they're right or not.

And that's a lie. The reality of it is that you send two false positives to people in a row and then they turn you off forever. There's very little appetite to chase down the wrong path because of a machine that told you people will do it because of a human coworker, but they won't do it because of the machine.

And so the bar for precision needs to be very hard on the, on anything that has to, that relates to observability. And the good news is that, I would say three, four years ago being completely right about root cause, most of the time was science fiction. And I think now it's definitely within reach.

We see it in the way, in what we've been developing. We see it in the progression of the evaluations for that. And, as you've been building on AI too, evaluation is the hardest part.

Guy Podjarny: It's [00:12:00] they're it's a joy, a nightmare to be building with a predictability.

But you definitely see the progress.

Olivier Pomel: We see actually very good progress there and we can clearly see on the horizon the moment where the, this technology is good enough to be put into the hands of customers on, in, in a large amount, large number of situations. So that's where we are.

But yes, the bar needs to be super high. One thing we found interestingly is that the bar's a little bit lower in some relative areas such as security. I think users and customers are more willing to take a bet on automated decisions or automated notifications for security than they are for operations. And I think the reason for that is that the risk reward is a little bit different. I think it's it's okay to crash a workload if you avoid a security incident. Whereas it's less okay to crash a workload to avoid crashing a workload.

Guy Podjarny: Yeah. Yeah. Yeah. That's a little bit of running.

It's really interesting though, because security first of all, like the [00:13:00] security that my experience at Snyk is actually like very similar in terms of the false positive, false negative element, right? It starts off by saying find all the issues. If you're not sure, tell me about it. I want to know.

The reality is that's actually the primary thing they would feel they don't tell you. But it's interesting though, your comment about identifying flaws or like maybe having more tolerance around sort of alerts because to an extent identifying whether something is or isn't an attempted breach is also something that's less definitive. So maybe just people have lower expectations. They just don't expect anyone including human assessors to be able to spot what is, what is correct or is not correct. And so it could be the impact or it could just be the sort of the expectation of the art of the possible.

Olivier Pomel: Yeah, yeah. And also just to level set note what we discussed, I think when you compare what's happening in operations and security to what is happening in other areas of AI, like these are hard problems to solve. When you think about, we think about self-driving cars, for example, which are just about working now.

What these do are, what, pretty much any human over the age of 16 can [00:14:00] do without even thinking about it, when we think of the incidents, at least the ones we've had and the ones we've seen our customers have, that it takes large teams of very qualified people and PhDs, and a lot of time to try and understand what's going on.

And sometimes weeks after the incident, you're still thinking about them and still working through them and understanding exactly what happened. So these are really hard problems to solve.

Guy Podjarny: Yeah, no, I agree. And I guess on top of that, there's a notion of understanding the application itself, right?

Olivier Pomel: Yeah.

Guy Podjarny: So I guess, so maybe two questions coming outta that. One is trust is is imperative, right? People want, they, they have low tolerance for mistakes on it, but the systems are, as you've also mentioned, not predictable. And I guess one question is, can they find it? But even if they could find it, it doesn't mean they can find it again if given the exact same set of information.

So I guess how do you think about solving it? Is the solution, sort of the copilot world and leaning into that until it's good enough. Is it about narrow slices? Is there a different lens? Like how do you still make progress, while, you don't have [00:15:00] perfection, you have expectation of perfection and the systems don't, don't yet allow, they're getting close, but they're a line of sight for perfection, but I don't think it's present, right?

Olivier Pomel: Yes. I think there's two issues there. One is what's the right form factor for the customers to interact with it the right way? And that's where you think about, is it better to have a copilot or is it be better to have an agent that does things on its own? Do you want to interact with it vocally?

Do you want a button? Do you want, like, how do you, what does that look like? And that's a big question there. The UI really sets up the expectations from the end user. So there's a big thing there. And I think we're still, as an industry, we're still figuring out what works. What's pretty clear is that the chat interface was a great start. It was, it opens everyone's imagination, but it's not the be all end all. I think most of the AI based functionality is not going to work from a chat interface in the long run.

The second question is, how do you build or how do you enter the market with a solution that shows [00:16:00] confidence and or builds confidence with the end user? And I think for that, the, you can get higher precision by having lower recall. And you want to start in maybe with corner cases, maybe with some small sub selection of all the things you can, all the incidents you could possibly solve, but focus on the ones that have the, the really high precisions first.

Now, the challenge with the naive approaches is that the LLMs in particular, are not good at understanding when they know and what they don't know, they'll always gladly answer.

Guy Podjarny: Yeah.

Olivier Pomel: And, might be wrong a lot of the time. So I think a lot of the technology that we're building there is around understanding when we're right, when we're not when we'll have a super high chance of being right,

so we can actually volunteer guests and maybe solve the issue directly for customers as opposed to not saying anything. One advantage we have as a company that's already used for observability is that we can choose, pick and choose basically the the use cases and the the issues for which we're going to to automatically take [00:17:00] action.

And so that's what we're focused on, focusing on today.

Guy Podjarny: Got it. Makes sense. So you're already monitoring the whole thing and so you can make a lot of attempts, see where is it, what are the cases in which you can reach a high confidence

conclusion of it and then reach in, which definitely is a kind of an incumbent advantage of already being in place and already seeing the data.

Olivier Pomel: Exactly. And that's not a choice that's so obvious. You don't have that choice in the, in some other parts of the industry. For example, for self-driving cars, you can't say, only going to take to to make left turns. I think the, that's the luxury we have in the, in our industry.

Guy Podjarny: Right, in that sort of breadth. So that makes a lot of sense. So I guess maybe leading from this into the sort of the world of of agents of it. You're in the spot. You look at all the data, you see the data for the specific cases where you have conclusions. How do you think about actions, what is the, these are the insights maybe that you have higher confidence,

do you see some that you reach and like such high confidence that you can even act on them? How do you think about the agentic responses to this data, the non-passive version of [00:18:00] observability in its stance today?

Olivier Pomel: I think that's definitely the goal. And we did, we see areas where we can do it.

I think the the lowest hanging fruits there are things such as errors, for example. So you have, you ship a new application and you have some five hundreds on the servers, and the, that's pretty clearly an error, pretty clearly needs to be remediated. It's usually not that hard to generate the code that will fix the errors.

But then there's quite a bit more, more work involved into getting that code across and validated and, and running in production. So that's one area where you can take action, I would say pretty automatically and pretty quickly. And that's definitely something we're working on right now.

There's a few other areas. I think it's luckily a few years ago we started building the ability to take action for our customers. And so we built a workflow automation product. We built functionality so customers could could build application that automate the the way they run their own operations internally.

And that's a big departure from what we had been doing for the first 10 years of the company. The first 10 [00:19:00] years we were very careful about only receiving data, never taking action, never reaching back. And I think all of that is is really coming to, to fruition now that, we're trying to close that loop and really automate response for our users.

I'm a firm believer that for AI to show its its value, you need to automate action. I think if you have to keep pestering the humans to do things you are not gonna be successful.

Guy Podjarny: I'm a firm believer in what you said, like you have to work towards that sort of autonomous piece, but really the key question is that sort of buildup of the trust.

And I think much of the industry has taken the kind of copilot approach of it. And I think you, you can have another unique opportunity of it, which is to take the cherry pick approach of I'm just gonna look at all that data and, and I'll I'll pick what works. I guess what has been, what has been kinda the most surprising or the cases that you wouldn't, maybe you gave me an example of something pretty obvious, like the system is basically dead, it's just returning all these sort of five hundreds on it.

Have there been like positive surprises about things the LLMs start showing [00:20:00] you that are insights you didn't think they'd be capable of?

Olivier Pomel: In general. I think every, everything that's impressive, you've probably seen on Twitter already. Or X I should say.

That's the I would say that's the blessing and the curse of LLMs, is that it's so easy to make a mind blowing demo. And so hard to make that work reliably all the time for everyone. And you've seen examples of, you, you have a crazy error and the LLM immediately understands what's what happens and writes beautiful code to fix it.

The problem is it doesn't always work that way. And then understanding how to get that code to actually ship it actually is usually pretty difficult. I think we're still very much in the phase of making sure all the plumbing is there so that this works reliably all the time for everyone.

Guy Podjarny: Got it. So I guess let me take this a little bit like in that practicality of taking these lenses and maybe, maybe talk about all aspects of it and one that comes to mind is security. And so you're dealing with analyzing logs, analyzing systems, clearly a lot of untrusted data. And we've had several conversations here probably because of my kind of passion for the [00:21:00] topic of it about security and how those, for an opportunity for prompt injection that might even come from those, those sort of untrusted fields that might try to elude or fool the the agent be it or some other product to take action when it shouldn't.

I guess my kind of have two questions. One is how limiting is this right now? Is this a theoretical concern you're seeing in practice? And then how do you think about overcoming that? It's another aspect of trust, right? That is not about capability, but rather about adversarial activity.

Olivier Pomel: Yeah it's always been a concern, right? So it's already the case that, and we've seen that you can try and inject stuff through logs, through traces, through pretty much everything that submits data. It used to be cross site scripting and you inject with things into the logs.

And we've definitely seen that. So what that means is that everything needs to be heavily sandboxed. Everything needs to be like, you need to code defensively against any data that comes in. Obviously with the LLMs, like the surface of a, of attack is a little bit fuzzier. So the problem is I would say [00:22:00] harder to fully contain, as long as you end up passing some data to the LLM in the end.

That's definitely something we're aware of and something we're building against. The, we see that problem too in that, in what I just mentioned to to solve, to resolve errors and fix the code for that you do end up having to generate and execute code.

Anytime you execute code, you at great risk, right? So everything there also needs to be heavily sandboxed, heavily isolated so that you can do that and not feel the consequences of having all of that trusted code that's going to execute everywhere. But, in general it's a, it's good habit to imagine that anything that processes external data, might end up executing untrusted code.

Guy Podjarny: I think the so it's interesting. I had Mateo on the show on it who is in Lakera, AI security company and we talked about two things. One is how indeed the control plane and the data plane in security or sorry, in a world of LLMs, the control plane and data plane are merged and so you're basically executing everything, like all the data gets run [00:23:00] right, gets executed by the LLM, so it's hard to separate it. And two is that from an agentic perspective, if you think about maybe one definition of agent security versus LLM security is that LLM security tries to fool the human, right? I might try to get the LLM to give an alert that shouldn't, right?

Or reach a site that shouldn't, but then agents actually take action on those. And so you're taking it up to the next level, right? You're actually trying to guide the system. And so it's interesting that you say that that you run codes to generate it. I guess that's it's a more powerful but more dangerous action to it.

But in general do you think this is an eventual limit to the autonomy of the systems or is it or I guess do you anticipate that our sort of trust level in the agent's decisions would reach par with our trust level with the sort of the humans that that make the action, forget how you execute it well but you've chosen to restart a server. You've chosen to redeploy code. Those are decisions.

Olivier Pomel: Yeah. I think what this means is we probably just won't have completely [00:24:00] end-to-end models. So we can build the right limits around the models. So we can build enough trust there.

I think that's might be different from, with the desirable end studies, maybe in, in self-driving, self-driving, you hear that everybody now is building around end-to-end models. You get cameras in and accelerator put that out, and that's it. I think that's gonna be a little bit different for the kind of systems we build precisely because we want to have all sorts of checks and balances and validations along the way.

So you can't have adversarial systems or injections and things like that yield the the wrong actions there. And on, on our end, it's also an opportunity because we also build the observability tools both for correctness, but also safety and performance, and so these questions we have when we build those agent systems around who are we certain that they're, they're not being abused and they're doing the right thing and we can trust them. These applications all of our customers have and they want to answer for their own applications. And so we build the tooling for that as well.

Guy Podjarny: Yeah you're able to monitor your own actions, to an extent, you can have the same the right competency, in the, both the tech and the and the [00:25:00] team itself, which is pretty cool. I think that's really interesting and I wonder, I, I spent a fair bit of time thinking about the the scopes of autonomy.

And oftentimes, like the self-driving car is a good example of it. You you don't hold them accountable or you don't know who to hold accountable if they make a mistake. And so agent systems might have the same problem of the limit, might not be a technological one, but rather where is it that you're a whatever what is a reasonable liability for Datadog as a company to take for taking an action otherwise would've been a user's decision for it?

Does that come up like the sort of risk management almost like compliance risk, legal risk threshold on it versus technology as you think about Datadog? .

Olivier Pomel: It's not uncharted territory. You have systems, that automate updates, for example.

And as we've seen in several very public cases, like a, an update can, can have, really horrible adversarial impact on the, on customers. So that's not completely new. Of course, there's going to be maybe extensions of that. Maybe it's gonna be more refinements of the, [00:26:00] of the legal framework around it.

But I trust that, that's going to get figured out. But to your point, it, every company, not every company, but every other company has had a an issue where the intern, dropped their production database someday trying to to fix an issue. That might happen with early versions of these agents too, so that does create an interesting question of who ends up being responsible for that in the end. But again, I think it's a not very different from all the various ways in which automation is only being used.

Guy Podjarny: Yeah. And say how much you've seen, firsthand or been part of the migration to the cloud.

And I think it's true that in cloud you had to be like a little bit embracing of the failures to be able to be in the lead, right? Those who embrace the cloud at the beginning suffered from some gaps, from whatever container sandbox vulnerabilities to reliability issues of some of the clouds at the beginning to things that were like not irrational fears but actual kind of flaws.

And you had to maybe embrace the failures, right? And lean in and [00:27:00] accept the slightly finicky behavior of this system at the beginning to be cloud native and benefit from those advantages before the rest of it. Do you feel it's the same here? Is that all of these concerns that I'm raising and all of these fears are ones where hey, if you wanna be one of the leading companies of the future, then as a user, I'm not even talking about Datadog as a company, although that's also true, but as a company building right now and using tools, embrace it, brace yourself for having some false reports, for having some sort of system that deleted, cause if you don't you're just gonna be left behind. Does that resonate with you?

Olivier Pomel: Yes. I think the technology is definitely rough around the edges, on the, for anybody who's building applications on AI today. I think that's definitely the case and that's also why, in a way, there's a, I would say a relatively small number of AI native companies that are developing fast and shipping a lot. But the broader set of companies that are building on AI are still mostly testing the water. Like they're still having to have applications in validation in early tests with [00:28:00] their own customers, in proof of concept pilots like you name it.

It's, it's harder to get all of those into production just because the the failure modes are not well understood yet. The, this, the technology is changing fast. There's a bit of discomfort with all of the new exposure you might get with it, so we definitely see that.

I would say, if I look back at the cloud migration the one thing that the companies that defined the cloud at the time, so mostly Amazon did extremely well, was that they they handled the security aspect extremely well. That was a big fear of everyone. Hey, I'm going to share my data or my compute with just about everybody else.

Won't they be able to see what I have? And they've been extremely strong about it from day one. I think it would be good for the same to happen with the, the everything that relates to to AI models today. So we can take those main fears off the table and let everybody else build with confidence.

Guy Podjarny: Yeah, I think that's a really good observation and it's interesting because, while I a hundred percent agree with what you said, I [00:29:00] also think cloud security misconfigurations are probably today the biggest sort of source of breaches and and security flaws within systems.

Those are not direct flaws of the kind of AWS and cloud providers however they are created by the ease of creating infrastructure and and configuring it. So you do have to wonder whether prompt injection will basically replace, replace that world and be the next vulnerability.

No. It there's a risk of that happening.

Olivier Pomel: Yes. We replace the, the open S3 bucket by by, hey, there's a prompt in there. There's a chat in there, so I know exactly what I get. I can get out of it,

Guy Podjarny: Yes, exactly. With a statistical attempt that sort of says whatever send me all your data from from any sort of chat interface that you that you come along.

Hey, let's get technical a sec here and talk about how to get observability right. So you've made the decision to build a foundation model in Toto. The more I think about it, the more I love the name is for Datadog. It's the penny took a while to drop here. And like what prompted that [00:30:00] decision?

Why not build on top of the existing LLMs and because, do you think of it as a replacement, as an addition?

Olivier Pomel: It was got a very different focus. The goal is to work on time series and to be predictive for time series. For that we, and the broad models don't work very well with time series, they get you some things, numerically they're not great in general. And for time series in particular they're really not fantastic. We also needed something that could be very small so that it could be run in process on very large numbers of of time series.

So the first version of Toto is a, is purely numerical. So we fed it a number of time series. We have large numbers of time series of Datadog, as you can imagine.

Guy Podjarny: Yeah. And this would be like metrics from operating all sorts of aspects of systems.

Olivier Pomel: Exactly. So some of the data is traditional time series model training data, which is not from us.

Some of it is like weather data and things like that. And, and a lot of it or should I say most of it is our own observability related data on which we have, some pretty strong [00:31:00] signals in terms of which, which time series are interesting or not, because we see which ones are being used for alerting.

We see which ones are being looked at, at which frequency, things like that. Which gives us a good sense of of quality of the signal. And what was somewhat shocking to us was that at very first model we built, which was built I would say in a little bit of a naive way.

This is the first time we built a, one of those broad, no deep learning models. So the first version of it is already, or was at the time of release, is instead of the on, on time series prediction, including for non observability use cases, which shows of course that deep learning is working and scaling.

I, I think we've learned that over the past couple of years. But also the value of having great data, to train those models and how it really makes the whole difference. So this first version we haven't released it to the broader world. I think we really wanted to use it as a starting point, as a baseline for more research we wanted to do on the model.

So we're busy building new versions of it today that are [00:32:00] multimodal in that, it's, I would say it's a lower version of multimodal compared to what you can think of for the finished model. It doesn't have fusion, but it uses in addition to numerical data for time series.

It also looks at the, the text data that surrounds those time series, whether that's the descriptions you have for the time series, the names, the time series, the tags but also the event data that we're going to see such as what happens in logs or other time series, not just one time series at a time but multiple time series that can be correlated.

So we're pumping all of that into the model. And the point there is to make it really good at being predictive. So that, you, we can detect anomalies better. Like the anomaly detection is all about comparing prediction with, with reality. But also in some situations we can actually get ahead of incidents.

We can say, hey, this, this monitor you have is going to trigger in one hour. And we know that because your monitor is on the, I don't know, the it's on the CPU of your database and we see [00:33:00] that the queue of queries that are lining up on the database is going up.

We don't see it, we don't encode that as a rule, but the model has learned that it works that way. And so we're pretty excited about it again because we've seen really impressive results in the very first research version of it. And we are heavily at work to, to build the next version.

Guy Podjarny: Yeah, it sounds super exciting and I relate to the gap that you see in the, in the model, by the way, a bit of a trend in kinda the world to find specific niches in which you can build slightly smaller, slightly faster but also better models for specific slices but using the transformer technology.

So love seeing that. And yeah, clearly you have the data. I'm curious like how you think about it from a product perspective. Is this meant to be an engine that only manifests in Datadog's products? Is do you think of building an open weights model and having the world build on top of it?

How do you think about it from a, from an offering lens?

Olivier Pomel: So everything's on the table right now. So it definitely will be part of Datadog at some point. And that's why we're building it, right? So we can use it. We'd love to, to [00:34:00] have an an open weight version of it. I think we're being very careful there in terms of what data we have and how we use it and contractually what we can and can't do and what's right, not right to do with the data.

But we are, we would love to have an open weight version of it, so we're working on that.

Guy Podjarny: Yeah. And do you envision eventually creating for large enough customers models on their specific data? Have you experimented with whether that achieves better results or not?

Olivier Pomel: That's definitely also one thing we're experimenting with. Today we see that the broader models work better across customers. Because customers also have like a diversity of data they're using, they, but at the same time, they're using a lot of shared components, so customers are using MySQL and Postgres, and it's the same database across customers.

And their sales metrics have sales in them, in a name, so it's also somewhat semantically similar across customers. So we're experimenting with that. I think we'll see where we end up.

Guy Podjarny: Yeah. Yeah. That makes sense. It's I love the analogy sometimes, experienced individuals, right?

And so if you think about having, a very experienced SRE [00:35:00] come into a brand new system, they actually will have a lot of knowledge to come along. Part of it is familiarity with the Postgres that is deployed here that they've seen before.

Part of it's just analytical to say I know that when memory consumption and CPU consumption behave like this, it's recipe for trouble. Yeah it, I I like the idea of of bringing that knowledge from customers to customers. And I guess it, it's interesting whether there's a fine tuning layer on top of that, on, on a specific specific customer's data.

Olivier Pomel: Yeah. And one, one thing that we've learned over time too is it's critically important to have low friction for products to be adopted and to show value. And so it's very important for whatever you have to work well on day one and to work well with people who just get started and they, they send your data and they want to see things happen right away.

If you can get better over time, it's better. But you never get there if you don't show value very quickly.

Guy Podjarny: Day one. Yeah. Love the product sensibility. So I guess one more question on that domain. You know what, again, when you think about the person and their knowledge of it, infrastructure knowledge is useful but also familiarity with a specific app in question.

Have [00:36:00] you looked into and do you anticipate, including code and application knowledge of it, things that might not have traditionally been in, in Datadog's remit?

Olivier Pomel: Yeah, so application we get already because we do tracing, right? And we do we, we understand what calls what and what's being run and things like that.

Code, it's, there's been a big effort from us over the past few years to to get the code into Datadog as well. The main driver for customers to start tracking their code in Datadog has been using our profiling product. So we have a continuous profiler that runs all the time in production.

And, it really benefits from having the full source code, full, fully readable source code so you can tie, what's actually executing to, to the code you have. And and we basically we're pushing towards that. We've also been building extensions to the IDEs so that customers can navigate directly from wherever they are in the code to what's happening on production.

So that's being, it's been a big effort from us.

Guy Podjarny: And I guess the, if you cast your eyes further I'm getting a little bit ahead of myself 'cause I wanna [00:37:00] finish with some futures in here. But I oftentimes think about the AI kind of impact on software development or GenAI impact on software development is really a continuation of the same trend that we've been seeing with DevOps and continuous deployment and all of those in which it accelerates software development, it allows more independents, more software gets created and updated faster and faster.

This is another multiplier and I've heard you say similar things in past podcasts on it. In that case, like how do you think about the loop between observability and bringing, bringing that knowledge back into code and modifying the code back in? Do you see that accelerating and does that impact the kind of, the remit of a platform like Datadog?

Should we, would we see the expected scope of product? Or scope of platform change.

Olivier Pomel: Yeah. So I will say, so just to level set, I think there's a, there's been a continuous, as you said improvement in productivity for development and that dates back 40 years, right? If you compare where [00:38:00] we are today to where we were 40 years ago or 50 years ago, with punch cards and

assembly language and, and, and already a thousand times more productive, if not more in terms of what we can build, we've had the more advanced languages, computer screens, we've, we we have open source libraries, we have the cloud, we have all of those things.

So you can do, you can build incredibly sophisticated things with very little code and very little mental exercise. I think we're looking at another a hundred times or a thousand times increase in productivity with AI. And as we go through this productivity increase developers have less and less of an understanding of how the application is working and what it's doing.

And so we're switching more and more of the work there from hey, let me type the code to okay, no I need to understand is it working? Is it working well? Is it costing me more money than it should? Is it doing what it's supposed to be doing for the business?

Guy Podjarny: Yeah.

Olivier Pomel: Or did it [00:39:00] break? Did something happen when other things, other systems change or the customers change or something else happen.

So these are the big questions we answer and are, I would say, the questions that are becoming more and more valuable as the the productivity gains, stack on top of each other.

Guy Podjarny: Yeah. And so I agree with that line of thinking and one of the things that I wonder is if you are, if you're not writing the code and you have less visibility to it and you need to understand those systems on it.

Like today when you observe, you're still observing in large parts of the observability world, you're observing code or eventually when you find a problem, when you see a flaw, it's either in the infrastructure or in the code. Those are the places you change to evolve. How do you think about, if you're less familiar with the code, how do you observe a system?

Does that require some fundamental change to the core tenants of how observability is done?

Olivier Pomel: I think the observability will be end to end. I think you, what you observe in the end is you observe your business. Are my users doing the right thing and are they finding value and are they being served properly?

And that's what you care about, right? And you might have more specific [00:40:00] questions, which are, oh, I shipped that feature. Is it being used? And making a difference? And even more specific questions such as, oh, I changed this line of code, is, does it have the right impact? Because sometimes you will still have to do and make these specific changes.

I think the future of observability is that covering this end-to-end and helping answer those questions and plugging back into, whatever the, the developers are using to conceive and build those applications to start with. Today, it's largely a number of different code editors and IDEs and, and they write the code and they ship it.

You're building the future of that in many ways as others are too. And, we'll see that would the end state is with all of those different new ways of building and conceiving applications?

Guy Podjarny: Yeah, I I love the user focus aspect of kind of those, this or business focus really of also saying eventually, with all due respect to the code, what you're observing should be your customer's experience or your sort of outcomes, and [00:41:00] see if those are right and those you shouldn't care about the code.

It doesn't matter if you've written a spec or if you've written code and doing it. The problem identification should focus on the outcome. Now, from here we would talk about how do you identify why the problem occurred, so you should adapt to whatever it is whether it is a spec in Tessl land, or whether it is code or whether it is something entirely different in in some other solution path.

Very cool. Okay so we talked about some sort of trust. We talked about just how do you get observability right and maybe some of level up into the future of it. I'd like to take a slight detour and talk specifically about your sort of second category from the beginning, which is applications built on top of LLMs.

I'm gonna put aside for this conversation those building the actual LLM, I think that's specialized knowledge. Not going to drill into that too much. I guess first of all clearly the world of observability, as you pointed out, has expanded to include logs to include APM alongside infrastructure monitoring.

When you think about LLM observability and observing those types of applications do you identify kind of new categories that you think will [00:42:00] get added to an observability platform as a default?

Olivier Pomel: Yes. Look, we call it just LLM observability for now, it might have a completely different name and shape two years from now.

For one thing, I don't even know if we'll talk about models as LLMs. Maybe we'll have a, maybe the shape of, of what's provided by the model vendors is gonna be a bit different. Maybe the naming is gonna be a bit different. We'll see. Maybe the focus on language will be reduced. I don't know.

Guy Podjarny: Yeah. But so AI, you're actually embracing AI as a term because it is a bit more generic just GenAI

Olivier Pomel: Yes. GenAI, it's it, problem is the more you the more often you pronounce the words AI or the letters AI, the more bullshit it sounds. Have to be careful.

Guy Podjarny: Any buzz word in the world, right? Organic, anything that would happen eventually would lose meaning.

Olivier Pomel: Yes. But we do think there's a growing part of the applications that's going to be non-deterministic. So you use models and those models are trained and maybe they're continuously trained, maybe not.

And you, what you have to measure is the outcomes. Like what, what actually happens? What do they produce? So you can't be [00:43:00] in a world where everything happens, when you define the application, you write the spec and that's it. And now, you know from the moment you've done that, you know the application is doing what it's supposed to be doing, you actually need to to understand what happens on the other side of it.

Now you've shipped the application is being used is what is happening there similar to what you thought was going to be happening. Are there new things happening? Is it happening in a safe way? Is it being abused? Is it helping to reach the right business outcomes? When folks use this non-deterministic part of the application do they buy more, do they stay longer?

Do they do whatever it is you wanted to do? In the first place. All of that are super hard questions to answer. And you need a new form of tooling for doing that. What's interesting there is that the companies that are building around that, they're typically not the companies that are building models like these other companies that are building on top of models.

We see their needs evolve as the the world of models evolve as well. So I think it's probably going to be a couple of years, at least before we know what this particular part of the stack ends up looking like.

Guy Podjarny: Yeah, [00:44:00] super interesting. So if I echo this back, you're saying, you say LLM observability and people might think, hey, you're capturing a chat trace and things like that.

And those might be correct in the meantime, but really the real problem with that LLM observability is that you really need business monitoring or you need outcome monitoring to be able to define it because the product is gonna be unpredictable. And so yeah you need the technicalities, you need the traces as well to be able to troubleshoot problems.

But if you want to know if there's an outage, you wanna know if there's a problem, you're not gonna get that from the traces. You're gonna get that from the outcome measurement. Is that, did I get that right?

Olivier Pomel: That's right. I think, the trace, is very low level.

I think it's rare that you're going to go and read step by step what happened, either in a chat or, in an agent or, and see what actually went on there. And I think it's really a, it's really debugging in validation of, some hypothesis you have about what happened.

Whereas what matters at the, in the end is, did you answer all the requests? Did you generate transactions? Did you generate some something else? And what you will see is yes, maybe you did, [00:45:00] there's a, there used to be 1% of those conversations that were not satisfactory to the end user, and now there's 5%.

Do we know why? And then you build down and you figure out, okay, actually it turns out those 5% are all in the same region of the embedding space. And that's because maybe there's a data source that changed and now we have bad data for that. Or maybe the models behind or maybe something happened there that changed and that causes an issue.

And you can validate that by looking at the specific conversations and all the specific, interactions between the agents and the various agents. And you understand, what went wrong there. So this is a whole new world of understanding what's, what's happening and debugging.

And again, what I was saying earlier is the main challenge, at least for us is that the space is changing very fast and the use cases are changing fast. When we first launched that product a year ago what we were seeing were mostly chat bots. That's what people were implementing.

Guy Podjarny: Yeah.

Olivier Pomel: Now we see a lot more agents than chat bots. So instead of end users asking questions from the models, you have agents asking questions [00:46:00] themselves to the model and looping. And so we'll see where we end up, I would say a couple of years from now.

Guy Podjarny: Yeah. I I love the love the thinking here, and I can easily see that indeed being the future of the observability.

And I guess that sort of brings you much more, blends the lines with product analytics companies. And it comes back to do you basically expect those spaces to merge, like the infrastructure application and then also business and product analytics?

Olivier Pomel: Yes. And we have a product that's a fairly new in product analytics.

We built that on top of a real user monitoring product, which is basically track the interactions users have with an application. We started from the performance aspect. We started user monitoring from the question of, do you have errors and what's the rendering speed and is it fast enough is degrading, things like that.

And now we're moving more into the understanding of clickstream. What are people doing? What are they using? What are they not using? Do you get the outcomes you want from that or not the outcomes you want? And and obviously, as I mentioned earlier, I think this is critically [00:47:00] important for anything that is AI related.

Because, for a typical application, you can have a, after you develop it and after you ship it or want to ship it to production, you have a pretty good idea of whether or not it does it's supposed to do. For an AI application, you don't, and you need to measure that in production.

Guy Podjarny: Yeah. And I, I think that's probably like the core of the problem, which is you, I guess observability or identifying a problem requires first understanding what isn't a problem. And, and that is just very hard in, in the world of GenAI or other LLMs or AI, how you call it, because it's just very loosely defined.

The systems are unpredictable to begin with. The types of tasks and missions we give them is very broad. And so how do you define it in the first place? Do you anticipate anomaly detection to even be possible or can it only be anomaly in that sort of slightly more narrow slice of business outcomes that you can identify the anomalies, but not in the activity?

Olivier Pomel: There's a number of anomalies you [00:48:00] can detect, so you can run all sorts of checks on the statistical distribution of what comes out of the models. And if you can understand when there are changes, when you use drift you can, there's a number of metrics you can track.

There's even ways, of there's fairly reliable ways of measuring or estimating whether or not you're getting hallucinations without even having access to the ground truth, just by looking at the distribution of the data. So there's good things you can do there that are good heuristics, I would say without, without having the users define, hey, this is my business objective and this is what I want to monitor.

Guy Podjarny: Yeah. Yeah. Very cool. Yeah and very much I guess at the end of the day extends the scope of what you are able to do with, with the data that you expand. So maybe one, maybe bring it back to security once again. Apologies for that. Another lens of attack is not just prompt injection but also as the data that a user interacts with and such blends with the execution and the functionality you also get into topics like abuse or [00:49:00] fraud or such with the interaction is one that is, that is very textual. Do you have thoughts of those? Is that a red line? You're not going to look at the data or is that also within the scope? Just trying to get a sense of where you see the lines forming, if anywhere, really.

Olivier Pomel: Yeah. It's hard to know exactly where the lines are forming yet. In great part because we don't know yet what's, what's the final feature set of the models themselves. So we will, the will a lot of the commercial models. And then by extension, all the free models after that, would they include some level of built in protections, built in filtering, so we don't know that, today they don't, but the space is young enough that it might change over time. One thing though that is pretty clear to us is that understanding the applications will involve looking at the data that comes in and out, that goes in and out of the models. I don't think there's any way of building observation in the future without that.

And for us, it changes the profile of the data a little bit. When we were [00:50:00] looking at traces and logs and things like that, this was mostly not primary data. Like we wouldn't see the doctor's prescriptions or the social security numbers. Like those typically don't end up in logs or traces or places like that.

When you look at the data that comes in and out of the models though, you'll get all that primary data. And so that, that has different implications on what you can do with this data, how you can store it, what access you can give to it, so there's a whole new set of I would say functionality and protections to build around that.

Guy Podjarny: Great. Yeah, and I guess the good news is from a business perspective, you need to make that investment to get the opportunity to open it up but then once you have that data, it also opens a set of opportunities for providing value to your customers.

Olivier Pomel: Yes. Yes. And again, you get as close as possible.

It's always been the dream. Like the dream has always been look you observe the application and the world is transforming digitally. So the application is running the business. Therefore, by observing the application, you observe the business. I think this is becoming even more true in the world of AI just because observing the application now [00:51:00] means actually looking at all of the primary data that is making the business.

And that's, that's why it's so exciting to us.

Guy Podjarny: Yeah, yeah. Very interesting. Let me take it to the world of UX and so cast your eyes, we've been moving further and further into the future here. As we're having the conversation, five plus years from now, you have good agentic system built up. You figured out some thresholds from a security perspective that you would engage. You've already mentioned chat is, like a limited type path. What do you envision as the interface or how does the interaction with the observability system change during that time? Do you, and I guess notably, do you envision it changing to be more like how you interact with the operator today that uses Datadog and Datadog becomes that interface? Or do you think that's just a a fantasy?

Olivier Pomel: I think, look we already see the interfaces that, that represent the systems as humans as being appealing to users. So some of it might be chat, though chat in a [00:52:00] way that's not in the you asking questions, but the machine working with you, cause it's hard to know which questions to ask otherwise. The, we see that also in the, in voice. We think there's many situations where, especially when there's a group of people trying to work in real time on a problem where voice will be a good way of interacting. And so we, we think in the end the software products and our product as one of them are going to look more like, like humans you, you deal with. And that might be the way we interact. I think there will still be UIs on top of it 'cause UIs are a great way to consume information and to understand what it is you can do. You when you start your car it's good to have a to have buttons and to know what's going on there and, not start a conversation with it.

Same thing with your oven. I think it helps and we still will have that but I think it will be complemented with more open-ended human-like interfaces.

Guy Podjarny: Yeah. Very interesting. So look we spoke about all sorts of very exciting bull case, if you will, opportunities that I think all are very valid and, and [00:53:00] real around expanding.

It also just interesting to think about how observability might change. What would you say is the bear case for kind of how observability is done today with AI? Is there a scary scenario from a, sitting in the, sort of the lens of the incumbent in Datadog when it comes to AI?

Olivier Pomel: I think the things that are scary to me are hype cycles and things being oversold. So I'm not, on earnings calls. I'm not overselling the the age of agent and things like that because I think, as of today, the technology is still not quite there yet.

It's exciting, we see it on the horizon, we see all the potentiality it's not something that is happening just today at scale, on the field. And so my worry is that the technology gets oversold and that trust relationship with the customers, the users is broken.

And it's happened many times before in monitoring and observability. We, as a company, we've been very reluctant to have AI on our website for the first 10 years of the company, just because I associated it so much with [00:54:00] bullshit and things that don't work and are oversold that, I wanted to avoid it.

There's a whole category of product called AIOps that existed 10 years ago. And there's usually nothing AI about it. It's, the AI part of it is Regex, so that's, that's what was built.

Guy Podjarny: Yeah.

Olivier Pomel: So I want to be careful about underselling and overdelivering, I think is the, the way we like to do things.

Guy Podjarny: Yeah. Very cool. Yeah. Fully relate to that. I remember explicitly taking AI off the sort of the Snyk AI powered static analysis product slides of it because it decreased trust amidst customers, even though it was actually doing good old fashioned AI on it and then needing to actually battle to get it back onto the slides as it got the mojo once more.

Olivier Pomel: And, when you talk to the folks that have been researching and building the technology for AI for the past 30 years. Like they've been through cycles of, of winters where, things were super hot and then, got very cold and nothing happened for 10 years. There was a bit of a fear that, after pre-training was picking up, like we would have a little bit of a [00:55:00] slowdown.

Turns out RL started picking up the slack pretty quickly. And there's another vector of scaling now that, we're all excited about. We still have to connect those two ends. We still have to get to the point where most agents are good enough for most use cases.

And that's not here yet.

Guy Podjarny: Yeah. And reliably, which is probably like a key question. Cool. There's, I've got like a whole hoard of other questions ask of you, but I think we're running outta time. Let me ask you two quick questions. One is, as you build the AI powered products from a engineer perspective, the people that you hire into your development teams or R&D teams building it, is there a difference in profile?

What have you seen to be like the people that are most successful right? Most adept at building those products?

Olivier Pomel: Yeah, there's not a big change in who we hire. I think, in general, we've always indexed most on the the ability for people to grow.

And I've seen that through my career before Datadog and at Datadog, where, when you hire people, usually they're super smart but only a small number of people are super smart and willing to grow. [00:56:00] And the difference here is, yes, I'm super smart, but if on my second day somebody I do something and somebody tells me actually you should try it a different way, how about this? The people we want are the ones who are going to reply, you know what, let me try. As opposed to no, I think I got it right. And very often people who are super smart are their own worst enemies in that way in that they know they're smart and they, they don't evolve.

They don't grow. So I think that's, that part is super similar. There's one new category of people we're hiring though, which is that all last year we started building an AI research team. We didn't have one before, like everything was applied, so we were building and everything into our product, but we didn't have a proper research team.

And the change there has been that if you compare to the way research was done 20 years ago, actually I started my career in research. I brought me to the US with a job at IBM research in upstate New York.

Guy Podjarny: It was cool at the time IBM research, maybe it is still today, but at the time I remember it being a very high appeal job.

Olivier Pomel: Yes, it was it was before Google research and [00:57:00] before all of that stuff, but the so at the time, the all products were built from research that was more than 10 years old. And it was very difficult for anything to cross directly from research into products. I remember that's why I left.

And today, if you compare to that time you find into product research that is less than six month old and that research for a large part is not ever published which means there is tremendous value infusing research and product development. And that's what we're doing today.

Guy Podjarny: Right. Yeah. Makes sense. And those, and that is a brand new profile on it. And you've had to staff up, where in the organization does it sit?

Olivier Pomel: It's under my co-founder and CTO Alexis.

Guy Podjarny: So it breaks the regular sort of product division?

Olivier Pomel: Yes. But look, it's still research with the, there's one thing that's different about research is that you don't, have a scrum every morning when you tell the researchers what to do, research has to have a bit more independence.

Guy Podjarny: Great. And then, I guess a last [00:58:00] question, it's just what surprised you the most as you've been building with on AI, with AI over the last couple years? Both for good, for bad? What surprised you the most?

Olivier Pomel: The rate of change is just what's the most surprising. You are in the same situation, like you're not new to this business.

You've built companies before, and I think we've never seen such a rate of innovation or rate of change. My, my favorite joke on a, on AI and I get asked all the time how much time it's saving me, and my answer is always that I haven't reached break, break even yet. I'm still spending more time learning about it.

And I expect that to last. I think it's changing so fast that it takes a lot of time to learn about it and to and to level up.

Guy Podjarny: Yeah. Yeah. I fully relate. And I think it also speaks back to the competency that you want in the team, which is a team that is able to deal with a rate of change that was always good, but now it's probably on steroids.

Olivier, this was excellent. A lot of just brilliant paths and very kind of promising paths of expansion also for Datadog. So that's really great. Thanks a lot for coming onto the show.

Olivier Pomel: Thank you. This was fun.

Guy Podjarny: And thanks [00:59:00] everybody for tuning in and I hope you join us for the next one.