Driving AI Powered Innovation on Azure

Patrick Moorhead: The Six Five is On the Road here in Chicago at Microsoft Ignite 2024. It has been an incredible event so far surprisingly, well, not surprisingly. We’re talking a lot about AI and whether it’s AI to benefit end users, executives, society as a whole, but this show is really focused on businesses trying to get the most value out of AI. And Dan, we can’t forget about developers because it’s about developers, developers, developers, I heard?

Daniel Newman: Yeah. Well, there’s really a pretty significant continuum. We’re a couple years into this kind of flurry, and again, you and I talk about this a lot. It’s not a couple of years old AI. I mean, we’re talking about decades of algorithms and machine learning, but we are talking about is this big inflection that’s taken place over the last few years, and enterprises are investing big. We’re seeing triple digit percent investment growth in many companies that are trying to figure out how to turn this into productivity. And so, we had an incredibly thorough comprehensive keynote today. I mean, the announcements were… There were so many of them.

Patrick Moorhead: That’s right.

Daniel Newman: I don’t know which I would even start if I had to pick my favorite path. But for me, it was all about how to get more value and how to turn that into returns for the enterprise and for those users and consumers.

Patrick Moorhead: Yeah, and whether it’s developers, AI engineers, I mean, everybody who has a technology badge at a company, they all had something that I think that they could get excited about. And to talk about this, we have Scott Guthrie. He needs really no introduction here. Scott, welcome to The Six Five.

Scott Guthrie: Thanks so much for having me.

Patrick Moorhead: Yeah, thank you.

Scott Guthrie: Good to be here.

Daniel Newman: Yeah, it is great to be here with you and first time on the show. You heard me in the preamble, Scott. I talked about it’s been a couple of years. Of course, the ChatGPT moment was this inflection. Then, there was kind of the everybody getting excited, lots of downloads, and then about six months later, and Microsoft was one of the first out of the gate with some really exciting advancements. Now, we’re 18 months beyond that. Talk about what are you most excited about, what are you most enthused about. What do you see at this maturity point in the market?

Scott Guthrie: Yeah, I think as you’ve mentioned, two years ago, it was kind of the ChatGPT moment where people said, “Whoa, the world’s changed.” I think this time last year, you had a lot of conversations about people that are saying, “We’re starting to do projects. We hope to be able to do this.” I think the part that’s changed and that I’m excited about now is you now have companies in all industries saying, “Here’s what we’re live in production with.” And I think that that productization of these large models for whether it’s the financial industry, and we had BlackRock on stage showing off and talking about how they’re integrating AI into Aladdin using Azure AI, whether it’s BMW who’s integrating it with IoT sensors in their cars and doing AI, whether it’s McKesson that we showed a good video of, they’re doing cancer drug discovery, you’re really starting to see it being used and being used both from traditional tech companies, but also leading edge companies in every industry.

Patrick Moorhead: It’s incredible. I like to look at generative AI. Well, if I look back, dial back, there’s been a lot of tops down edicts, sometimes moving off the mainframe to client server, going to the cloud, the board and CEOs. But ultimately, what happens is it all, it trickles down to the developer, and we’re seeing the same thing with generative AI. We had just this incredible pressure, “Hey, what are we doing in AI? How do we take advantage of this?” But ultimately, it ends up on the desks of the developer, so there’s a tremendous amount of pressure here. What does the developer experience look like in the future?

Scott Guthrie: I think developers are really the bleeding edge of how generative AI is going to be used to kind of transform every job ultimately. But I think if you look at GitHub Copilot as an example, that was the first application that we built and brought at Microsoft with these large transformational models. And it started off as, “How do we improve code completion and IntelliSense in the editor?” And then, we kind of added, “Oh, how could you add a chat window next to it so that you could ask more broader questions?” And now, it’s being used to generate unit tests. We showed off a demo in my keynote of how do you use it for advanced security to actually analyze for security vulnerabilities and then propose fixes for those vulnerabilities. You’re going to see it, and we’ve already got it in the pull request process, so code reviews and writing up the pull request notes.

And I think you’re going to sort of see AI be used across the software development life cycle, and then the other thing that we started showing today… And it’s fairly transformative to developers. I think we’ve got more than 3 million developers using it today. And on average, people are seeing about 55% productivity wins with coding. And then, the stat I like the most is just people enjoy their jobs more, and we see 70 to 80% of people say, “I feel more fulfilled because I’m working on tasks that actually I enjoy doing versus I’m not having to spend as much time on some of the work that is part of the job. It’s not the fun of it.” And I think you’re going to see this be driven forward. And then beyond just the core coding experience, I think the other thing that you’re going to see is, and we demoed some of this with our GitHub Copilot for Azure, is the ability to take platforms and infuse it into the chat so that you can actually go beyond just coding.

You can see how is the app doing in production. What is the cost of this app? How do I reduce the costs? And then, we also showed off with our Azure AI Foundry and GitHub integration, how can I use the models in GitHub? How can I do version control of the models in GitHub? How can I test evaluations? And you’re going to see really, I think, GitHub, in particular, be the home for developers, but also the home for AI development. And I think we’re just getting started in terms of the full potential of using generative AI to help developers do AI, and it’s going to be exciting over the next couple months.

Patrick Moorhead: Yeah. Just a personal aside, my son’s a data science major, and I always bounce these things off him, but he and his entire class are using VS Code GitHub Copilot.

Scott Guthrie: Great.

Patrick Moorhead: And I sent him a screenshot of a picture I took of you on stage on Data Foundry and also the links to the Azure. And he came back, and he’s like, “I’m stoked. Let’s go.” But the amount, the speed at which these tools are taking off, I think it’s pretty safe to say that AI landed first with the developers that were making AI. It’s just fascinating, and it’s been a long time since I’ve seen anything like this take off. It’s super exciting.

Scott Guthrie: The pace that we’re in right now is, to your point, I mean, I think back to… I think there’s really only been two or three transitions I’ve seen that have moved at this pace. I think one was the emergence of the smartphone. I think one was the emergence of the web, and then probably the one before that was probably the emergence with Windows where you suddenly had apps that within a year came out that just people hadn’t envisioned or imagined before. And I think this is both a technology moment, but it’s also a moment where end users are going to be profoundly changed in terms of how they interact with technology as well.

Daniel Newman: Yeah, the interesting thing is the data would probably prove that those earlier inflections actually didn’t happen as fast as this one because we kind of see on all those charts, but I think to your point, it’s sort of the feel, that you start to feel something meaningfully change. And by the way, I think it’s interesting to capture, for the record, it sounds like you’re very bullish on the role of the developer because there is a bit of a continuum of experts that kind of have this perspective of everything from developers will be really important to some people that are like, “Oh, in a few years, we won’t need them at all.” And I think we’ve all agreed that that’s not where it’s going, but I think somewhere in that continuum of how it’s going to evolve, but it sounds like Microsoft, sounds like you are very committed to continuing to invest heavily, and you see a ton of value-

Scott Guthrie: Absolutely.

Daniel Newman: … in that community, which by the way ties into nicely the announcement of AI Foundry. Talk a little bit about why you felt that was so important to move forward and why AI Foundry is such a big announcement for Microsoft.

Scott Guthrie: Well, I think one of the things that we’ve seen as organizations have kind of tried to move from proof of concept to production is that previously, certainly, a year ago, there’s so much AI technology coming out. There’s so many different models. There’s so many different tool chains. There’s so many different frameworks, and organizations and developers were really struggling to figure out, “Okay, how do I make all this work together and how do I make the right set of technology choices and do it not just to release once,” but everything this day and age is a continuous release process. And so, if you pick a model or maybe you pick two or three models and you get in production and then a new version of those models come out, how do you know that you’re not going to both improve quality in some areas and regress in others? How do you know the latency is not going to have a problem? How do you protect against jailbreaks? How do you protect against potential safety issues? And if you require everyone to kind of take very small Lego blocks and put them together, what we found is a lot of organizations were struggling with that, and we’re finding that they couldn’t quite get into production.

And so, our goal with AI Foundry, the Azure AI Foundry services, how do we take more of those end-to-end workflows and put them together still be very open so you can always use the technology or the model that you want, but how can we make it easy to go from idea to production? And then, once you’re in production, how do you run it so that you can understand cost management, you can understand security, you can do model upgrades in a seamless way? And Foundry kind of automates all of that in the end. And so, for example, when a new model comes out or if you want to test a different model provider, you can actually even do kind of split testing where you can say, “I want 4% of my users using the new version, 96% the other, and then let’s do evals and see what’s the difference in terms of accuracy, in terms of groundedness. Am I seeing different business values, slide it up or slide it down?” And that ability to automate and have kind of a CI/CD process, if you will, it really enables organizations to bet bigger on AI, go faster, and then do it in a secure and reliable way.

Patrick Moorhead: Yeah. So Scott, as we’ve seen with modern applications, we fractalize, whether it’s microservices, whether it’s multiple APIs. At some point, and I think we can all agree that AI applications have even more complexity. So once customers get AI apps into production, what structure do you have there to help IT professionals and operators to manage all of this?

Scott Guthrie: Well, then that’s a big part of what AI Foundry provides is that there’s both. I’d call it the developer phase, but importantly, there’s also the operations phase and the security phase. And so, simple examples would be how do you do real-time monitoring of your apps? Foundry gives you a way that you can actually see, “Okay. How many AI inferencing per second are you doing? Are they reliable? Are you seeing errors show up?” You can even automatically with Foundry set up security alerts so that instead of not only the IT team that’s managing the AI infrastructure, but maybe for example, your CISO office or your DevSecOps team.

How do they get an alert if they detect there’s a user that seems to be trying to jailbreak us? How can you take that signal from the AI system but also look at what other web pages on the site are they using? Are they accessing other APIs? What else are we seeing from this IP address? And that ability to kind of integrate across developers and across IT and potentially the CISO office, pretty much every large-scale AI application needs to be able to do that and be able to do it 24/7. And that’s part of what we’re trying to do with Foundry is kind of we’ve turned it in some way, the app server for the age of AI.

Patrick Moorhead: Right. Yeah. Satya reinforced that nomenclature in our meeting with him. So this is the observability layer at the bottom that was-

Scott Guthrie: Observability and I’d say management, deployment, and the security layer, if you will, and compliance boundary, and those things are ultimately essential if you’re going to actually really use something in production and really use AI to kind of change your business. And I think we’re on the forefront in terms of providing those capabilities today. And one of the things I also said in the keynote is what’s nice is that all of the Microsoft Copilots that we build internally are built on the same stack. And so not only-

Patrick Moorhead: Customer zero. Yeah.

Scott Guthrie: So we’re customer zero, and in some way, we have more AI applications and more users using AI today than any other company on the planet. And so, having that battle hardening that we’re doing on top of that stack be the same thing that any organization that walks up to Azure can take advantage of, I think, is a key reason why it is so mature, and it’s really going to help us scale and give our customers confidence that it’s going to meet their security and operational needs as well.

Patrick Moorhead: Absolutely.

Daniel Newman: Well, Scott, let’s finish up here talking a little bit about trust and security. We know that we talked a little bit about the speed, diffusion of innovation, how fast it’s going. And I think there’s always been a bit of this, how fast do we go? How much do we bring trust, privacy, security into play? And I think AI has been a great example. And so, Satya and, I believe, you and all the others that got on stage today seemed very focused on making sure that wrapper of security was mentioned.

Scott Guthrie: Yep.

Daniel Newman: But how are you negotiating that in terms of strategy, product development, building, working with your partner’s community, Scott, to make sure that you maintain security, that you build the trust, that you keep data private, and at the same time, you can’t sacrifice going even a minute slower?

Scott Guthrie: Yep.

Daniel Newman: So how are you kind of doing that right now? Because it seems like a lot to take on.

Scott Guthrie: It’s hard. Balance-

Daniel Newman: At the pace, what did you have? How many hundreds of announcements it felt like today? I mean-

Scott Guthrie: It’s not easy. I would say it’s… But at the end of the day, both quality and security are the most important things that we can do. At the end of the day, if you don’t focus a lot on those two, nothing else matters. And so, that’s why we’ve been pretty clear, that is job number one, and that is what we were very clear that teams prioritize above all else. And if that means we slip an announcement or need to slip a feature, that’s okay, but make sure it’s as long as… But don’t ship anything if it’s not secure and it’s not high quality. And we’ve been driving this year a big initiative we call SFI, which has been literally probably 50 to 60,000 engineers. We’ve kind of said, “Tools down. We’re going to focus on security for many months,” and we’re still doing that, and it’s going to go on forever.

But it’s been a fairly intensive process where we’ve done a tremendous amount of defense in depth and hardening and then also building the platform that’s going to enable us to keep our systems secure, but then also build the platform that’s going to enable our customers to also use the same tools to make their systems secure. And so, a lot of what we’re doing with SFI is both hardening our systems, but also exposing those same tools and the same APIs and evangelizing to our customers how they can do it as well. We’re doing it both with software, but then also some of the announcements even this week that we’re shipping. We announced, for example, our new Azure HSM module, which is a custom chip that we’re putting into every single server going forward that allows us to kind of store secrets in an HSM, so it’s not in memory on any server.

We’re doing a bunch of work with our Azure Boost, which allows us to not only improve performance of storage and network, but also offloads and protects the system even more from a security boundary perspective. And I mentioned earlier, GitHub Advanced Security is another great example that we’re using internally as well to kind of do credential scanning, looking at code for potential vulnerabilities. And we take very much on the security front, I call it a platform approach, which is how can we take core capabilities, turn them into platforms again that we use as customer zero, but then also that we provide our customers so that they can also leverage as well. But you’re never done-

Daniel Newman: Sure.

Scott Guthrie: … and it’s going to be something that we all as an industry need to keep laser-focused on forever.

Daniel Newman: Yeah. Well, Scott, I want to thank you so much for joining us here on The Six Five. We know it’s a busy week, and you’re probably really going to be running from thing to thing, but congratulations on all the announcements. Look forward to having you back again soon. Have a great event.

Scott Guthrie: Thank you so much for having me. It’s been great to be here.

Patrick Moorhead: Thanks.

Daniel Newman: Thanks, Scott. And thank you so much, all of you out there watching the Six Five. We are On the Road here in Chicago at Microsoft Ignite 2024. That was Scott Guthrie. Great conversation. Tune in, subscribe for all of our content here at the event, and of course, all of our coverage on The Six Five. Pat, time to say goodbye.

Patrick Moorhead: Adios.

Daniel Newman: See you all later.