Accelerate AI Innovation in the Cloud

Patrick Moorhead:
Welcome back to the Six Five Summit 2024, and as hopefully you know it’s all about AI, the two flavors of AI, which is a continuing building out the infrastructure for AI, but as we’ve seen increasingly in 2024 enterprises and consumers getting benefit from it. And I can’t think of anybody other than Microsoft Azure opening up our track opener for cloud infrastructure for the summit. Introduce Amanda and Mike. Great to see you.

Amanda Silver:
Thanks for having us.

Mike Hulme:
Great to be here, Pat.

Patrick Moorhead:
Absolutely. Yeah, I’ve been tracking Microsoft and stuff that you did for years before it was Azure and I guess it was Microsoft Server before then, but it’s amazing to see what you have been able to build, and I think… Imagine that, Gen AI has been the talk it seems like for 18 months. AI wasn’t just invented 18 months ago, it’s been around for a long time, but boy, this generative AI thing has really heated everything up. And I’m curious, how have you seen generative AI adoption evolve and what are some of the common use cases you’ve seen your customers adopt?

And maybe we’ll start off with you, Mike.

Mike Hulme:
Yeah, absolutely. Well, thanks. I’m actually a fairly recent addition to the Microsoft team, so over those 18 months that you’re talking about, that first year I was watching really from the outside and really admiring all the great things that were happening here. And for the past six months or so, I’ve been able to have a front row seat. So obviously an exciting time period. Things have come so far in the last year, and probably the biggest thing that we’re seeing is this shift from more of an experimentation phase into a production phase. So all that energy and excitement that we saw over the last year of people really starting to understand what this can do for them and how they might actually take advantage of it, and now really translating into things that are delivering real value. That puts a different set of requirements onto the customers, it actually creates a different mandate for us as a vendor that’s really working to build out this tech stack as well.

So I’d say over the last year, what we’ve really been doing is learning, learning from our customers and then also learning from the services that we run ourselves, whether that’s Teams or Bing or GitHub Copilot, all of these are things that help us become smarter and then actually build that innovation back into the customers solutions that we can actually bring out for them. So we see this impact around production apps, really across the board for our customers’ application strategy. So that’s new applications and people that are building in intelligence and using data in a different way on new applications for their business, as well as taking existing apps and looking for ways to modernize those and modernize those faster in order to infuse them with more intelligence. And we actually see many customers leapfrogging their modernization strategies and saying, “I used to be modernizing just to get to a new infrastructure or a new architecture, but now I want to modernize really for the benefit of AI.”

Patrick Moorhead:
That’s great. Amanda, did you have anything to add to that? I’m sure you do.

Amanda Silver:
Well, unlike Mike, I’ve been at Microsoft now for 23 years, so I’ve seen a pretty incredible transformation of the company during that time in many, many different dimensions. And in some ways the first era of the cloud was really about replacing on-premises lab hardware. But I think in some senses, this generative AI period is really finding almost the killer app for the cloud. It’s how can we take the power of the cloud and actually transform it into intelligence that we can bring to all of our scenarios? And what’s interesting about it from our customer’s perspective, and we’ve certainly experienced this ourselves as we’ve been building out our own solutions, is there’s really five vectors of maturity that have to happen to be able to enact this.

The first is obviously the DevOps journey, which I think everybody has been on for the last 10, 15 years. The next is moving not just to cloud adoption, but to actually cloud-native architectures. And then the next one is platform engineering, thinking about how DevOps works in conjunction with cloud-native architectures.

Then the other thing that you really need to enable intelligent applications is data unification and integration because you also need to be able to reason over the data streams that you’re getting from your customers. And then finally, intelligent apps and building with AI is in and of itself its own maturity model in some senses where folks really start early with experimentation. They might start with very simple scenarios, and then over time, once they realize the power of it, they start to integrate it into more and more of their user experience and even a lot of their back-end operations. And so when we think about use cases, some of the most common use cases that we’re seeing is obviously summarization and Q&A. And this is obviously very often used for support scenarios, data-driven decisions. How can I actually make sure that we’re giving a particular customer the right treatment so that they don’t churn and they’re retained or they’re getting the offer that they need. Personalization so that your users are feeling more engaged with the products that you’re building. And then lastly, automation.

And we’re seeing most of the customers really explore the first and second very rapidly. A few are exploring the third and a much smaller percentage are actually really thinking about this as how they’re going to power automation of their businesses. But I think that just the Gen AI wave has really seen an incredible amount of innovation just over the last year, I think we’re going to see our customers really rapidly evolve over this coming year.

Patrick Moorhead:
Yeah, we’re seeing a lot of those similar applications in our research, and I’m so glad you hit on the data. With a lot of the enterprise that we run into is once they realize what it takes to do the data, especially when you’re crossing the streams between ERP data and customer service data or something like that, it seems to get hairy.

So Amanda, I’m going to keep you on the spot here. What are some of the key learnings that you’ve taken from your journey in Gen AI that you might be able to, your customers might be able to apply and learn from?

Amanda Silver:
Yeah, I mean, our philosophy is always how can we take our learnings and actually build it into our developer tools and platforms so that our customers can also get those same benefits from what we’ve learned? And we’ve been working on GitHub Copilot, which is one of the most popular Gen AI tools that actually helps developers build, write code more quickly and learn about their code basis more quickly. And we’ve been taking our learnings from building that, and we’ve actually been incorporating that into our tools and our platforms. And some of the things that we’ve learned is really the inner and outer loop.

When you think about developer cycle, there’s the inner loop where the developer does the edit, build, debug kind of process, and then there’s the outer loop where they collaborate with the rest of their team and they do continuous integration and things like that. What we’re seeing is that when you move to build features with AI, that actually changes a bit. What you need to do is you need to evaluate how good is the model at accomplishing the goal that you want it to accomplish. And then you need to actually do cost comparisons to understand which model you should be employing. And you need to actually look at the prompt evaluation to be able to have confidence that it’s going to stay grounded in the data that it should be grounded in and not suggest things that don’t make sense for the conversation or are off-topic. And so it starts with really just changing the way that developers actually work. And what we’re seeing with that is it used to be that there were machine learning scientists, data scientists and developers. And what we’re seeing emerge is this idea of an AI engineer who really understands the models and the capabilities, but also understands prompt evaluation and how to incorporate that into the application architecture to not just deliver great user experiences, but also great cost and performance and security and quality and everything that we do as application developers.

And so the next thing that comes there is not only do we need to make sure that our application platform really supports what they need so that we can have fantastic cost optimizations and things like that so that they can manage the tokens that they need to give to each of the different applications or be able to take, if there’s a computation that comes back from the large language model as an example, that they can actually do that in a safe way without risking the other tenants in their multi-tenant application. So we need to build a lot of those intrinsics so that developers can really build the applications that they want to.

And then lastly, it impacts how you think about platform engineering. What kinds of governance and policies do you need to enforce across your application at state as they start to build with AI? So one of the first things that we started to do inside of Microsoft is we have responsible AI requirements, and we had to make sure that we were actually applying those filters and those threat models in a sense to figure out what did we need to check for before we shipped things to our customers.

Patrick Moorhead:
Now this is great stuff and so many learnings so quickly, and you’re learning even inside of Microsoft. And then ultimately you need to tee up the tools and services that make developers even more productive and efficient, and ultimately those enterprises do all the magic that generative AI promises. So how are you building the right set of tools and capabilities to help developers build Gen AI apps? Amanda or Mike, I’ll put you both on the spot here.

Mike Hulme:
Well, Amanda is the expert as you can see, especially from all the great things that we’ve learned internally. And maybe I’ll just outline a couple of things that we think about. I think with every shift that we have in applications and across our careers, we’ve seen so many, there’s so different things that you have to think about. You have to think about end-to-end, what changes and how organizations can really absorb a shift like that. AI is really an interesting one because there’s a greater affinity between the technology and the business outcome. So one of the things that we have to really think through is how we help customers identify a clear ROI or a business value for what they’re doing. When there’s so much excitement in the market about a new technology there’s often this energy to do something fast, but we also want to make sure that the impact is there and that the value comes through. So we’re thinking through-

Patrick Moorhead:
Yeah, I mean after a while, that tops down board of directors, C-suite pressure down actually has to come up with something tangible in the end and valuable. You got to nail that first one in POC, otherwise it throws everything off and trust is lost.

Mike Hulme:
That’s actually something that we saw with the cloud generation as well. There was a lot of mandate to move fast and move to the cloud really quickly, and then the burden really ends up on the developers and the IT leaders to really do it right, and to convert that into a really tangible strategy with value. I think we’re even seeing that even more so within the AI market right now. So number one, that’s one thing that we’re looking at is really how do we establish whether it’s just a conversation or a model or even templates that can actually help organizations move quickly and start to build some of the most in-demand patterns that they need within their applications. But the second thing is really in the tooling. I think part of what we’re looking at strategically here is how do we help the largest number of developers come with the skills that they have and work within the tools that they love and actually be productive, be efficient, and build the kinds of services that their business needs.

And then another area is really looking at how do we remove some of that friction between the development and deployment processes such that we’re building applications that are secure, that are scalable, that have the performance, that have the reliability that they need to be in production, but we’re eliminating that burden from the developer while also ensuring that application meets the needs of what an AI application or an intelligent application really needs from a scalability standpoint without eroding things like cost. Again, all of that’s about protecting value and making sure that the business gets the most out of that AI application. But Amanda, you have some great ideas about how we’re putting that into motion and can dive a little bit deeper.

Amanda Silver:
Yeah, I mean, I think one of the things that we see is you start to tiptoe into Gen AI and then pretty quickly you’re neck deep incorporating it into a bunch of different applications. But what we see in terms of the adoption model of a bunch of customers is that they really start with generating the ideas of all of the different use cases that they think are going to be valuable for their organization. And a lot of them will end up being customer-facing, but a lot of them are really more about driving back office efficiency. So the first step is really just evaluating the use cases. Then they start to do proof of concepts with internal apps to see, to Mike’s point that ROI, what is the return? How effective can it be in terms of improving efficiency? Then they might move, once they reach success there, to external apps and exposing it to customers and then moving to production. And obviously that comes with a whole host of additional requirements as you make it available to more and more customers.

But then eventually you have to get back to the point that Mike was talking about, which is what is the value that you actually think you’re providing to your customer and to your business? And to do that, you actually need to build in a loop for continuous improvement. So you need to actually have data feed from the intelligent app scenario, and then you need that to actually feed back into the application that you’re building.

And then we want to make sure that from our perspective, that we’re really enabling the developers and the businesses to move through those different adoption stages. So we want to help them explore these different use cases and then give them the tooling and the developer experiences that allow them to really start easily and explore and then iterate very effectively and deploy quickly, but then be able to get plugins that allow them to do experimentation in production so they can actually see the statistically significant results of their AI scenario that they’re trying to apply it to. And then to that point, then it starts to become this question of how do they achieve the cost optimizations as they scale the application to more and more users?

Patrick Moorhead:
Now, I appreciate that and great conversations about Microsoft tools and its customers and your journey, but Microsoft has always been a partner company, and we’ve seen this, which ended up being true, enterprises wanted a combination of closed and open models, and we’ve seen what you’ve been doing with different types of model providers. What about on the application space? Who are your partners with on that? And maybe Mike, we’ll start with you.

Mike Hulme:
Yeah, absolutely. So we see just a rich opportunity to work with a big ecosystem of partners around this overall AI tool chain. And our goal is to give developers the flexibility to work with the tools that they need to and then make it incredibly easy for them to bring those tools into something that’s integrated and feels very natural for them. So what we’ve actually been doing is working with a growing set of partners, think of them as AI ISVs or AI tool providers that really provide discrete functionality. And a lot of this functionality is actually really widely adopted by that particular AI engineer that Amanda outlined earlier. So we’re seeing a lot of affinity for some great technologies out there. And in fact, at our Microsoft Build event, we announced some new and expanded relationships with a set of core providers. A lot of these are coming together in what we would consider to be an AI stack focused on very specific use cases like building a rag enabled application.

So just at Build, we announced new relationships with Arise and Lang Chain, Llama Index, expanded relationships with Hugging Face and Pinecone, and all of these are great services. They’re in use within the type of AI engineer that we think is on the front lines of building these services. But beyond those relationships, one of the things we really want to do is make it incredibly easy for developers to get access to those services. So within each of them, we’re building unique integrations. Some of those are direct to Azure, some of those are through GitHub. And we’re also looking at ways that we actually embed those technologies into the templates I mentioned earlier, so that the building blocks for the services and applications that developers are building actually have pre-builds and integrated access to those services.

Our goal here is really for the right services for a small set to give developers a first party experience so that they feel like they’re operating within an integrated environment. And in the end, it’s about really helping developers use the things that they love and bring the skills that they have and be incredibly effective at building the applications that they need to.

Patrick Moorhead:
Yeah. Amanda, how about the, I don’t want to say the nuts and bolts side, but any adders from you?

Amanda Silver:
Yeah, I mean, I think to Mike’s point, that point around having it be as integrated as possible is a really critical dimension of it. Developers oftentimes have to interface with a huge number of disparate tools, and every time they switch tools, that actually impacts their focus and concentration. And what we see is that it takes about 23 minutes every time they do a context switch to regain that focus and concentration. And so what we’re really trying to do is to make it so that all of these great tool and platform vendors are able to integrate into the developer experience where developers hang out, which is Visual Studio Code in Visual Studio in GitHub.

And so, one of the things that we announced at Build is that we’re making GitHub Copilot extensible. And so we can take basically all of these different types of extensibility scenarios for all of these different ISVs and incorporate their experience directly into GitHub Copilot. So if you think about the pair programmer experience that GitHub Copilot provides to you, now it has Docker knowledge, now it has Pinecone knowledge now, it actually could have knowledge about the specific frameworks that are internal to your organization and give you better insights as a developer, how you should be building that application that’s going to adhere to the best practices of what it means to use Pinecone or what it means to use Lang Chain or whatever.

Our goal here is to make sure that ultimately that developers who use our platform, which is an open platform, allows you to build any programming language for any platform target, but it is open and modular and extensible by design. And so what that means is you can bring the best of breed tools for any scenario or job to be done and incorporate that into your developer experience.

Patrick Moorhead:
Yeah, I really do like the meet your developers where they are. You can start with a Microsoft experience and then go from there or you can start with a Docker experience or Pinecone and go from there. I like that. It’s almost like, how could you say no?

So early on in the discussion with Azure and AI, we heard a lot about Azure AI services, essentially your app platform. We also heard about Azure OpenAI. And I’m curious, what is your strategy for making Azure the go-to platform for developers building apps? And by the way, I think your initial value prop was really good, but maybe just reiterate what your strategy is and what you’re doing.

Amanda Silver:
Yeah, I mean, I think it really starts with, first of all, making Azure a great place to build AI itself. And obviously, OpenAI was built on top of the Azure infrastructure. We have many, many ISVs that are building their own IP, which is itself an AI model that are using the same core Azure infrastructure to build that technology. But then on top of it, we want to make sure that Azure really provides a great way to explore the breadth of all of the different models that are out there. So we have, for example, with Azure AI Studio, the model as a service, model catalog in a sense, so that you can explore different models and actually swap them out so that it’s really easy to compare the performance versus cost of one model versus another, including not just models that are served up by Microsoft on Azure, but also client models that you might actually, like the recently announced fee model that you might want to have run locally for cost optimization purposes if it’s still meeting your goals.

And so with that, we have to make sure that we have the evaluation loop really well-supported. But then beyond that, we want to make sure that not only do you have the AI models being served up by Azure and allow you to have cost optimizations for your solution, but also that the application model is also primed to host the unique requirements that are really coming in with the era of intelligent apps and AI. So we have to make sure that we have support so that they can get the scale and the reliability that they need. And one of the things that’s so interesting and challenging in some senses about working with AI is it’s not discreet and it’s not as predictable as our previous era of technology. And so a lot of times what you have to do is you’re basically working in a stochastic world and you need to create guardrails, but also you sometimes need to dynamically revector which model you direct traffic to based on the prompt that’s coming in from the user. And so being able to do that with, for example, our Gen AI gateway and Azure API management really helps with that.

And then also be able to do things like extract and store document data, have memory and history so that you can provide more context to the model as you’re trying to come up with the right response, provide summarization capabilities. A lot of this really is about the support for the interactive human interface of these applications, but as I mentioned, a lot of the scenarios are also back office scenarios to improve workflow processes. And for that, you also need to have data ingestion and data cleanup tasks where large language models can be incredibly helpful for that.

Patrick Moorhead:
No, that’s great. And this has been a great conversation, and we’re coming up to the end so we have time for one more question here. We talked about the incredible past related to developers and Microsoft Server. By the way, I worked for a company that was your first Windows NT and Windows Server OEM. But I want to talk about the future here on what’s, not looking for you to give us your entire roadmap, but what should we expect in the future? And maybe we’ll start off with you, Mike.

Mike Hulme:
Yeah. So I think if we look in the immediate future, what we really see is this idea of just intelligence in applications. We think that’s just going to be table stakes. That’s going to be the norm for most applications. In fact, in general, the conversation is about how do I get the vast majority or all of my applications to have some sort of intelligence built into it, whether that’s a new app or modernizing an existing app. And I think there’s just going to be this massive wave of AI applications that are then coming in, and we have to start thinking about the infrastructure that underpins all of that and ensuring that it really provides all the optimal requirements and qualities that those applications need.

So the immediate future is really about helping everyone move to production and do it at scale. And that means we’re thinking about how that process continues to evolve and become more sophisticated while also becoming more simplified. How we remove that friction that can exist between developers so that they can just continue to focus on building their services. And also making sure that the new requirements for these applications, whether that’s around security or performance or getting a better lens on things like cost as these applications really scale in production, that all of those are managed and they really have what they need, such that the outcome for the business is one that meets their requirements. But that’s just the beginning. I mean, we have some great things coming, and I’ll let Amanda talk a little bit around what’s down the road.

Amanda Silver:
Yeah, I mean, think in a sense, I think this is the year that Gen AI is crossing the chasm. For the last year, it’s been a lot of bleeding edge folks incorporating it into their solutions. But I think this year what we’re going to see is that folks who have not taken a look at it are going to take a look at it, and they’re going to get started very quickly with default templates and things like that that we are making available in GitHub and targeting Azure. But also what we’re going to see is sophistication, more sophistication of the folks who are already building with generative AI and more standardization of best practices across the entire industry. And so we’re going to see, in many cases, different best practices like experimentation become really solidified, or evaluation loops just become normal part of your developer stack.

Patrick Moorhead:
No, this is great. Amanda, Mike, I really want to thank you for the opener for the cloud infrastructure track here. I’m a student of history, and not that the past always repeats itself, in fact, many times in inflection points, it doesn’t. But I think we all have a really good idea of what you’re doing at Azure. And by the way, if you want to know where to start, hit the Azure Developer template, gallery. There’s a lot of great stuff in there to do. So Amanda, Mike, thank you so much.

Amanda Silver:
Thank you.

Mike Hulme:
Thank you, Pat.

Patrick Moorhead:
Yes, this is Pat Moorhead. That was the track opener for the cloud infrastructure. It’s all things AI. It’s Six Five Summit 2024. We appreciate you tuning in. Take care. And hey, you can view all of the amazing videos and tracks that we have out there, and they’ll be available on YouTube in a couple of weeks too. Take care.