NoteYou can check out Part 2 and Part 3 of the series here.
Let me guess. You’ve seen the light when it comes to the cloud and migrated your application to Microsoft Azure. Your site is hosted on an app service big enough to run a BitCoin mega-mine, and your database is a technological spectacle of SQL SaaS goodness. You get everyone in the company together to toss you on their shoulders and marvel at your new cloud-powered creation, and its performance…well… sucks. Before you go rolling your site back to the 486DX in the supply closet, let me give you some tips in this two-part blog series on how to get it running smoothly in Microsoft’s cloud.
I’ve been working with Azure a long time. In technology-terms, I suppose I’m a great-grandfather when it comes to cloud computing. If not that, at least a distant relative that shows up at family dinners and steals stamps. Either way, since 2008, I’ve created applications for it, written articles about it, given bootcamps for it, and waved the Azure banner on every street corner I could. I love the platform and am constantly amazed at how much you can achieve with it. As an Azure MVP, I let everyone know how great working with the cloud can be at every conference, event, code camp, wedding, and parent-teacher meeting I attend. Yeah, I kind of like working with Azure.
But to be honest, when running in the cloud, things don’t always go smoothly. I’ve seen applications that can be lightning fast in one environment crawl to 14.4K speeds when deployed to Azure. I’ve had the most seasoned software engineers question everything about the platform, and question how it can be so great with such bad latency and performance. They sit baffled at their deployments, with no obvious reason for their painfully slow sites.
I totally get it! The cloud gets touted as this wunderkind of technology that will solve your problems!
But what gives with the craptastic speeds?!? I’ve worked with a lot of clients and developers over the years to help them answer that question when it comes to their Azure applications. Let me illuminate the possibilities…
Read the manual
One of the biggest reasons for bad Azure performance comes down to education. This is especially true of developers just getting started with the cloud and not fully understanding what the platform is. Azure isn’t one thing. It’s 100s of things (and growing!). It’s infrastructure, connectivity, functionality, scalability, and flexibility, all rolled up into a five-letter name that no one can seem to agree on how to pronounce. Because of its vast capabilities, fully grasping all the individual pieces can be a daunting task, especially if you’re new to the game.
Every time I begin to diagnose someone’s Azure woes, I first start by ascertaining what they know about the cloud. Do they get why cloud computing even exists? Do they understand Cloud Services vs App Services? Did they lift-and-shift their app to the cloud, or was it architected there from the beginning? Did they have any Azure expertise on-hand for the process, or did they wing it? These are all important factors when it comes to why a particular architecture or configuration was chosen, and can set the stage for some seriously long application loading time if not done correctly.
Remove the variables
As any developer who has tried to hunt down that phantom bug that only Linda in accounting can produce, you need to take a pragmatic approach when it comes to determining why your Azure site isn’t performing. Rule out as much of the equation as possible. If you have 100 variables in your environment, you’ll spend most of your time hunting and pecking your way to insanity. Save yourself some headache (and therapy) and eliminate as much of the riffraff as possible.
If you have multiple apps on the same server, isolate them to their own instances. Are your application database and storage in different data centers? Set up your own Three’s Company episode and get all your resources in the same region. Does the issue only happen to authenticated users or a specific area of the site? Try to identify the common denominators to help isolate the issue.
Knowing what NOT to look at can be one of the best steps you take toward getting your performance issues resolved.
What’s causing the problem?
I combined decks to make it more exciting!
OK, so you know your team is skilled in ways of Azure-ology, and you’ve removed as many variables as you can. Now, you need to start trying to identify where the problem is. If you’re lucky, it’s a single factor in the environment that causes the problem. If that’s the case, stop what you’re doing and go buy a lottery ticket or something.
Chances are, you’ll have several components that are causing you grief. You just have to figure out which ones. And you will need to prepare yourself to change more than one thing. In many cases, it’s several aspects of the deployment that contribute to the slowdown.
App Service
Determining if the web server (App service) is the root of your problems tends to be a pretty simple process. In the Azure portal, you can check out the server performance and see if you’re getting spikes in your CPU and memory. If you find your app is causing your server to max out on a regular basis, you want to start thinking about scaling up or out. One of the best parts of Azure is the ability to scale your web environment quickly, automatically if needed. Check out your App Service plan settings and adjust until you get the performance you’re looking for.
This is one of the simplest things you can do to address performance, but only if your problems are web server related. To be honest, I rarely see the app service as the cause of the problem. More often than not, it’s a data and/or architecture related issue. If the server looks like an EKG for the Tin Man (no heart beats, i.e., no spikes), it’s time to keep searching.
Database
A lot of developers will start with hosting their database in SQL Database (or Azure SQL…or SQL Azure…or SQL Database for Azure. I’ve seen more than a few name changes.). And for good reason! It’s an extremely scalable SQL environment that you can get for a few dollars a month (depending on your needs). Your data is automatically replicated on the backend for DR and you can easily deploy your database via SSMS and connect your website in seconds. What’s not to love??
Well, there’s a mythical creature called a DTU (Database Throughput Unit) that has eluded developers for years. DTUs are not a single measurement, but rather a collection of variables that Microsoft uses to calculate the resources your database is consuming. These are CPU, memory, and read-write rates. (trust me, after nine years, I still get a little confused). Microsoft uses DTUs to figure what kind of resources your database is consuming, to make sure everyone on the platform is getting what they pay for.
When it comes to bad performing Azure sites, DTU spikes are usually at the top of the discussion. Whether it’s long running queries, poorly-performing indexes, or just bad code that causes way too many calls, an overworked SQL Database can slow down any application. If you find your SQL Database is beaten down and struggling, check out the Support & Troubleshooting section of the Azure portal for help. Use those tools to figure out which queries are taking up the most time, and when your DTUs are spiking.
DTU issues are probably the most common culprit I’ve seen in the past 9 years.
Serving images
Just hold that position for another 32 minutes.
Going back to the first point about education, handling images is where I’ve most often seen developers not using Azure to its potential. When it comes to media, developers should rethink the “deploy-with-site” model and explore the unique capabilities that Azure can offer. Because Azure storage is so flexible (and easy to configure), not using it for images in an application is a huge mistake that could be costing your site a lot of precious performance. The Azure CDN functionality gets included with Storage, so serving your site media from a global network of data centers (instead of where your site is deployed) can drastically speed up load times throughout your site.
In addition, offloading your images to the CDN means less calls to your main URL to load content, which means a lot better parallel request management.
Maybe it’s the code
Aww, man! I got the Gray Screen of Death!
No amount of memory, processors, or servers can fix bad code. When all the architecture areas have been addressed and a site is still slow, I always ask about the custom functionality. This is where things can get tricky, and take a lot of time.
If you’re using a 3rd party platform for your site, chances are that company has put a lot time into making sure the “vanilla” version of the product will run in any environment. With that in mind, your custom code may be the villain. Make sure your code base is up to date with the latest fixes, and then start digging into your functionality.
You will want to make sure you are accessing your data as efficiently as possible. Are you leveraging caching and vendor APIs to store data in memory? Are you pulling back only the data you need, reducing the overall payload? Are you trying to divide by zero or some other wormhole-producing action in your custom functionality?
You need to dig through every piece of your code and make sure it’s as optimal as possible. Unfortunately, reworking code is never an easy task and one you’re going to have to pack a lunch for, especially if the intern in the cubicle next you who wrote the entire contact management module.
Diligence is key here, so don’t cut corners! Be sure to check out anything with a database connection, memory access, or functionality that relies on external systems. Learn how to leverage Azure caching and storage to improve speed and latency. You may find you need to update several areas of your site. Azure is a big mountain to climb, so take it one step at a time.
Time to think things over
OK, so your mind may be a bit like a Game of Thrones family tree right now. You’re probably already processing all the possible areas you may have to update, configure, and rework within your site. I’m going to wrap this blog up here to give you time to process things. In my next article, I’m going to give my recommendations on how you should architect your Azure applications to minimize performance issues in the future. It will also be packed with more tips on steps to take to identify issues, implement stop-gap solutions, and build better Azure sites. Until next time, friends.
UPDATE!
Want to get some real tips and tricks? Check out Part 2 of the series!