Barry Luijbregts

How to troubleshoot Azure Web Apps

As part of its Platform-as-a-service (PaaS) offer, Microsoft offers the Azure App Service Web App (previously known as Azure Website). You can use Web Apps to host your website in the Azure Cloud (somewhere in a Microsoft Datacenter). This hosting is different from hosting in a traditional datacenter or the servers in your company. Not only this, but the underlying platform (PaaS) forces you to design and develop your website in a different way.
Illustration 1 shows the difference between traditional hosting and hosting of an Azure Web App in PaaS:
(Illustration 1: Traditional hosting vs. PaaS in Azure)
In traditional scenarios, you develop a web application (e.g. with ASP.NET) and host it in IIS on a (virtual) server in your datacenter or the server room a couple doors down. Over the years, we have become spoiled with the notion that we can rely on the physical machine on which our applications and services run. For instance; often we still assume that we can get to the file system or that session information will stay in the memory of the server.
When you develop a Web App and host in in Azure, the story changes. The basic idea behind a Web App is that we consume the hosting as a service, which is geographically scalable, redundant and secure. In practice, this means that we no longer have a 1-on-1 relationship between the application and the server. The Web App is hosted on 1 or multiple servers. Or the Web App is running on a shared server, together with application of other Azure customers. The magical Azure abstraction layer (Azure Service Fabric) makes sure that the Web App keeps running. When a server fails, Azure provisions a new one. When the application requires more performance, Azure will provide more server power by running more instance of your application and routing traffic through a load-balancer. In short, you can no longer assume anything about the server on which your Web App runs. You can no longer assume to be able to access the file system of the server as you could be moved to a new server at any point in time. The same goes for other resources, like the server memory.
This requires a new way of working. Your Web App should assume that resources like the file system and server memory are not (always) available. This is something you should already have been doing in your applications (I’m sure you have), but by hosting in Azure, this aspect becomes more visible as resources change more often (server gets added, changed, removed). You should also either work stateless or manage your application state differently. You have to assume that multiple instances of your Web App could be running and that traffic is distributed to these instances by a load-balancer. This means that you can’t keep the state of your application in the memory of just one server, as the state would than not be available on other instances where your user might end up on through the load-balancer.
Troubleshooting of a Web App is also different from troubleshooting an application hosted in a traditional datacenter. This is mainly due to the fact that you cannot log on to the server (through RDP) on which your application is running. Because of this, you can’t view your server log files and use other traditional troubleshooting techniques. This can be difficult if your are having trouble with your application.
I started early on with software development in the Azure Cloud. To me, it’s all about giving away some control to gain the benefits of using capabilities out-of-the-box. Nowadays, I really like developing for the cloud. However, a few years back, Azure was on a completely different level of maturity than it is now. This caused a lot of my customers to not yet get into Azure, often because the loss of control seemed too large to them. A lot of times this came down to the fact that support employees would not have enough tools to troubleshoot the application. A lot has improved in this area
So what are your options after you publish to Azure for the first time and face a weird error message? How can you then get to the bottom of the error and fix it?In this blog post, I’ll show you a couple ways of how to do this and make your manager happy. As always in the world of Microsoft,there are many ways to accomplish the same thing. Therefore, I will restrict this blog post to the following topics:
* Remote debugging
* Log file analysis
* The Azure App Service Support portal
* The Kudu website
* Application insights.

1. Debug an Azure Web App

A good way to troubleshoot errors, is by debugging your applications. Luckily, Azure Web Apps exposes the ability to use remote debugging. To not make this too easy, there are also drawbacks for remote debugging, like that the application stops when you hit a breakpoint, which is not what you want in production.
To enable remote debugging, you need to change the settings of your Web App. Please note: you need Visual Studio 2015, 2013 or 2012 with update 4 and you have to have the Azure SDK installed on your machine, which you can get here. Also, remote debugging does not work with Community Editions of Visual Studio.

Step 1: Configure your Web App

Go to the Azure management portal (for this post I use the preview portal https://preview.portal.azure.com). Once logged on, go to your Web App and select All Settings > Application Settings. This takes you to the screen in illustration 2.
Illustration 2: Configure remote debugging in the Azure Preview portal)
Switch Remote Debugging to On en select the Visual Studio version that you are going to use. Don’t forget to click Save. You can also change these settings from the Server Explorer Window in Visual Studio.

Step 2: Debug from Visual Studio

Open Visual Studio and open the Server Explorer Window. This will show an Azure node. If you expand it, Visual Studio will ask you for your Azure logon credentials. Once logged on, almost all of your Azure resources are shown in the Server Explorer.
Next you can attach the debugger by selecting Attach Debugger from the context menu of your Web App, like in illustration 3.
After this, the debug process is pretty much the same as what you're used to. You can see how far the code will take you by placing strategic breakpoints, or wait for an exception to happen and hope to catch it. For the latter you can help yourself by choosing the right exceptions to be caught by Visual Studio in the Debug Exception settings (by default it’s CTRL+ALT+E)
Things to note on remote debugging:
  • Remote debugging only works when you have a debug version of your application deployed
  • After 48 hours, remote debugging will shut itself off (for performance and security reasons)
  • Remote debugging in production is not a good idea, because:
  - You need to publish a debug version of your app. This is not good for your application performance
  - When you hit a breakpoint, your application stops and users might be waiting on you to step through the code
  - Remote debugging in production is possible by using IntelliTrace, which currently is only available for Visual Studio Ultimate Users. As of July 20, 2015, this will also be available for Visual Studio 2015 Enterprise edition users
Remote debugging is a good idea for development and test scenarios.
Debugging can only get you so far. It can’t help you if your application doesn’t even get to a point where you can set breakpoints and analyze things. In those cases, you need different ways to troubleshoot your application. One possibility is to analyze your log files.).

Azure Web App Logs

So you don’t have access to the server on which your Web App is running. Fortunately, there are ways to get access to log files and help you out.
There are several types of log files that can help you troubleshoot your Web App:
* Application logging
    * This log contains trace information you write from your application
    * Application logging can only be turned on for 12 hours. After that, it shuts itself off
    * You can indicate the logging level: Error, Warning, Information or Verbose
* Web server logging
    * Information of the webserver itself
* Detailed error messages
    * Detailed information about exceptions in your Web App
* Failed request tracing
    * Specific information about requests to your Web App that fail
Before we can look at log files, we need to configure the Web App to log things. Navigate to your Web App in the Azure Preview portal and go to All Settings > Diagnostics logs. This takes you to the screen illustration 4.
(illustration 4: Logging configuration in the Azure Preview portal)
Depending on the type of logging you need, you enable it per type and click Save to enable your settings. Also these settings can be changed from the Server Explorer in Visual Studio.

Application Logging

Application logging contains messages which you write from your application through Trace messages.
This is only useful when you are investigating a specific piece of code in your application. I used this a couple of times to analyze performance problems. When you need to analyze exceptions, you’ll probably run into the fact that your trace messages don’t trace the exact bit you need. Here is how you use it:
The **System.Diagnostics** namespace allows you to write messages from your application. You do this by using Trace.WriteLine:
1 public ActionResult TestTracing() 
2 { 
3 Trace.WriteLine("TestTracing bericht"); 
5 return View(); 
6 }
After calling this method, we can find this trace message in the application log. There are many ways to save logs and access them (through FTP, Azure Storage, the Azure Preview Portal and third-party applications). For now, we will use Visual Studio to analyze the log files.
Go to Server Explorer and select View Streaming Logs from the context menu of your Web App, like in illustration 5.
(Illustration 5: View Streaming logs from Server Explorer)
Now, the application log is visible as a streaming log in the Output Window (illustration 6).
(Illustration 6: Streaming Azure Logs in Visual Studio Output Window)

Web Server Logging

The Output Window can also be used to analyze the Web Server logging. You can configure which type of log files are visible in the Output Window in the Logging Options (visible in illustration 7). You can reach these through the settings button that is highlighted in illustration 6.
(Illustration 7: Streaming Logging Options in Visual Studio)

Detailed Error Messages

If you select All logs in illustration 7, the detailed error messages will also be shown in the Output Window. These messages point to detailed error pages that are very useful for bug hunting.
(Illustration 8: Detailed error messages in Streaming Azure Logs)

Failed request tracing

The failed request tracing logs are most useful when you are tracking down an obscure bug, that would only be tracked down by analyzing individual requests in the server logs of IIS.
The failed request tracing logs are a bit more difficult to view than the other logs. This is because these logs come in XML files and the streaming logs in the Output Window only show .txt, .html and .log files. However, you can download the failed request tracing logs through FTP (amongst other means).
To use the FTP of a Web App, you first need to configure the FTP login credentials. You can do this through the Azure Preview Portal. Go to your Web App and navigate to All Settings > Deployment Credentials and fill out a username and password for the FTP user. This is the same user you can use to manually publish your Web App (from Visual Studio).
You can find the FTP hostname in the Azure Preview Portal under All Settings > Diagnostic Logs. Now, you can navigate to the FTP site using an FTP client (or web browser) and access the Failed request tracing logs.
You can find the logfiles in the folder LogFiles > W3SVC (and some numbers). Here you’ll find XML files which contain the failed requests that will hopefully get you further in your quest to hunt and kill your bug.
(Illustration 9: Failed Request logs in FTP)

Azure App Service Support

A new feature that became available in public preview is Azure App Service Support. You can reach the Azure App Service Support portal by navigating to (Web App) Tools > Troubleshoot > Go (for live http traffic or Mitigate) in the Azure Preview Portal. This will bring you to the page shown in illustration 10.
(Illustration 10: Azure App Service Support Portal)
The support portal provides you with the following features:
* Observe: See the average requests/sec and errors/sec per hostname. Very useful to see how many errors occur in your site (if any).
* Analyze: This functionality is impressive, especially because it is still n preview:
    * You can start a diagnostic session. In this, you can configure what you want to monitor. You can choose from: Event Viewer Logs, Memory Dumps, Http logs, PHP Error logs and PHP Process report. You can analyze all of these metrics;
    * You can see the event log and view details of events;
    * There’s a metrics tab, which shows you graphs of CPU / Memory, Http requests and network traffic. Very useful if you are looking for a quick overview of the health of your service. From here you can also restart the W3P process that is running your service;
    * You can view the Failed error request logs. This is only possible if you are capturing these logs.
    * Note: these functions (diagnostic session, metrics and failed error request logs) are not available for Apps running on the Free / Shared pricing tier.
* Mitigate: This allows you to solve problems with “auto healing”. In the auto healing functionality, you can create rules to restart the W3P process. You can create rules based on different filters: Maximum number of requests, Http status code, slow requests and amount of private RAM in use.
The support portal provides quite some functionalities, even if it is still in public preview. The functionalities will surely be added onto if the portal is found to be a success. If so, my guess is that this will be the central place to troubleshoot your Azure App Services Apps.

The Kudu website

There is yet another way to troubleshoot your Web App: the Kudu website. This is a site that runs next to your Web App. You can find the Kudu project on GitHub: https://github.com/projectkudu/kudu 
On the GitHub page, you can read the following: “Kudu is the engine behind git deployments in Azure Web Sites. It can also run outside of Azure.” This is one use of the Kudu website. However, it can do much more, like show the Apps’ configuration, processes and health.
You can get to the Kudu website by navigating to https://yourwebsite.scm.azurewebsites.net. You will be asked to log in with your Azure credentials. Once logged in, you will see the page in illustration 11.
(Illustration 11: the Kudu website)
You can see all sorts of things in the Kudu website, from environment variables to running processes. You can view log files and look at streaming logs (just like in Visual Studio). You can also navigate through the filesystem of the machine (or sandbox) where your Web App is running. Here you can view, but also edit, delete and add files.
You can also manage site extensions from the Kudu website. These are extensions that are developed by companies or the community. There are all sorts of extensions and the list is growing. E.g. extensions to help you troubleshoot your App, extensions for Visual Studio Online, for analyzing your Web App, like New Relic and more…
The site extensions are also available from the Azure Preview portal. Go to your Web App and navigate to All Settings > Extensions.

Application Insights

To analyze your Web App and trace errors (not so much for troubleshooting), you can use Microsoft Application Insights. This is a suite provided by Microsoft that brings capabilities similar to those of Google Analytics. Application Insights goes further than showing you error messages; you can analyze the complete usage of your Web App. At the moment, Application Insights is in public preview and can be used from Visual Studio, the Azure Preview Portal and Visual Studio Online. There are several ways to get Application Insights working with your Web App:
* From Visual Studio. When you create a new project, you can add Application Insights from the get-go or later, when you have already created the Web App
* From the Azure Portal. You can create an Application Insights instance for your Web App. To make this work, you need to add a piece of JavaScript in the pages of your App, in order to collect data
In Application Insights, you can see the following data:
* Availability
    * To see if your SLA is being met
* Failures
    * Failed requests, dependency failures, exceptions and server requests
* Performance
    * See how fast or slow your pages are
* Server metrics
    * You can see the server metrics of all the servers on which your App runs, like CPU and RAM usage
* Usage
    * You can see how many (unique) visitors saw which pages and how long they stayed there.
    * You can also see characteristics of your visitors, like the browser and OS they use and the screen size and such
* Custom events
    * You can define your own events by calling Application Insights from your App and writing a custom message. This is the same principle as Trace messages.
This is all very useful when you want to monitor your Web App to see if there are any errors and how many. When an error occurs, you can easily drill down to the page on which it happened and zoom in on the exception. Illustration 12 shows the overview of the Application Insights Failures page.
(Illustration 12: Application Insights Failures page)
I use Application Insights at the moment mostly for usage tracking and analytics. The troubleshooting part of Application Insights has potential. However, the manner in which you can zoom into the error is still pretty basic, specifically when you compare it to the App Service Support Portal.

Conclusion

Developing and managing Azure Web Apps is very different than doing the same in the traditional Website + Server + OS + IIS model. You cannot rely on a 1-on-1 relationship between Web App and the server it is running on. Luckily there are plenty of ways to troubleshoot a Web App. It is obvious that Microsoft is putting a lot of effort and resources in expanding the developer experience for Azure.
It looks like Microsoft is focusing on the Azure App Service Support portal for troubleshooting (Web)Apps. When this becomes a success, other options, like Kudu, might fade away.
One thing is for sure; the developments in the world of Azure are going at lightning speed, so we can expect more capabilities soon.
Created with