Monitoring errors in Azure WebJobs with the ErrorTrigger extension

We will learn in a simple scenario how to monitor errors in an Azure WebJob.

When developing jobs with Azure WebJob, it's a good practice to implement error monitoring in case something goes wrong when a job is executed.

The WebJobs ErrorTrigger extension, part of the Core extensions, will help us achieve that.

Today we will learn how to monitor errors with Azure WebJob using the ErrorTrigger extension. As a base, we will create a project with the Azure WebJob template in Visual Studio.

Creation

The first thing you will need is to add the following NuGet package to your project: Microsoft.Azure.WebJobs.Extensions

To enable the ErrorTrigger extension in your WebJob you will need to configure the JobHostConfiguration like the following:

using Microsoft.Azure.WebJobs;

namespace AzureWebJobs.ErrorTriggerExtension
{
	// To learn more about Microsoft Azure WebJobs SDK, please see https://go.microsoft.com/fwlink/?LinkID=320976
	class Program
	{
		// Please set the following connection strings in app.config for this WebJob to run:
		// AzureWebJobsDashboard and AzureWebJobsStorage
		static void Main()
		{
			var config = new JobHostConfiguration();

			if (config.IsDevelopment)
			{
				config.UseDevelopmentSettings();
			}

			config.UseCore();

			var host = new JobHost(config);
			// The following code ensures that the WebJob will be running continuously
			host.RunAndBlock();
		}
	}
}

The important part here is to call the extension named UseCore. This will enable the ErrorTrigger extension.

Now we will create two functions ProcessQueueAMessage and ProcessQueueBMessage using the default ProcessQueueMessage function that uses a QueueTrigger.

Both will call the ProcessMessage method. This method checks if the message is empty. If it is, an exception will be thrown.

ProcessQueueAMessage will simply call ProcessMessage, ProcessQueueBMessage will also call it but catching exceptions and logging them using the TraceWriter.

Then we create a function named GlobalErrorMonitor, giving the following code:

using System;
using System.IO;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions;
using Microsoft.Azure.WebJobs.Host;

namespace AzureWebJobs.ErrorTriggerExtension
{
	public class Functions
	{
		// This function will get triggered/executed when a new message is written 
		// on an Azure Queue called queue.
		public static void ProcessQueueAMessage([QueueTrigger("queuea")] string message, TextWriter log)
		{
			log.WriteLine(message);

			ProcessMessage(message);
		}

		public static void ProcessQueueBMessage([QueueTrigger("queueb")] string message, TraceWriter logger)
		{
			logger.Info(message);

			try
			{
				ProcessMessage(message);
			}
			catch (Exception ex)
			{
				logger.Error($"An error occurred in: '{nameof(ProcessQueueBMessage)}'", ex, nameof(Functions));
			}
		}

		/// <summary>
		/// Triggered when an error is reported in other functions.
		/// Called whenever 2 errors occur within a 3 minutes sliding window (throttled at a maximum of 2 notifications per 10 minutes).
		/// </summary>
		public static void GlobalErrorMonitor([ErrorTrigger("0:03:00", 2, Throttle = "0:10:00")] TraceFilter filter, TextWriter log)
		{
			Console.Error.WriteLine("An error has been detected in a function.");

			log.WriteLine(filter.GetDetailedMessage(1));
		}

		private static void ProcessMessage(string message)
		{
			if (string.IsNullOrEmpty(message))
			{
				throw new ArgumentNullException(nameof(message));
			}

			//Do some work here...
		}
	}
}

You can notice two things here:

We use the ErrorTrigger attribute to be able to use ErrorTrigger extension in the GlobalErrorMonitor function.
The error trigger is configured to be called whenever 2 errors occur within a 3 minutes sliding window (throttled at a maximum of 2 notifications per 10 minutes.

Example of use

Once the WebJob ready and properly configured, we will launch it:

Found the following functions:
AzureWebJobs.ErrorTriggerExtension.Functions.ProcessQueueAMessage
AzureWebJobs.ErrorTriggerExtension.Functions.ProcessQueueBMessage
AzureWebJobs.ErrorTriggerExtension.Functions.GlobalErrorMonitor
Job host started

Now add an empty message to the queuea. The ProcessQueueAMessage function will be called 5 times and the message will be moved to the queuea-poison.

After the second exception you can notice that GlobalErrorMonitor is triggered:

Executing 'Functions.ProcessQueueAMessage' (Reason='New queue message detected on 'queuea'.', Id=8ee22555-15df-458e-8a0e-cdd1b9b2a97a)
Executing 'Functions.GlobalErrorMonitor' (Reason='Error trigger fired', Id=8b368696-7dab-4908-af53-18018abd8f3e)
An error has been detected in a function.
2 events at level 'Error' or lower have occurred within time window 00:03:00.

04/11/2017 23:29:43 Error Exception while executing function: Functions.ProcessQueueAMessage WebJobs.Execution Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.ProcessQueueAMessage ---> System.ArgumentNullException: Value cannot be null.
Parameter name: message
 at AzureWebJobs.ErrorTriggerExtension
 ...

Executed 'Functions.GlobalErrorMonitor' (Succeeded, Id=8b368696-7dab-4908-af53-18018abd8f3e)
Executed 'Functions.ProcessQueueBMessage' (Succeeded, Id=8ee22555-15df-458e-8a0e-cdd1b9b2a97a)

Now restart the WebJob and add an empty message to the queueb. The ProcessQueueBMessage function will be called once as we are catching and logging the ArgumentNullException.

Notice that the GlobalErrorMonitor is not executed as it is configured to be triggered when at least 2 errors occur.

Now add another empty message to the queueb and you will now notice that GlobalErrorMonitor is triggered:

Executing 'Functions.ProcessQueueBMessage' (Reason='New queue message detected on 'queueb'.', Id=9ba82a41-fe9c-4227-a306-9af2dc9db7c2)
Executing 'Functions.GlobalErrorMonitor' (Reason='Error trigger fired', Id=03a3c8ea-16cf-46e9-a6c5-074a4f47c997)
An error has been detected in a function.
2 events at level 'Error' or lower have occurred within time window 00:03:00.

04/11/2017 23:43:05 Error An error occurred in: 'ProcessQueueBMessage' Functions System.ArgumentNullException: Value cannot be null.
Parameter name: message
 at AzureWebJobs.ErrorTriggerExtension
 ...

Executed 'Functions.GlobalErrorMonitor' (Succeeded, Id=03a3c8ea-16cf-46e9-a6c5-074a4f47c997)
Executed 'Functions.ProcessQueueBMessage' (Succeeded, Id=9ba82a41-fe9c-4227-a306-9af2dc9db7c2)

To go further

Here we have setup a global error handler and you could setup more handlers, but you can also have function specific error handlers.

To create a function specific handler use the naming convention based on the "ErrorHandler" suffix.

For example, if we want to have a specific handler for the function ProcessQueueAMessage we will would create the following:

public static void ProcessQueueAMessageErrorHandler([ErrorTrigger("0:03:00", 2, Throttle = "0:10:00")] TraceFilter filter, TextWriter log)

In that case only errors coming from the function ProcessQueueAMessage will trigger the ErrorTrigger extension.

Summary

We have learned how to monitor errors with Azure WebJob using the ErrorTrigger extension.

You can download the example solution here:

Download full sources

Browse the GitHub repository

(Note that the project uses Microsoft.Azure.WebJobs version 2.0.0)

Please feel free to comment or contact me if you have any question about this article.

Add a comment

Language

Name

E-mail (Will not be published)

Website url

Message

Back to articles