The following is from Azure Developer Training lab for AZ-203

Introduction to Azure Functions

Azure Functions is a solution for easily running small pieces of code, or “functions,” in the cloud. You can write just the code you need for the problem at hand, without worrying about a whole application or the infrastructure to run it. Functions can make development even more productive, and you can use your development language of choice, such asC#, F#, Node.js, Java, or PHP. Pay only for the time your code runs and trust Azure to scale as needed. Azure Functions lets you develop serverless applications on Microsoft Azure.

What can I do with Functions?

Functions is a great solution for processing data, integrating systems, working with the internet-of-things (IoT), and building simple APIs and microservices. Consider Functions for tasks like image or order processing, file maintenance, or for any tasks that you want to run on a schedule.

Functions provides templates to get you started with key scenarios, including the following:

• HTTPTrigger – Trigger the execution of your code by using an HTTPrequest.
• TimerTrigger – Execute cleanup or other batch tasks on a predefined schedule.
• GitHub webhook – Respond to events that occur in your GitHub repositories. Generic webhook – Process webhook HTTP requests from any service that supports webhooks.
• CosmosDBTrigger – Process Azure Cosmos DB documents when they are added or updated in collections in a NoSQL database.
• BlobTrigger – Process Azure Storage blobs when they are added to containers. You might use this function for image resizing.
• QueueTrigger – Respond to messages as they arrive in an Azure Storage queue.
• EventHubTrigger – Respond to events delivered to an Azure Event Hub.Particularly useful in application instrumentation, user experience or workflow processing, and Internet of Things (IoT) scenarios.
• ServiceBusQueueTrigger – Connect your code to other Azure services or on-premises services by listening to message queues.
• ServiceBusTopicTrigger – Connect your code to other Azure services or on-premises services by subscribing to topics.

Azure Functions supports triggers, which are ways to start execution of your code, and bindings, which are ways to simplify coding for input and output data.

Integrations

Azure Functions integrates with various Azure and 3rd-party services. These services can trigger your function and start execution, or they can serve as input and output for your code. The following service integrations are supported by Azure Functions:

• Azure Cosmos DB
• Azure Event Hubs
• Azure Event Grid
• Azure Notification Hubs
• Azure Service Bus (queues and topics)
• Azure Storage (blob, queues, and tables)
• On-premises (using Service Bus)
• Twilio (SMS messages)

How much does Functions cost?

Azure Functions has two kinds of pricing plans. Choose the one that best fits your needs:

• Consumption plan – When your function runs, Azure provides all of the necessary computational resources. You don’t have to worry about resource management, and you only pay for the time that your code runs.
• App Service plan – Run your functions just like your web apps. When you are already using App Service for your other applications, you can run your functions on the same plan at no additional cost.

For more information about hosting plans, see Azure Functions hosting plancomparison.

Azure Functions scale and hosting concepts

Azure Functions runs in two different modes: Consumption plan and AzureApp Service plan. The Consumption plan automatically allocates compute power when your code is running. Your app is scaled out when needed to handle load, and scaled down when code is not running. You don’t have to pay for idle VMs or reserve capacity in advance.

When you create a function app, you choose the hosting plan for functions in the app. In either plan, an instance of the Azure Functions host executes the functions. The type of plan controls:

• How host instances are scaled out.
• The resources that are available to each host.

Important: You must choose the type of hosting plan during the creation of the function app. You can’t change it afterward.

On an App Service plan, you can scale between tiers to allocate different amount of resources. On the Consumption plan, Azure Functions automatically handles all resource allocation.

Consumption plan

When you’re using a Consumption plan, instances of the Azure Functions host are dynamically added and removed based on the number of incoming events.This serverless plan scales automatically, and you’re charged for computer resources only when your functions are running. On a Consumption plan, a function execution times out after a configurable period of time.

Note: The default timeout for functions on a Consumption plan is 5 minutes.The value can be increased for the Function App up to a maximum of 10minutes by changing the property functionTimeout in the host.json project file.

Billing is based on number of executions, execution time, and memory used.Billing is aggregated across all functions within a function app.

The Consumption plan is the default hosting plan and offers the following benefits:

• Pay only when your functions are running.
• Scale out automatically, even during periods of high load.

App Service plan

In the dedicated App Service plan, your function apps run on dedicated VMs on Basic, Standard, Premium, and Isolated SKUs, which is the same as otherApp Service apps. Dedicated VMs are allocated to your function app, which means the functions host can be always running. App Service plans supportLinux.

Consider an App Service plan in the following cases:

• You have existing, underutilized VMs that are already running other AppService instances.
• Your function apps run continuously, or nearly continuously. In this case, an App Service Plan can be more cost-effective.
• You need more CPU or memory options than what is provided on theConsumption plan.
• Your code needs to run longer than the maximum execution time allowed on the Consumption plan, which is up to 10 minutes.
• You require features that are only available on an App Service plan, such as support for App Service Environment, VNET/VPN connectivity, and larger VM sizes.
• You want to run your function app on Linux, or you want to provide a custom image on which to run your functions.

A VM decouples cost from number of executions, execution time, and memory used. As a result, you won’t pay more than the cost of the VM instance that you allocate. With an App Service plan, you can manually scale out by adding more VM instances, or you can enable autoscale.

When running JavaScript functions on an App Service plan, you should choose a plan that has fewer vCPUs.

Always On

If you run on an App Service plan, you should enable the Always on setting so that your function app runs correctly. On an App Service plan, the functions runtime goes idle after a few minutes of inactivity, so only HTTP triggers will “wake up” your functions. Always on is available only on anApp Service plan. On a Consumption plan, the platform activates function apps automatically.

Storage account requirements

On either a Consumption plan or an App Service plan, a function app requires a general Azure Storage account, which supports Azure Blob, Queue,Files, and Table storage. This is because Functions relies on Azure Storage for operations such as managing triggers and logging function executions, but some storage accounts do not support queues and tables. These accounts, which include blob-only storage accounts (including premium storage) and general-purpose storage accounts with zone-redundant storage replication, are filtered-out from your existing Storage Account selections when you create a function app.

How the Consumption plan works

In the Consumption plan, the scale controller automatically scales CPU and memory resources by adding additional instances of the Functions host, based on the number of events that its functions are triggered on. Each instance of the Functions host is limited to 1.5 GB of memory. An instance of the host is the function app, meaning all functions within a function app share resource within an instance and scale at the same time. Function apps that share the same Consumption plan are scaled independently.

When you use the Consumption hosting plan, function code files are stored onAzure Files shares on the function’s main storage account. When you delete the main storage account of the function app, the function code files are deleted and cannot be recovered.

Note: When you’re using a blob trigger on a Consumption plan, there can be up to a 10-minute delay in processing new blobs. This delay occurs when a function app has gone idle. After the function app is running, blobs are processed immediately. To avoid this cold-start delay, use an App Serviceplan with Always On enabled, or use the Event Grid trigger.

Runtime scaling

Azure Functions uses a component called the scale controller to monitor the rate of events and determine whether to scale out or scale in. The scale controller uses heuristics for each trigger type. For example, when you’re using an Azure Queue storage trigger, it scales based on the queue length and the age of the oldest queue message.

The unit of scale is the function app. When the function app is scaled out, additional resources are allocated to run multiple instances of the AzureFunctions host. Conversely, as compute demand is reduced, the scale controller removes function host instances. The number of instances is eventually scaled down to zero when no functions are running within a function app.

Understanding scaling behaviors

Scaling can vary on a number of factors, and scale differently based on the trigger and language selected. However there are a few aspects of scaling that exist in the system today:

Different triggers may also have different scaling limits as well.

Azure Functions triggers and bindings concepts

This section is a conceptual overview of triggers and bindings in AzureFunctions. Features that are common to all bindings and all supported languages are described here.

Overview

A trigger defines how a function is invoked. A function must have exactly one trigger. Triggers have associated data, which is usually the payload that triggered the function.

Input and output bindings provide a declarative way to connect to data from within your code. Bindings are optional and a function can have multiple input and output bindings.

Triggers and bindings let you avoid hardcoding the details of the services that you’re working with. Your function receives data (for example, the content of a queue message) in function parameters. You send data (for example, to create a queue message) by using the return value of the function. In C# andC# script, alternative ways to send data are out parameters and collect or objects.

When you develop functions by using the Azure portal, triggers and bindings are configured in a function.json file. The portal provides a UI for this configuration but you can edit the file directly by changing to the Advancededitor.

When you develop functions by using Visual Studio to create a class library, you configure triggers and bindings by decorating methods and parameters with attributes.

Example trigger and binding

Suppose you want to write a new row to Azure Table storage whenever a new message appears in Azure Queue storage. This scenario can be implemented using an Azure Queue storage trigger and an Azure Table storage output binding.

Here’s a function.json file for this scenario.

{ "bindings": [ { "name": "order", "type": "queueTrigger", "direction": "in", "queueName": "myqueue-items", "connection": "MY_STORAGE_ACCT_APP_SETTING" }, { "name": "$return", "type": "table", "direction": "out", "tableName": "outTable", "connection": "MY_TABLE_STORAGE_ACCT_APP_SETTING" } ] }

The first element in the bindings array is the Queue storage trigger. The type and direction properties identify the trigger. The name property identifies the function parameter that receives the queue message content. The name of the queue to monitor is in queueName, and the connection string is in the app setting identified by connection.

The second element in the bindings array is the Azure Table Storage output binding. The type and direction properties identify the binding.The name property specifies how the function provides the new table row, in this case by using the function return value. The name of the table is in tableName, and the connection string is in the app setting identified by connection.

To view and edit the contents of function.json in the Azure portal, click theAdvanced editor option on the Integrate tab of your function.

Note: The value of connection is the name of an app setting that contains the connection string, not the connection string itself. Bindings use connection strings stored in app settings to enforce the best practice that function.json does not contain service secrets.

Here’s C# script code that works with this trigger and binding. Notice that thename of the parameter that provides the queue message content is order; this name is required because the name property value in function.json is order.

#r "Newtonsoft.Json" using Microsoft.Extensions.Logging; using Newtonsoft.Json.Linq; // From an incoming queue message that is a JSON object, add fields and write to Table storage // The method return value creates a new row in Table Storage public static Person Run(JObject order, ILogger log) { return new Person() { PartitionKey = "Orders", RowKey = Guid.NewGuid().ToString(), Name = order["Name"].ToString(), MobileNumber = order["MobileNumber"].ToString() }; } public class Person { public string PartitionKey { get; set; } public string RowKey { get; set; } public string Name { get; set; } public string MobileNumber { get; set; } }

Register binding extensions

In some development environments, you have to explicitly register a binding that you want to use. Binding extensions are provided in NuGet packages, and to register an extension you install a package. The following table indicates when and how you register binding extensions.

The following binding types are exceptions that don’t require explicit registration because they are automatically registered in all versions and environments: HTTP and timer.

Binding direction

All triggers and bindings have a direction property in the function.json file:

• For triggers, the direction is always in
• Input and output bindings use in and out
• Some bindings support a special direction inout. If you use inout, only the Advanced editor is available in the Integrate tab.

When you use attributes in a class library to configure triggers and bindings, the direction is provided in an attribute constructor or inferred from the parameter type.

Using the function return value

In languages that have a return value, you can bind an output binding to the return value:

• In a C# class library, apply the output binding attribute to the method return value.
• In other languages, set the name property in function.json to$return.

If there are multiple output bindings, use the return value for only one of them.

In C# and C# script, alternative ways to send data to an output binding are out parameters and collector objects.

C# example

Here’s C# code that uses the return value for an output binding:

[FunctionName("QueueTrigger")] [return: Blob("output-container/{id}")] public static string Run([QueueTrigger("inputqueue")]WorkItem input, ILogger log) { string json = string.Format("{{ \"id\": \"{0}\" }}", input.Id); log.LogInformation($"C# script processed queue message. Item={json}"); return json; }

Binding dataType property

In .NET, use the parameter type to define the data type for input data. For instance, use string to bind to the text of a queue trigger, a byte array toread as binary and a custom type to deserialize to a POCO object.

For languages that are dynamically typed such as JavaScript, use the dataType property in the function.json file. For example, to read the content of an HTTP request in binary format, set dataType to binary:

{ "type": "httpTrigger", "name": "req", "direction": "in", "dataType": "binary" }

Other options for dataType are stream and string.

Binding expressions and patterns

One of the most powerful features of triggers and bindings is binding expressions. In the function.json file and in function parameters and code, you can use expressions that resolve to values from various sources.

Most expressions are identified by wrapping them in curly braces. For example, in a queue trigger function, {queueTrigger} resolves to the queue message text. If the path property for a blob output binding is container/{queueTrigger} and the function is triggered by a queue message HelloWorld, a blob named HelloWorld is created.

Types of binding expressions

• App settings
• Trigger file name
• Trigger metadata
• JSON payloads
• New GUID
• Current date and time

Binding expressions – app settings

As a best practice, secrets and connection strings should be managed using app settings, rather than configuration files. This limits access to these secrets and makes it safe to store files such as function.json in public source control repositories.

App settings are also useful whenever you want to change configuration based on the environment. For example, in a test environment, you may want to monitor a different queue or blob storage container.

App setting binding expressions are identified differently from other binding expressions: they are wrapped in percent signs rather than curly braces. For example if the blob output binding path is%Environment%/newblob.txt and the Environment app setting value is Development, a blob will be created in the Development container.

When a function is running locally, app setting values come from the local.settings.json file.

Note that the connection property of triggers and bindings is a special case and automatically resolves values as app settings, without percent signs.

Binding expressions – trigger file name

The path for a Blob trigger can be a pattern that lets you refer to the name of the triggering blob in other bindings and function code. The pattern canal so include filtering criteria that specify which blobs can trigger a function invocation.

Binding expressions – trigger metadata

In addition to the data payload provided by a trigger (such as the content of the queue message that triggered a function), many triggers provide additional metadata values. These values can be used as input parameters inC# and F# or properties on the context.bindings object in JavaScript.

For example, an Azure Queue storage trigger supports the following properties:

• QueueTrigger – triggering message content if a valid string
• DequeueCount
• ExpirationTime
• Id
• InsertionTime
• NextVisibleTime
• PopReceipt

These metadata values are accessible in function.json file properties. For example, suppose you use a queue trigger and the queue message contains the name of a blob you want to read. In the function.json file, you can use queueTrigger metadata property in the blob path property.

Binding expressions – JSON payloads

When a trigger payload is JSON, you can refer to its properties in configuration for other bindings in the same function and in function code.

The following example shows the function.json file for a webhook function that receives a blob name in JSON:{“BlobName”:”HelloWorld.txt”}. A Blob input binding reads the blob, and the HTTP output binding returns the blob contents in the HTTPresponse. Notice that the Blob input binding gets the blob name by   to the BlobName property”path”:”strings/{BlobName}”:

{ "bindings": [ { "name": "info", "type": "httpTrigger", "direction": "in", "webHookType": "genericJson" }, { "name": "blobContents", "type": "blob", "direction": "in", "path": "strings/{BlobName}", "connection": "AzureWebJobsStorage" }, { "name": "res", "type": "http", "direction": "out" } ] }

Binding expressions – create GUIDs

The {rand-guid} binding expression creates a GUID. The following blobpath in a function.json file creates a blob with a name like 50710cb5-84b9-4d87-9d83-a03d6976a682.txt.

{ "type": "blob", "name": "blobOutput", "direction": "out", "path": "my-output-container/{rand-guid}" }

Binding expressions – current time

The binding expression DateTime resolves to DateTime.UtcNow. Thefollowing blob path in a function.json file creates a blob with a name like2018-02-16T17-59-55Z.txt.

{ "type": "blob", "name": "blobOutput", "direction": "out", "path": "my-output-container/{DateTime}" }

Optimize the performance and reliability of AzureFunctions

This section provides guidance to improve the performance and reliability of your serverless function apps.

General best practices

The following are best practices in how you build and architect your serverless solutions using Azure Functions.

Avoid long running functions

Large, long-running functions can cause unexpected timeout issues. A function can become large due to many Node.js dependencies. Importing dependencies can also cause increased load times that result in unexpected timeouts. Dependencies are loaded both explicitly and implicitly. A single module loaded by your code may load its own additional modules.

Whenever possible, refactor large functions into smaller function sets that work together and return responses fast. For example, a webhook or HTTPtrigger function might require an acknowledgment response within a certain time limit; it is common for webhooks to require an immediate response. You can pass the HTTP trigger payload into a queue to be processed by a queue trigger function. This approach allows you to defer the actual work and return an immediate response.

Cross function communication

Durable Functions and Azure Logic Apps are built to manage state transitions and communication between multiple functions.

If not using Durable Functions or Logic Apps to integrate with multiple functions, it is generally a best practice to use storage queues for cross function communication. The main reason is storage queues are cheaper and much easier to provision.

Individual messages in a storage queue are limited in size to 64 KB. If you need to pass larger messages between functions, an Azure Service Bus queue could be used to support message sizes up to 256 KB in the Standard tier, and up to 1 MB in the Premium tier.

Service Bus topics are useful if you require message filtering before processing.

Event hubs are useful to support high volume communications.

Write functions to be stateless

Functions should be stateless and idempotent if possible. Associate any required state information with your data. For example, an order being processed would likely have an associated state member. A function could process an order based on that state while the function itself remains stateless.

Idempotent functions are especially recommended with timer triggers. For example, if you have something that absolutely must run once a day, write it so it can run any time during the day with the same results. The function can exit when there is no work for a particular day. Also if a previous run failed to complete, the next run should pick up where it left off.

Write defensive functions

Assume your function could encounter an exception at any time. Design your functions with the ability to continue from a previous fail point during the next execution. Consider a scenario that requires the following actions:

1. Query for 10,000 rows in a db.
2. Create a queue message for each of those rows to process further down the line.

Depending on how complex your system is, you may have: involved downstream services behaving badly, networking outages, or quota limits reached, etc. All of these can affect your function at any time. You need to design your functions to be prepared for it.

How does your code react if a failure occurs after inserting 5,000 of those items into a queue for processing? Track items in a set that you’ve completed.Otherwise, you might insert them again next time. This can have a serious impact on your work flow.

If a queue item was already processed, allow your function to be a no-op.

Scalability best practices

There are a number of factors which impact how instances of your function app scale. The details were covered easlier in this lesson. The following are some best practices to ensure optimal scalability of a function app.

Share and manage connections

Re-use connections to external resources whenever possible. See how to manage connections in Azure Functions.

Don’t mix test and production code in the same function app

Functions within a function app share resources. For example, memory is shared. If you’re using a function app in production, don’t add test-related functions and resources to it. It can cause unexpected overhead during production code execution.

Be careful what you load in your production function apps. Memory is averaged across each function in the app.

If you have a shared assembly referenced in multiple .Net functions, put it in a common shared folder. Reference the assembly with a statement similar to the following example if using C# Scripts (.csx):

Otherwise, it is easy to accidentally deploy multiple test versions of the same binary that behave differently between functions.

Don’t use verbose logging in production code. It has a negative performance impact.

Use async code but avoid blocking calls

Asynchronous programming is a recommended best practice. However, always avoid referencing the Result property or calling Wait method on a Task instance. This approach can lead to thread exhaustion.

Tip: If you plan to use the HTTP or WebHook bindings, plan to avoid port exhaustion that can be caused by improper instantiation of HttpClient.

Receive messages in batch whenever possible

Some triggers like Event Hub enable receiving a batch of messages on a single invocation. Batching messages has much better performance. You can configure the max batch size in the host.json file as detailed in the host.jsonreference documentation.

For C# functions you can change the type to a strongly-typed array. For example, instead of EventData sensorEvent the method signature could beEventData[] sensorEvent. For other languages you’ll need to explicitly set the cardinality property in your function.json to many in order to enable batching as shown here.

Configure host behaviors to better handle concurrency

The host.json file in the function app allows for configuration of host runtime and trigger behaviors. In addition to batching behaviors, you can manage concurrency for a number of triggers. Often adjusting the values in these options can help each instance scale appropriately for the demands of the invoked functions.

Settings in the hosts file apply across all functions within the app, within a single instance of the function. For example, if you had a function app with 2HTTP functions and concurrent requests set to 25, a request to either HTTPtrigger would count towards the shared 25 concurrent requests. If that function app scaled to 10 instances, the 2 functions would effectively allow250 concurrent requests (10 instances * 25 concurrent requests per instance).

Developing on Visual Studios

Getting started

Azure Functions Tools for Visual Studio 2017 is an extension for Visual Studio that lets you develop, test, and deploy C# functions to Azure.

The Azure Functions Tools provides the following benefits:

• Edit, build, and run functions on your local development computer.
• Publish your Azure Functions project directly to Azure.
• Use WebJobs attributes to declare function bindings directly in the C#code instead of maintaining a separate function.json for binding definitions.
• Develop and deploy pre-compiled C# functions. Pre-complied functions provide a better cold-start performance than C# script-based functions.
• Code your functions in C# while having all of the benefits of VisualStudio development.

Important: Don’t mix local development with portal development in the same function app. When you publish from a local project to a function app, the deployment process overwrites any functions that you developed in the portal.

Prerequisites

Azure Functions Tools is included in the Azure development workload ofVisual Studio 2017 version 15.5, or a later version. Make sure you include theAzure development workload in your Visual Studio 2017 installation. Also make sure that your Visual Studio is up-to-date and that you are using the most recent version of the Azure Functions tools.

You will also need:

• An active Azure subscription
• An Azure Storage account.

Creating an Azure Functions project

The Azure Functions project template in Visual Studio creates a project that can be published to a function app in Azure. A function app lets you group functions as a logical unit for management, deployment, and sharing of resources.

1. In Visual Studio, select New > Project from the File menu.
2. In the New Project dialog, select Installed, expand Visual C# > Cloud, select Azure Functions, type a Name for your project, and click OK.The function app name must be valid as a C# namespace, so don’t use underscores, hyphens, or any other non alphanumeric characters.
3. Use the settings specified in the table below.
   

   Setting	SuggestedValue	Description
   Version	AzureFunctionsv1	This creates a function project that uses the version 1 runtime of Azure Functions. The version 2 runtime, which supports .NETCore, is currently in preview.
   Template	HTTPtrigger	This creates a function triggered by anHTTP request.
   Storageaccount	Storageemulator	An HTTP trigger doesn’t use the Storageaccount connection. All other trigger types require a valid Storage account connectionstring.
   Accessrights	Anonymous	The created function can be triggered by any client without providing a key. This authorization setting makes it easy to test your new function.

4. Click OK to create the function project and HTTP triggered function.

The project template creates a C# project, installs the Microsoft.NET.Sdk.Functions NuGet package, and sets the target framework. Functions 1.x targets the .NET Framework, and Functions 2.xtargets .NET Standard. The new project has the following files:

• host.json: Lets you configure the Functions host. These settings apply both when running locally and in Azure.
• local.settings.json: Maintains settings used when running functions locally. These settings are not used by Azure, they are used by the AzureFunctions Core Tools. Use this file to specify app settings for variables required by your functions. Add a new item to the Values array for each connection required by the functions bindings in your project.

Configure the project for local development

The Functions runtime uses an Azure Storage account internally. For all trigger types other than HTTP and webhooks, you must set theValues.AzureWebJobsStorage key to a valid Azure Storage account connection string. Your function app can also use the Azure storage emulator for the AzureWebJobsStorage connection setting that is required by the project. To use the emulator, set the value of AzureWebJobsStorage toUseDevelopmentStorage=true. You must change this setting to an actual storage connection before deployment.

To set the storage account connection string:

• In Visual Studio, open Cloud Explorer, expand Storage Account > YourStorage Account, then select Properties and copy the PrimaryConnection String value.
• In your project, open the local.settings.json file and set the value of theAzureWebJobsStorage key to the connection string you copied.
• Repeat the previous step to add unique keys to the Values array for any other connections required by your functions.

Creating a function

In pre-compiled functions, the bindings used by the function are defined by applying attributes in the code. When you use the Azure Functions Tools to create your functions from the provided templates, these attributes are applied for you.

1. In Solution Explorer, right-click on your project node and select Add >New Item. Select Azure Function, type a Name for the class, and click Add.
2. Choose your trigger, set the binding properties, and click Create. The following example shows the settings when creating a Queue storagetriggered function.
3. 
4. This trigger example uses a connection string with a key named QueueStorage. This connection string setting must be defined in the local.settings.json file.
5. Examine the newly added class. You see a static Run method, that is attributed with the FunctionName attribute. This attribute indicates that the method is the entry point for the function. For example, the followingC# class represents a basic Queue storage triggered function:

   using System; using Microsoft.Azure.WebJobs; using Microsoft.Azure.WebJobs.Host; namespace FunctionApp1 { public static class Function1 { [FunctionName("QueueTriggerCSharp")] public static void Run([QueueTrigger("myqueue-items", Connection = "QueueStorage")]string myQueueItem, TraceWriter log) { log.Info($"C# Queue trigger function processed: {myQueueItem}"); } } }

   A binding-specific attribute is applied to each binding parameter supplied to the entry point method. The attribute takes the binding information as parameters. In the previous example, the first parameter has a QueueTrigger attribute applied, indicating queue triggered function. The queue name and connection string setting name are passed as parameters to the QueueTrigger attribute.

You can use the above procedure to add more functions to your function app project. Each function in the project can have a different trigger, but a function must have exactly one trigger.

Add bindings to the Azure Function

As with triggers, input and output bindings are added to your function asbinding attributes. Add bindings to a function as follows:

1. Make sure you have configured the project for local development.
2. Add the appropriate NuGet extension package for the specific binding.The binding-specific NuGet package requirements are found in thereference article for the binding. For example, find packagerequirements for the Event Hubs trigger in the Event Hubs bindingreference article.
3. If there are app settings that the binding needs, add them to the Valuescollection in the local setting file. These values are used when thefunction runs locally. When the function runs in the function app inAzure, the function app settings are used.
4. Add the appropriate binding attribute to the method signature. In thefollowing example, a queue message triggers the function, and theoutput binding creates a new queue message with the same text in adifferent queue.

   public static class SimpleExampleWithOutput { [FunctionName(“CopyQueueMessage”)] public static void Run( [QueueTrigger(“myqueue-items-source”, Connection = “AzureWebJobsStorage”)] string myQueueItem, [Queue(“myqueue-items-destination”, Connection = “AzureWebJobsStorage”)] out string myQueueItemCopy, TraceWriter log) { log.Info($”CopyQueueMessage function processed: {myQueueItem}”); myQueueItemCopy = myQueueItem; } }

   The connection to Queue storage is obtained from theAzureWebJobsStorage setting. For more information, see the referencearticle for the specific binding.

Testing and publishing Azure Functions

Azure Functions Core Tools lets you run Azure Functions project on yourlocal development computer. You are prompted to install these tools the firsttime you start a function from Visual Studio.

To test your function, press F5. If prompted, accept the request from VisualStudio to download and install Azure Functions Core (CLI) tools. You mayalso need to enable a firewall exception so that the tools can handle HTTPrequests.

With the project running, you can test your code as you would test deployedfunction. For more information, see Strategies for testing your code in AzureFunctions. When running in debug mode, breakpoints are hit in Visual Studioas expected.

Publish to Azure

1. In Solution Explorer, right-click the project and select Publish.
2. Select Azure Function App, choose Create New, and then selectPublish.
3. If you haven’t already connected Visual Studio to your Azure account,select Add an account….
4. In the Create App Service dialog, use the Hosting settings as specified inthe table below:

5. Click Create to create a function app and related resources in Azure withthese settings and deploy your function project code.
6. After the deployment is complete, make a note of the Site URL value,which is the address of your function app in Azure.

Function app settings

Any settings you added in the local.settings.json must be also added to thefunction app in Azure. These settings are not uploaded automatically whenyou publish the project.

The easiest way to upload the required settings to your function app in Azureis to use the Manage Application Settings… link that is displayed after yousuccessfully publish your project.

This displays the Application Settings dialog for the function app, where youcan add new application settings or modify existing ones.

You can also manage application settings in one of these other ways:

• Using the Azure portal.
• Using the –publish-local-settings publish option in theAzure Functions Core Tools.
• Using the Azure CLI.

Implement Durable Functions

Durable Functions overview

Durable Functions is an extension of Azure Functions and Azure WebJobs that lets you write stateful functions in a serverless environment. The extension manages state, checkpoints, and restarts for you.

The extension lets you define stateful workflows in a new type of function called an orchestrator function. Here are some of the advantages of orchestrator functions:

• They define workflows in code. No JSON schemas or designers are needed.
• They can call other functions synchronously and asynchronously. Output from called functions can be saved to local variables.
• They automatically checkpoint their progress whenever the function awaits. Local state is never lost if the process recycles or the VM reboots.

Note: Durable Functions is an advanced extension for Azure Functions that is not appropriate for all applications. The rest of this section assumes that you have a strong familiarity with Azure Functions concepts and the challenges involved in serverless application development.

The primary use case for Durable Functions is simplifying complex, stateful coordination problems in serverless applications. The following sections describe some typical application patterns that can benefit from DurableFunctions.

Pattern #1: Function chaining

Function chaining refers to the pattern of executing a sequence of functions ina particular order. Often the output of one function needs to be applied to the input of another function.

Durable Functions allows you to implement this pattern concisely in code.

C# script

public static async Task<object> Run(DurableOrchestrationContext ctx) { try { var x = await ctx.CallActivityAsync<object>("F1"); var y = await ctx.CallActivityAsync<object>("F2", x); var z = await ctx.CallActivityAsync<object>("F3", y); return await ctx.CallActivityAsync<object>("F4", z); } catch (Exception) { // error handling/compensation goes here } }

Note: There are subtle differences while writing a precompiled durable function in C# vs the C# script sample shown before. A C# precompiled function would require durable parameters to be decorated with respective attributes. An example is [OrchestrationTrigger] attribute forDurableOrchestrationContext parameter. If the parameters are not properly decorated, the runtime would not be able to inject the variables to the function and would give error.

The values “F1”, “F2”, “F3”, and “F4” are the names of other functions in the function app. Control flow is implemented using normal imperative coding constructs. That is, code executes top-down and can involve existing language control flow semantics, like conditionals, and loops. Error handling logic can be included in try/catch/finally blocks.

The ctx parameter (DurableOrchestrationContext) provides methods for invoking other functions by name, passing parameters, and returning function output. Each time the code calls await, the DurableFunctions framework checkpoints the progress of the current function instance. If the process or VM recycles midway through the execution, the function instance resumes from the previous await call. More on this restart behavior later.

Pattern #2: Fan-out/fan-in

Fan-out/fan-in refers to the pattern of executing multiple functions in parallel, and then waiting for all to finish. Often some aggregation work is done on results returned from the functions.

With normal functions, fanning out can be done by having the function send multiple messages to a queue. However, fanning back in is much more challenging. You’d have to write code to track when the queue-triggered functions end and store function outputs. The Durable Functions extension handles this pattern with relatively simple code.

C# script

public static async Task Run(DurableOrchestrationContext ctx) { var parallelTasks = new List<Task<int>>(); // get a list of N work items to process in parallel object[] workBatch = await ctx.CallActivityAsync<object[]>("F1"); for (int i = 0; i < workBatch.Length; i++) { Task<int> task = ctx.CallActivityAsync<int>("F2", workBatch[i]); parallelTasks.Add(task); } await Task.WhenAll(parallelTasks); // aggregate all N outputs and send result to F3 int sum = parallelTasks.Sum(t => t.Result); await ctx.CallActivityAsync("F3", sum); }

The fan-out work is distributed to multiple instances of function F2, and thework is tracked by using a dynamic list of tasks. The .NET Task.WhenAllAPI is called to wait for all of the called functions to finish. Then the F2function outputs are aggregated from the dynamic task list and passed on to the F3 function.

The automatic checkpointing that happens at the await call onTask.WhenAll ensures that any crash or reboot midway through does not require a restart of any already completed tasks.

Pattern #3: Async HTTP APIs

The third pattern is all about the problem of coordinating the state of long-running operations with external clients. A common way to implement this pattern is by having the long-running action triggered by an HTTP call, and then redirecting the client to a status endpoint that they can poll to learn when the operation completes.

Durable Functions provides built-in APIs that simplify the code you write for interacting with long-running function executions. Once an instance is started, the extension exposes webhook HTTP APIs that query the orchestrator function status. The following example shows the REST commands to start an orchestrator and to query its status. For clarity, some details are omitted from the example.

> curl -X POST https://myfunc.azurewebsites.net/orchestrators/DoWork -H "Content-Length: 0" -i HTTP/1.1 202 Accepted Content-Type: application/json Location: https://myfunc.azurewebsites.net/admin/extensions/DurableTaskExtension/b79baf67f717453ca9e86c5da21e03ec {"id":"b79baf67f717453ca9e86c5da21e03ec", ...} > curl https://myfunc.azurewebsites.net/admin/extensions/DurableTaskExtension/b79baf67f717453ca9e86c5da21e03ec -i HTTP/1.1 202 Accepted Content-Type: application/json Location: https://myfunc.azurewebsites.net/admin/extensions/DurableTaskExtension/b79baf67f717453ca9e86c5da21e03ec {"runtimeStatus":"Running","lastUpdatedTime":"2017-03-16T21:20:47Z", ...} > curl https://myfunc.azurewebsites.net/admin/extensions/DurableTaskExtension/b79baf67f717453ca9e86c5da21e03ec -i HTTP/1.1 200 OK Content-Length: 175 Content-Type: application/json {"runtimeStatus":"Completed","lastUpdatedTime":"2017-03-16T21:20:57Z", ...}

Because the state is managed by the Durable Functions runtime, you don’thave to implement your own status tracking mechanism.

Even though the Durable Functions extension has built-in webhooks for managing long-running orchestrations, you can implement this pattern yourself using your own function triggers (such as HTTP, queue, or EventHub) and the orchestrationClient binding. For example, you could use a queue message to trigger termination. Or you could use an HTTP trigger protected by an Azure Active Directory authentication policy instead of the built-in webhooks that use a generated key for authentication.

// HTTP-triggered function to start a new orchestrator function instance. public static async Task<HttpResponseMessage> Run( HttpRequestMessage req, DurableOrchestrationClient starter, string functionName, ILogger log) { // Function name comes from the request URL. // Function input comes from the request content. dynamic eventData = await req.Content.ReadAsAsync<object>(); string instanceId = await starter.StartNewAsync(functionName, eventData); log.LogInformation($"Started orchestration with ID = '{instanceId}'."); return starter.CreateCheckStatusResponse(req, instanceId); }

The DurableOrchestrationClientstarter parameter is a value from the orchestrationClient output binding, which is part of theDurable Functions extension. It provides methods for starting, sending events to, terminating, and querying for new or existing orchestrator function instances. In the previous example, an HTTP triggered-function takes in a functionName value from the incoming URL and passes that value toStartNewAsync. This binding API then returns a response that contains aLocation header and additional information about the instance that can later be used to look up the status of the started instance or terminate it.

Pattern #4: Monitoring

The monitor pattern refers to a flexible recurring process in a workflow – for example, polling until certain conditions are met. A regular timer-trigger can address a simple scenario, such as a periodic cleanup job, but its interval is static and managing instance lifetimes becomes complex. Durable Functions enables flexible recurrence intervals, task lifetime management, and the ability to create multiple monitor processes from a single orchestration.

An example would be reversing the earlier async HTTP API scenario. Instead of exposing an endpoint for an external client to monitor a long-running operation, the long-running monitor consumes an external endpoint, waiting for some state change.

Using Durable Functions, multiple monitors that observe arbitrary endpoint scan be created in a few lines of code. The monitors can end execution when some condition is met, or be terminated by theDurableOrchestrationClient, and their wait interval can be changed based on some condition (i.e. exponential backoff.) The following code implements a basic monitor.

C# script

public static async Task Run(DurableOrchestrationContext ctx) { int jobId = ctx.GetInput<int>(); int pollingInterval = GetPollingInterval(); DateTime expiryTime = GetExpiryTime(); while (ctx.CurrentUtcDateTime < expiryTime) { var jobStatus = await ctx.CallActivityAsync<string>("GetJobStatus", jobId); if (jobStatus == "Completed") { // Perform action when condition met await ctx.CallActivityAsync("SendAlert", machineId); break; } // Orchestration will sleep until this time var nextCheck = ctx.CurrentUtcDateTime.AddSeconds(pollingInterval); await ctx.CreateTimer(nextCheck, CancellationToken.None); } // Perform further work here, or let the orchestration end }

When a request is received, a new orchestration instance is created for that job ID. The instance polls a status until a condition is met and the loop is exited. A durable timer is used to control the polling interval. Further work can then be performed, or the orchestration can end. When the ctx. CurrentUtcDateTime exceeds the expiryTime, the monitor ends.

Pattern #5: Human interaction

Many processes involve some kind of human interaction. The tricky thing about involving humans in an automated process is that people are not always as highly available and responsive as cloud services. Automated processes must allow for this, and they often do so by using timeouts and compensation logic.

One example of a business process that involves human interaction is an approval process. For example, approval from a manager might be required for an expense report that exceeds a certain amount. If the manager does not approve within 72 hours (maybe they went on vacation), an escalation process kicks in to get the approval from someone else (perhaps the manager’s manager).

This pattern can be implemented using an orchestrator function. The orchestrator would use a durable timer to request approval and escalate incase of timeout. It would wait for an external event, which would be the notification generated by some human interaction.

C# script

public static async Task Run(DurableOrchestrationContext ctx) { await ctx.CallActivityAsync("RequestApproval"); using (var timeoutCts = new CancellationTokenSource()) { DateTime dueTime = ctx.CurrentUtcDateTime.AddHours(72); Task durableTimeout = ctx.CreateTimer(dueTime, timeoutCts.Token); Task<bool> approvalEvent = ctx.WaitForExternalEvent<bool>("ApprovalEvent"); if (approvalEvent == await Task.WhenAny(approvalEvent, durableTimeout)) { timeoutCts.Cancel(); await ctx.CallActivityAsync("ProcessApproval", approvalEvent.Result); } else { await ctx.CallActivityAsync("Escalate"); } } }

The durable timer is created by calling ctx.CreateTimer. Thenotification is received by ctx.WaitForExternalEvent. AndTask.WhenAny is called to decide whether to escalate (timeout happens first) or process approval (approval is received before timeout).

An external client can deliver the event notification to a waiting orchestrator function using either the built-in HTTP APIs or by usingDurableOrchestrationClient.RaiseEventAsync API from another function:

public static async Task Run(string instanceId, DurableOrchestrationClient client) { bool isApproved = true; await client.RaiseEventAsync(instanceId, "ApprovalEvent", isApproved); }

The technology

Behind the scenes, the Durable Functions extension is built on top of theDurable Task Framework, an open-source library on GitHub for building durable task orchestrations. Much like how Azure Functions is the serverless evolution of Azure WebJobs, Durable Functions is the serverless evolution ofthe Durable Task Framework. The Durable Task Framework is used heavily within Microsoft and outside as well to automate mission-critical processes.It’s a natural fit for the serverless Azure Functions environment.

Event sourcing, checkpointing, and replay

Orchestrator functions reliably maintain their execution state using a designpattern known as Event Sourcing. Instead of directly storing the current state of an orchestration, the durable extension uses an append-only store to recordthe full series of actions taken by the function orchestration. This has many benefits, including improving performance, scalability, and responsivenesscompared to “dumping” the full runtime state. Other benefits include providing eventual consistency for transactional data and maintaining fullaudit trails and history. The audit trails themselves enable reliable compensating actions.

The use of Event Sourcing by this extension is transparent. Under the covers, the await operator in an orchestrator function yields control of the orchestrator thread back to the Durable Task Framework dispatcher. The dispatcher then commits any new actions that the orchestrator function scheduled (such as calling one or more child functions or scheduling a durable timer) to storage. This transparent commit action appends to the execution history of the orchestration instance. The history is stored in a storage table. The commit action then adds messages to a queue to schedule the actual work. At this point, the orchestrator function can be unloaded from memory. Billing for it stops if you’re using the Azure Functions ConsumptionPlan. When there is more work to do, the function is restarted and its state is reconstructed.

Once an orchestration function is given more work to do (for example, a response message is received or a durable timer expires), the orchestrator wakes up again and re-executes the entire function from the start in order to rebuild the local state. If during this replay the code tries to call a function (or do any other async work), the Durable Task Framework consults with the execution history of the current orchestration. If it finds that the activity function has already executed and yielded some result, it replays that function’s result, and the orchestrator code continues running. This continues happening until the function code gets to a point where either it is finished or it has scheduled new async work.

Orchestrator code constraints

The replay behavior creates constraints on the type of code that can bewritten in an orchestrator function. For example, orchestrator code must bedeterministic, as it will be replayed multiple times and must produce the sameresult each time.

Language support

Currently C# (Functions v1 and v2), F# and JavaScript (Functions v2 only)are the only supported languages for Durable Functions. This includesorchestrator functions and activity functions. In the future, we will addsupport for all languages that Azure Functions supports. See the AzureFunctions GitHub repository issues list to see the latest status of ouradditional language support work.

Storage and scalability

The Durable Functions extension uses Azure Storage queues, tables, andblobs to persist execution history state and trigger function execution. Thedefault storage account for the function app can be used, or you canconfigure a separate storage account. You might want a separate account dueto storage throughput limits. The orchestrator code you write does not need to (and should not) interact with the entities in these storage accounts. Theentities are managed directly by the Durable Task Framework as animplementation detail.

Orchestrator functions schedule activity functions and receive their responsesvia internal queue messages. When a function app runs in the AzureFunctions Consumption plan, these queues are monitored by the AzureFunctions Scale Controller and new compute instances are added as needed.When scaled out to multiple VMs, an orchestrator function may run on oneVM while activity functions it calls run on several different VMs. You can findmore details on the scale behavior of Durable Functions in Performance andscale.

Table storage is used to store the execution history for orchestrator accounts.Whenever an instance rehydrates on a particular VM, it fetches its executionhistory from table storage so that it can rebuild its local state. One of theconvenient things about having the history available in Table storage is thatyou can take a look and see the history of your orchestrations using toolssuch as Microsoft Azure Storage Explorer.

Storage blobs are used primarily as a leasing mechanism to coordinate thescale-out of orchestration instances across multiple VMs. They are also usedto hold data for large messages which cannot be stored directly in tables orqueues.

Create a Durable Function in C#

In this tutorial, you learn how to use the Visual Studio 2017 tools for AzureFunctions to locally create and test a “hello world” durable function. Thisfunction will orchestrate and chain together calls to other functions. You thenpublish the function code to Azure. These tools are available as part of theAzure development workload in Visual Studio 2017.

Prerequisites

To complete this tutorial:

• Install Visual Studio 2017 and ensure that the Azure developmentworkload is also installed.
• Make sure you have the latest Azure Functions tools.
• Verify you have the Azure Storage Emulator installed and running.

If you don’t have an Azure subscription, create a free account before youbegin.

Create a function app project

The Azure Functions project template in Visual Studio creates a project thatcan be published to a function app in Azure. A function app lets you groupfunctions as a logical unit for management, deployment, and sharing ofresources.

1. In Visual Studio, select New > Project from the File menu.
2. In the New Project dialog, select Installed, expand Visual C# > Cloud,select Azure Functions, type a Name for your project, and click OK.The function app name must be valid as a C# namespace, so don’t useunderscores, hyphens, or any other nonalphanumeric characters.
3. Use the settings specified in the table that follows the image.
   

   Setting	Suggested value	Description
   Version	Azure Functions 2.x (.NET Core)	Creates a functionproject that uses theversion 2.x runtimeof Azure Functionswhich supports .NETCore. AzureFunctions 1.xsupports the .NETFramework.
   Template	Empty	This creates anempty function app.
   Storage account	Storage Emulator	A storage account isrequired for durablefunction statemanagement.

4. Click OK to create an empty function project.

Add functions to the app

Visual Studio creates an empty function app project. It contains the basicconfiguration files needed for an app, but doesn’t yet contain any functions.We’ll need to add a durable function template to the project.

1. Right-click the project in Visual Studio and select Add > New AzureFunction.
2. Verify Azure Function is selected from the add menu, and give your C#file a name. Press Add.
3. Select the Durable Functions Orchestration template and click Ok.

A new durable function will be added to the app. Open the new file to viewthe contents. This durable function is a simple function chaining example.

• The RunOrchestrator method is associated with the orchestratorfunction. This function will start, create a list, and add the result of threefunctions calls to the list. When the three function calls are completed, itwill return the list. The function it is calling is the SayHello method (default it will be called “_Hello”).
• The SayHello function will return a hello.
• The HttpStart method describes the function that will start instancesof the orchestration. It is associated with an HTTP trigger that will starta new instance of the orchestrator and return back a check statusresponse.

Now that you’ve created your function project and a durable function, youcan test it on your local computer.

Test the function locally

Azure Functions Core Tools lets you run an Azure Functions project on yourlocal development computer. You are prompted to install these tools the firsttime you start a function from Visual Studio.

1. To test your function, press F5. If prompted, accept the request fromVisual Studio to download and install Azure Functions Core (CLI) tools.You may also need to enable a firewall exception so that the tools canhandle HTTP requests.
2. Copy the URL of your function from the Azure Functions runtime output.
3. Paste the URL for the HTTP request into your browser’s address bar andexecute the request. The following shows the response in the browser tothe local GET request returned by the function:The response is the initial result from the HTTP function letting us knowthe durable orchestration has started successfully. It is not yet the endresult of the orchestration. The response includes a few useful URLs. Fornow, let’s query the status of the orchestration.
4. Copy the URL value for statusQueryGetUri and pasting it in thebrowser’s address bar and execute the request.The request will query theorchestration instance for the status. You should get an eventualresponse that looks like the following. This shows us the instance hascompleted, and includes the outputs or results of the durable function.

   { “instanceId”: “d495cb0ac10d4e13b22729c37e335190”, “runtimeStatus”: “Completed”, “input”: null, “customStatus”: null, “output”: [ “Hello Tokyo!”, “Hello Seattle!”, “Hello London!” ], “createdTime”: “2018-11-08T07:07:40Z”, “lastUpdatedTime”: “2018-11-08T07:07:52Z” }

5. To stop debugging, press Shift + F5.

After you have verified that the function runs correctly on your localcomputer, it’s time to publish the project to Azure.

Publish the project to Azure

You must have a function app in your Azure subscription before you canpublish your project. You can create a function app right from Visual Studio.

1. In Solution Explorer, right-click the project and select Publish.
2. Select Azure Function App, choose Create New, and then selectPublish.When you enable Run from Zip, your function app in Azure is rundirectly from the deployment package.Caution: When you choose SelectExisting, all files in the existing function app in Azure are overwritten byfiles from the local project. Only use this option when republishingupdates to an existing function app.
3. If you haven’t already connected Visual Studio to your Azure account,select Add an account….
4. In the Create App Service dialog, use the Hosting settings as specifiedin the table below the image:
   

   Setting	Suggested value	Description
   App Name	Globally uniquename	Name that uniquelyidentifies your newfunction app.
   Subscription	Choose yoursubscription	The Azuresubscription to use.
   Resource Group	myResourceGroup	Name of the resourcegroup in which tocreate your functionapp. Choose New tocreate a newresource group.
   App Service Plan	Consumption plan	Make sure to choosethe Consumptionunder Size after youclick New to create aserverless plan. Also,choose a Location ina regionnear you ornear other servicesyour functionsaccess. When yourun in a plan otherthan Consumption,you must manage thescaling of yourfunction app.
   Storage Account	General purposestorage account	An Azure storageaccount is requiredby the Functionsruntime. Click Newto create a generalpurpose storageaccount. You canalso use an existingaccount that meetsthe storage accountrequirements.

5. Click Create to create a function app and related resources in Azure withthese settings and deploy your function project code.
6. After the deployment is complete, make a note of the Site URL value,which is the address of your function app in Azure.

Test your function in Azure

1. Copy the base URL of the function app from the Publish profile page.Replace the localhost:port portion of the URL you used whentesting the function locally with the new base URL.The URL that callsyour durable function HTTP trigger should be in the following format:

   http://<APP_NAME>.azurewebsites.net/api/<FUNCTION_NAME>_HttpStart

2. Paste this new URL for the HTTP request into your browser’s addressbar. You should get the same status response as before when using thepublished app.

Fan-out/fan-in Durable Function example

Fan-out/fan-in refers to the pattern of executing multiple functionsconcurrently and then performing some aggregation on the results. Thislesson explains a sample that uses Durable Functions to implement a fan-out/fan-in scenario. The sample is a durable function that backs up all orsome of an app’s site content into Azure Storage.

Scenario overview

In this sample, the functions upload all files under a specified directoryrecursively into blob storage. They also count the total number of bytes thatwere uploaded.

It’s possible to write a single function that takes care of everything. The mainproblem you would run into is scalability. A single function execution canonly run on a single VM, so the throughput will be limited by the throughputof that single VM. Another problem is reliability. If there’s a failure midwaythrough, or if the entire process takes more than 5 minutes, the backup couldfail in a partially-completed state. It would then need to be restarted.

A more robust approach would be to write two regular functions: one wouldenumerate the files and add the file names to a queue, and another wouldread from the queue and upload the files to blob storage. This is better interms of throughput and reliability, but it requires you to provision andmanage a queue. More importantly, significant complexity is introduced interms of state management and coordination if you want to do anything more,like report the total number of bytes uploaded.

A Durable Functions approach gives you all of the mentioned benefits withvery low overhead.

The functions

This lesson explains the following functions in the sample app:

• E2_BackupSiteContent
• E2_GetFileList
• E2_CopyFileToBlob

The following sections explain the configuration and code that are used forC# scripting. The code for Visual Studio development is shown at the end ofthe lesson.

The cloud backup orchestration (Visual Studio Code andAzure portal sample code)

The E2_BackupSiteContent function uses the standard function.jsonfor orchestrator functions.

{ "bindings": [ { "name": "backupContext", "type": "orchestrationTrigger", "direction": "in" } ], "disabled": false }

Here is the code that implements the orchestrator function:

C#

#r "Microsoft.Azure.WebJobs.Extensions.DurableTask" public static async Task<long> Run(DurableOrchestrationContext backupContext) { string rootDirectory = Environment.ExpandEnvironmentVariables(backupContext.GetInput<string>() ?? ""); if (string.IsNullOrEmpty(rootDirectory)) { throw new ArgumentException("A directory path is required as an input."); } if (!Directory.Exists(rootDirectory)) { throw new DirectoryNotFoundException($"Could not find a directory named '{rootDirectory}'."); } string[] files = await backupContext.CallActivityAsync<string[]>( "E2_GetFileList", rootDirectory); var tasks = new Task<long>[files.Length]; for (int i = 0; i < files.Length; i++) { tasks[i] = backupContext.CallActivityAsync<long>( "E2_CopyFileToBlob", files[i]); } await Task.WhenAll(tasks); long totalBytes = tasks.Sum(t => t.Result); return totalBytes; }

JavaScript (Functions v2 only)

This orchestrator function essentially does the following:

1. Takes a rootDirectory value as an input parameter.
2. Calls a function to get a recursive list of files under rootDirectory.
3. Makes multiple parallel function calls to upload each file into AzureBlob Storage.
4. Waits for all uploads to complete.
5. Returns the sum total bytes that were uploaded to Azure Blob Storage.

Notice the await Task.WhenAll(tasks); (C#) and yieldcontext.df.Task.all(tasks); (JS) line. All the calls to theE2_CopyFileToBlob function were not awaited. This is intentional toallow them to run in parallel. When we pass this array of tasks toTask.WhenAll, we get back a task that won’t complete until all the copyoperations have completed. If you’re familiar with the Task Parallel Library (TPL) in .NET, then this is not new to you. The difference is that these taskscould be running on multiple VMs concurrently, and the Durable Functionsextension ensures that the end-to-end execution is resilient to processrecycling.

Tasks are very similar to the JavaScript concept of promises. However,Promise.all has some differences from Task.WhenAll. The conceptof Task.WhenAll has been ported over as part of the durable-functions JavaScript module and is exclusive to it.

After awaiting from Task.WhenAll (or yielding fromcontext.df.Task.all), we know that all function calls have completedand have returned values back to us. Each call to E2_CopyFileToBlobreturns the number of bytes uploaded, so calculating the sum total byte countis a matter of adding all those return values together.

Helper activity functions

The helper activity functions, as with other samples, are just regular functionsthat use the activityTrigger trigger binding. For example, thefunction.json file for E2_GetFileList looks like the following:

{ "bindings": [ { "name": "rootDirectory", "type": "activityTrigger", "direction": "in" } ], "disabled": false }

And here is the implementation:

C#

#r "Microsoft.Azure.WebJobs.Extensions.DurableTask" #r "Microsoft.Extensions.Logging" public static string[] Run(string rootDirectory, ILogger log) { string[] files = Directory.GetFiles(rootDirectory, "*", SearchOption.AllDirectories); log.LogInformation($"Found {files.Length} file(s) under {rootDirectory}."); return files; }

JavaScript (Functions v2 only)

const readdirp = require("readdirp"); module.exports = function (context, rootDirectory) { context.log(`Searching for files under '${rootDirectory}'...`); const allFilePaths = []; readdirp( {root: rootDirectory, entryType: 'all'}, function (fileInfo) { if (!fileInfo.stat.isDirectory()) { allFilePaths.push(fileInfo.fullPath); } }, function (err, res) { if (err) { throw err; } context.log(`Found ${allFilePaths.length} under ${rootDirectory}.`); context.done(null, allFilePaths); } ); };

The JavaScript implementation of E2_GetFileList uses the readdirpmodule to recursively read the directory structure.

Note: You might be wondering why you couldn’t just put this code directlyinto the orchestrator function. You could, but this would break one of thefundamental rules of orchestrator functions, which is that they should neverdo I/O, including local file system access.

The function.json file for E2_CopyFileToBlob is similarly simple:

{ "bindings": [ { "name": "filePath", "type": "activityTrigger", "direction": "in" } ], "disabled": false }

The C# implementation is also pretty straightforward. It happens to use someadvanced features of Azure Functions bindings (that is, the use of theBinder parameter), but you don’t need to worry about those details for thepurpose of this walkthrough.

C#

#r "Microsoft.Azure.WebJobs.Extensions.DurableTask" #r "Microsoft.Azure.WebJobs.Extensions.Storage" #r "Microsoft.Extensions.Logging" #r "Microsoft.WindowsAzure.Storage" using Microsoft.WindowsAzure.Storage.Blob; public static async Task<long> Run( string filePath, Binder binder, ILogger log) { long byteCount = new FileInfo(filePath).Length; // strip the drive letter prefix and convert to forward slashes string blobPath = filePath .Substring(Path.GetPathRoot(filePath).Length) .Replace('\\', '/'); string outputLocation = $"backups/{blobPath}"; log.LogInformation($"Copying '{filePath}' to '{outputLocation}'. Total bytes = {byteCount}."); // copy the file contents into a blob using (Stream source = File.Open(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) using (Stream destination = await binder.BindAsync<CloudBlobStream>( new BlobAttribute(outputLocation))) { await source.CopyToAsync(destination); } return byteCount; }

JavaScript (Functions v2 only)

The JavaScript implementation does not have access to the Binder featureof Azure Functions, so the Azure Storage SDK for Node takes its place.

const fs = require("fs"); const path = require("path"); const storage = require("azure-storage"); module.exports = function (context, filePath) { const container = "backups"; const root = path.parse(filePath).root; const blobPath = filePath .substring(root.length) .replace("\\", "/"); const outputLocation = `backups/${blobPath}`; const blobService = storage.createBlobService(process.env['AzureWebJobsStorage']); blobService.createContainerIfNotExists(container, (error) => { if (error) { throw error; } fs.stat(filePath, function (error, stats) { if (error) { throw error; } context.log(`Copying '${filePath}' to '${outputLocation}'. Total bytes = ${stats.size}.`); const readStream = fs.createReadStream(filePath); blobService.createBlockBlobFromStream(container, blobPath, readStream, stats.size, function (error) { if (error) { throw error; } context.done(null, stats.size); }); }); }); };

The implementation loads the file from disk and asynchronously streams thecontents into a blob of the same name in the “backups” container. The returnvalue is the number of bytes copied to storage, that is then used by theorchestrator function to compute the aggregate sum.

Note: This is a perfect example of moving I/O operations into anactivityTrigger function. Not only can the work be distributed acrossmany different VMs, but you also get the benefits of checkpointing theprogress. If the host process gets terminated for any reason, you know whichuploads have already completed.

Run the sample

You can start the orchestration by sending the following HTTP POST request.

POST http://{host}/orchestrators/E2_BackupSiteContent Content-Type: application/json Content-Length: 20 "D:\\home\\LogFiles"

Note: The HttpStart function that you are invoking only works withJSON-formatted content. For this reason, the Content-Type:application/json header is required and the directory path is encodedas a JSON string. Moreover, HTTP snippet assumes there is an entry in thehost.json file which removes the default api/ prefix from all HTTPtrigger functions URLs. You can find the markup for this configuration in thehost.json file in the samples.

This HTTP request triggers the E2_BackupSiteContent orchestratorand passes the string D:\home\LogFiles as a parameter. The responseprovides a link to get the status of the backup operation:

HTTP/1.1 202 Accepted Content-Length: 719 Content-Type: application/json; charset=utf-8 Location: http://{host}/admin/extensions/DurableTaskExtension/instances/b4e9bdcc435d460f8dc008115ff0a8a9?taskHub=DurableFunctionsHub&connection=Storage&code={systemKey} (...trimmed...)

Depending on how many log files you have in your function app, thisoperation could take several minutes to complete. You can get the latest statusby querying the URL in the Location header of the previous HTTP 202response.

GET http://{host}/admin/extensions/DurableTaskExtension/instances/b4e9bdcc435d460f8dc008115ff0a8a9?taskHub=DurableFunctionsHub&connection=Storage&code={systemKey}

HTTP/1.1 202 Accepted Content-Length: 148 Content-Type: application/json; charset=utf-8 Location: http://{host}/admin/extensions/DurableTaskExtension/instances/b4e9bdcc435d460f8dc008115ff0a8a9?taskHub=DurableFunctionsHub&connection=Storage&code={systemKey} {"runtimeStatus":"Running","input":"D:\\home\\LogFiles","output":null,"createdTime":"2017-06-29T18:50:55Z","lastUpdatedTime":"2017-06-29T18:51:16Z"}

In this case, the function is still running. You are able to see the input that wassaved into the orchestrator state and the last updated time. You can continueto use the Location header values to poll for completion. When the statusis “Completed”, you see an HTTP response value similar to the following:

HTTP/1.1 200 OK Content-Length: 152 Content-Type: application/json; charset=utf-8 {"runtimeStatus":"Completed","input":"D:\\home\\LogFiles","output":452071,"createdTime":"2017-06-29T18:50:55Z","lastUpdatedTime":"2017-06-29T18:51:26Z"}

Now you can see that the orchestration is complete and approximately howmuch time it took to complete. You also see a value for the output field,which indicates that around 450 KB of logs were uploaded.

References