Serverless Computing- Azure Functions

In recent past we have witnessed a paradigm shift in the way applications were architected, and lot of this can be attributed to the emergence of different public cloud providers mainly Azure & AWS. With monolithic three tier architectures fading from the scene we are embracing an emerging programming style which is very functional and event driven in nature. Under this- it requires to have implementations that can react quickly to events, and run the code in response to that. 

Here is intriguing Quote from Wilbur Creech,where he said :

Many people believe that decentralization means loss of control, That's simply not true. You can improve control if you look at control as the control of events and not people.

Now this statement can never be more true than today where we are looking to achieve decentralization at each level.

So, what essentially you are looking for is to author discrete, event-driven application code.

Authoring such application is just one portion of the story, once that’s in place you have to be worried about the platform to achieve best out of your code in most optimal way, particularly keeping the adhoc nature of its execution in mind.

Existing Landscape

Azure Storage - High Level Architecture

Windows Azure storage is a cloud storage service that is highly durable, available, and scalable. Once your data is stored in Azure storage, you can access that data any time and from anywhere. It provides the following four abstractions (services)- Blob storage, Table storage, Queue storage, and File storage. Each of these has different role to play, you can get more information here…

In conjunction to aforementioned services It also provides the storage foundation for Azure Virtual Machines in the form of persistent data disks.

The goal of this article is not to explain the offering but to understand the fundamentals of Azure storage using which they are able able to achieve all the design goal sets out. Let’s try to decipher how Microsoft do some of these things on Azure storage under the cover.

Any data stored on Azure Storage is triplicated by default, in normal circumstances it creates at least 3 copies of that inside storage stamp (and potentially another region with geo-resiliency enabled). So, before diving into storage architecture it’s only fair to put some lights on Stamps-

Azure Storage Stamp

Azure Divides things into Stamps where each stamp has its own fabric controller. A single Storage Stamp can best be understood as a cluster of N racks of storage nodes, where each rack is built out as a separate fault domain with redundant networking and power. Clusters typically range from 10 to 20 racks with 18 disk-heavy storage nodes per rack.

Microsoft deploys these stamps in its Azure data centers across the world, and adds more stamps as demand grows. Inside the datacenter it's used as a unit of deployment and management. They provide huge assistance in achieving fault-tolerance.

When user creates storage accounts all the data of that is stored on a single stamp. They do get migrated between stamp only when need arises, the way it works is that Microsoft always make sure that single storage stamp should only be utilized ~75% of its capacity & bandwidth. This is because ~20% is kept as a reserve for –

1. Disk short stroking to gain better seek times and higher throughput by utilizing the outer tracks of the disks
2. To continue providing storage capacity and availability in the presence of a rack failure within a stamp. When the storage stamp reaches ~75% utilization, the location service migrates accounts to different stamps using Inter-Stamp replication

Location Service

Location Service in Azure is the one responsible for storage stamps management. Another notable responsibility of this to manage the account name-spaces across all stamps. Internally this Location Service itself is distributed across two geographical locations for its own disaster recovery. That makes it immune from geo failure.

As shown in the architecture diagram- We have a Location Service with two storage stamps and within the stamps we have the all three-layer mentioned. While tracking the resources Location Service look for them at each storage stamp in production across all the locations. Now when an application requests a new account for storing data, it specifies the location affinity for the storage (e.g., US North). The LS then chooses a storage stamp within that location as the primary stamp for the account using heuristics based on the load information across all stamps (which considers the fullness of the stamps and other metrics such as network and transaction utilization). The LS then stores the account metadata information in the chosen storage stamp, which tells the stamp to start taking traffic for the assigned account. The LS then updates DNS to allow requests to now route from the name https://AccountName.service.core.windows.net/ to that storage stamp’s IP which it has exposed for external traffic

Architecture Layers inside Stamps

system To keep the data durable, consistent and available with in specific Azure Region lest understand the different layer which constitute the stamp. Windows Azure Storage is a layered system, so let's take it from the bottom up-

Stream Layer: The first or lowest layer on this subsystem which also gets referred as DFS (Distributed File System) layer. This layer stores the bits on disk and is actually in charge of handling the disk, it’s the responsibility of this layer to persist your data on the disk by distributing and replicating the data across many servers to provide durability within a storage stamp. you can think of the underling system as JBOD with in the stamp, so your data when you store in the Azure Storage Service it gets stored in the DFS into these files which is called extent and these extend gets replicated three times across UD’s (update domain)/FD’s (fault domain). The unique thing about this file system is that this is append only file system so when you override any data azure always keeps on appending the data to these extents.

Partition Layer: Then comes the partition layer which can be called as the brain of Azure Storage service. Most of the decision making happens at this layer. It has this significance because it serves two unique purposes-First this layer is built for managing and understanding higher level data abstractions (Blob, Table, Queue, Files). So, it’s this layer which understand what blob, is table is and how to perform transaction on those objects.Secondly it providing a scalable object namespace, and responsible for massive scalable indexIn addition to above two it does help in providing transaction ordering and strong consistency for objects, storing object data on top of the stream layer.  '

Front End Layer:  And finally, at the top of Azure Storage Service we have front end layer which provides a rest protocol for those abstractions (Blob, Table, Queue, Files). The Front-End layer consists of a set of stateless servers that take incoming requests. Upon receiving a request, an FE looks up the AccountName, authenticates and authorizes the request, then routes the request to a partition server in the partition layer (based on the PartitionName). Every write request which comes to the system will have a partition key specified to it. So, in nutshell it does extend its roles for authentication/authorization.

Global Name-space

One of the key design goal of Azure Storage Service is to provide a single global namespace that allows the data to be stored and accessed in a consistent manner from any location in the world. To provide this capability Microsoft leverages DNS as part of the storage namespace and break the storage namespace into three parts: an account name, a partition name, and an object name. As a result, all data is accessible via a uniform URI of the form:

http(s)://AccountName.1.core.windows.net/PartitionName/ObjectName

Two Replication Engines

To address the replication challenge which is backbone of all the Azure Storage Service design goals Azure in general two types of replication-

 Intra-Stamp Replication: Under this replication model Azure keeps your data durable within a region or stamp. It provides synchronous replication and is focused on making sure all the data written into a stamp is kept durable within that stamp. It keeps three replicas of the data across different nodes in different fault domains/Update Domain to keep data durable within the stamp in the face of disk, node, and rack failures. Intra-stamp replication is done completely by the stream layer and is on the critical path of the customer’s write requests. Once a transaction has been replicated successfully with intra-stamp replication, success can be returned back to the customer. Under this engine Azure provides strong consistency, until data is written on all the three places transaction is not committed. And because the replication implementation is happening at stream layer it can provide

Inter-Stamp Replication: Under this replication model Azure provides asynchronous replication and is focused on replicating data across stamps. Inter-stamp replication is done in the background and is off the critical path of the customer’s request. Because of asynchronous replication the write operation is not strongly consistent. This replication is at the object level, where either the whole object is replicated or recent delta changes are replicated for a given account. Inter-stamp replication is configured for an account by the location service and performed by the partition layer. 

“Inter-stamp replication is focused on replicating objects and the transactions applied to those objects, whereas intra-stamp replication is focused on replicating blocks of disk storage that are used to make up the objects.”

Conclusion

When Windows Azure Storage was being comprehended, it had some design goals to achieve -

Consistency: One of the design goal of Microsoft for WAS was to provide strong consistency with in a region. By virtue of mentioned technological brilliance all the data which goes to Azure Storage gets triplicated and All committed data across all 3 replicas are identical.

Durability: Second design goal was that the data must be durable that means there must be proper replication mechanism in place, to address the same all data stored with at least 3 replicas

Availability: Can read from any 3 replicas; If any issues writing seal extent and continue appending to new extent

Performance/Scale: Retry based on 95% latencies; Auto scale out and load balance based on load/capacity. As it should automatically scale to meet the customers peak demand

*You also need a global name space to access the data around the world

Azure Virtual Machine – Best Practices

Provisioning a VM in Azure involves more components than  the VM itself. With adaptation of ARM Azure has decomposed the VM into set of services.There are compute, networking, storage and other peripheral elements. While provisioning the VM (either from Portal or Template) We should always consider these services as individuals and then provide their references where needed. For example Create the VNet and appropriate subnets and while  creating the VM provide  Subnet as reference.That way  it becomes easy to manage and change things on demand ex- You can detach & attach NIC cards  if in case your communication model changes.

Here is high level decomposition of services which constitute a Virtual Machine in Azure-

• Resource Group- A Resource group work as container for set of resources who shares same life cycle and management policies.As stated before VM is actually combination of multiple service glued togather, we do need Resource Group to hold them in one basket.

• Virtual Network-With ARM way of deployment each VM has to be deployed inside a Virtual Network to provide isolation/control on provisioned infrastructure.Once VNet in place we have to divide this further into subnet (VLAN concept in data center) and that is where individual VM will be provisioned.in nutshell it help you defines internal addressing.

• Virtual Machine- For IaaS VM has been primary construct. Now when you are migrating the VM from onprem, you can directly convert the machines into VHD's( Azure supports only VHD, support for VHDX is not yet facilitated).To achieve this we can make use of waagent( if your workload is Linux based)or sysprep ( on Windows front) . Once thats in-place we should port them to Azure storage.While provisioning the VM reference those VHD files from blob storage. On other hand if you are creating a fresh VM you can choose the same from the list of published images on azure gallery.

• OS Disk- It's VHD file backed by Azure storage to provide persistence operating disk for Azure VM's. Because of it's persistence it's immune from host machine going down.

• Temporary disk- This is ephemeral storage given to you free of cost by Azure.Any VM when gets created will always come with temporary disk (the D: drive on Windows). This disk is stored on a physical drive on the host machine. Because it's not backed by Azure storage it's tagged with Machines life cycle events.There is high chance of losing the data even during the reboot of VM. That limits the use of this storage only for the data which is temporary in nature.

• Data Disks- Just like OS disk this also gets stored on Azure storage as Page Blob. In contrast to temporary Disk this provides persistent storage for  application data. 

• Public IP - When provisioned, VM will get a private IP from the list of available ips in specified subnet. But that cant be used for public communication like RDP till you have VPN connectivity in place.To facilitate the same you can create one Public IP and associate that with aforementioned VM.

• NIC- NIC's(Network Interface Cards) are essential for any Virtual Machine being provisioned in Azure. It encapsulates all the network and communication related informations. With ARM you can associate multiple NIC's on given VM( selected tier has to support that).Given the circumstances you can detach the NIC from one VM and attach the same to another.

• NSG- NSG (Network Security Groups) are used to control the flow of traffic in and out of your VM. It's an extension to ACL's(Access Control Lists) which were used uding the classic model. Under this you can control the traffic(allow/deny) at different level, it presents the opportunity to get associated with subnet ,NIC or both. It's bi-directional in nature which means you can control ingress and egress both the traffic.

• Diagnostics- Diagnostic logging provides you appropriate logs which could latter be used for troubleshooting purpose.

Having discussed the major components, let's dive deep into some of the Best Practices-

1.Use Azure Managed Disks With VMs

When you choose this you don’t have to worry about placing the disks across multiple storage accounts in order to ensure you stay within the scalability limits for the storage accounts.Now it's Azure responsibility to do this management for you. - Azure handles that for you.Earlier to improve IOPS(input output operations per second) you have the responsibility to place each data disks into single storage account.Moreover it provides additional benefits like-Support for Azure backUp Service,simple and scalable VM deployment along with granular access control 

2.Preventing Charges For Azure VM When Not In Use

One thing about Azure worth considering is that it makes distinction between "stopped" and "deallocated" states of the VM. You are charged when the VM status is stopped, but not when the VM is deallocated.

If you are operating from Azure Portal, press of Stop button will always deallocates the VM. However, if you are shutting it down through the OS while logged in, the VM goes into stopped mode , while in stopped mode VM's will still be charged.

Same holds true in case of deletion - When you delete delete a VM, the VHDs that are associated with this will not automatically get deleted. That means you can safely delete the VM without losing data. However, you will still be charged for storage. To avoide unnecessary incur you have got to delete the file from blob storage.

Azure Virtual Machine - Architecture

Microsoft Azure is built on Microsoft’s definition of commodity infrastructure. The most intriguing part of Azure is its cloud operating system that is at its heart. During the initial days of azure when it started it stated using fork of windows as its underlying platform Back then they named it as red dog operating system & red dog hypervisor. If you go into the history of Azure the project which became azure was originally named as project red dog. David Cutler was the brain behind designing and developing the various Red Dog core components and it was he who gave this name.in his own words- the premises of Red Dog (RD) is being able to share a single compute node across several properties. This enables better utilization of compute resources and the flexibility to move capacity as properties are added, deleted, and need more or less compute power. This is turn drives down capital and operational expenses.

It was actually a custom version of windows and the driving reason for this customization was because hyper v during those didn’t had the features which was needed for Azure (particularly support for booting from VHD). if you try to understand the main components of its architecture we can count four pillars-

1. Fabric Controller
2. Storage
3. Integrated Development Tools and Emulated Execution Environment
4. OS and Hypervisor

Those were initial (early 2006) days of azure as it matured running a fork of an OS is not ideal (in terms of cost and complexity), so Azure team talked to the Windows team, and efforts were made to use Windows itself. As time passed windows eventually caught up and now Azure runs on Windows.

Azure Fabric Controller

Among there one component which contributed immensely in its success is fabric controller. The fabric controller owns all the resources in the entire cloud and runs on a subset of nodes in a durable cluster. It manages the placement, provisioning, updating, patching, capacity, load balancing, and scale out of nodes in the cloud all without any operational intervention.

Fabric Controller which still is backbone of azure compute is the kernel of the Microsoft Azure cloud operating system. Azure Fabric Controller regulates the creation, provisioning, de-provisioning and supervising of all the virtual machines and their back-end physical server. In other words It provisions, stores, delivers, monitors and commands the virtual machines (VMs) and physical servers that make up Azure. One added benefit is that It also detects and responds to both software and hardware failure automatically.

Patch Management

When we try to understand the underlying mechanism/workflow which Microsoft follows for patch management the common misconception is that it keeps updating all the nodes just like we do in our environment. But things in cloud is little different, AS Azure hosts are image-based (hosts boot from VHD) and it follows the image based deployment. So instead of just having patches delivered, azure roll out new VHD of the host operating system. Means they are not actually going and patching everyone but instead azure update at one place and because its orchestrated update it can use this image to update the whole environment.

This offers a major advantage in host maintenance as the volume itself can be replaced, enabling quick rollback. Host updates role out every few weeks (4-6 weeks), with an approach where updates are well-tested before they are rolled out broadly to the data centers. It’s the responsibility of Microsoft to ensure that each roll out is tested before updating the data center servers. To do so they start this implementation with few fabric controller stamps which could be called as pilot cluster and then once through they will gradually push the updated to production (Data Center) hosts. The underlying technology behind this is called Update Domain (UDs). When you create VM’s and put them in an availability set they get bucketed into update domain (by default you get 5 but there are provisions to increase them to 20). So, all the VMs part of availability set will get distributed equally among these UDs. With this the patching will take place in batches and Microsoft will ensure that at a time only single update domain should go for patching. You can call this as staged rollout. To understand this in more detail let’s see how Fabric controller manages the partitioning-

Partitioning

Under Azure’s Fabric Controller it has two types of partitions: Update Domains(UDs) and Fault Domains(FDs). These two are responsible for not only high availability for also for resiliency of infrastructure with this in place in empowers the Azure with ability to recover from failures and continue to function. It's not about avoiding failures, but responding to failures in a way that avoids downtime or data loss.

Update Domain: An Update Domain is used to upgrade a service’s role instances in groups. Azure deploys service instances into multiple update domains. For an in-place update, the FC brings down all the instances in one update domain, updates them, and then restarts them before moving to the next update domain. This approach prevents the entire service from being unavailable during the update process.

Fault Domain: Fault Domain defines potential points of hardware or network failure. For any role with more than one instance, the FC ensures that the instances are distributed across multiple fault domains, in order to prevent isolated hardware failures from disrupting service. All exposure to server and cluster failure in Azure is governed by fault domains.

Azure Compute Stamp

As in Azure, things gets divided into stamps where each stamp will have one fabric controller and this fabric controller is the one responsible for managing the VMs inside that stamp. In Azure, there are only two type of stamps, it could either be compute stamp or storage stamp. This Fabric controller is also not single; it has its distributed branches. Based on the available information, azure will have 5 replicas of the fabric controller where it uses synchronous mechanism to replicate the state. In this setup, there will be one primary and to which control pane will talk to. Now it’s the responsibility of this primary to act on the instruction (example- provision a VM) and also let other replicas know about it. And when at least 3 of them acknowledge the fact that this operation is going to happen then the operation take place (this is called quorum based approach).

VM Availability

Talking about Azure Virtual Machines there are three major components (Compute, Storage, Networking) which constitute Azure VM.While discussing Azure Virtual Machine (VM) resiliency with customers, they typically assume it is comparable to their on-prem VM architecture and as such, features from on-prem is expected in Azure. Well it is not the case, thus I wanted to put this together to provide more clarity on the VM construct in Azure to better understand how VM availability in Azure is typically more resilient then most on-prem configuration.

"Talking about Azure Virtual Machines there are three major components (Compute, Storage, Networking) which constitute Azure VM. So, when we talk about Virtual machine in Azure we must take two dependencies into consideration. Windows Azure Compute (to run the VM’s), and Windows Azure Storage (to persist the state of those VM’s). What this means is that you don’t have a single SLA, instead you actually have two SLA’s. And as such, they need to be aggregated since a failure in either, could render your service temporarily unavailable."

Under this article lets have our discussions on Compute(VM) and Storage components.

Azure Storage:You can check my other article where I have talked about this in great details, on how an Azure Storage Stamp is a cluster of servers hosted in Azure Datacenter. These Stamps follows layer architecture with built-in redundancy to provide High Availability. Under this multiple (most of the times 3) replicas of each file, referred as Extent, are maintained on multiple different servers partitioned between Update Domains and Fault Domains. Each write operations are performed Synchronously (till we are talking about intra Stamp replication) and control is returned only after the 3 copies completed the write, thus making the write operation strongly consistent.

Virtual Machine:

Microsoft Azure has provided a means to detect health of virtual machines running on the platform and to perform auto-recovery of those virtual machines should they ever fail. This process of auto-recovery is referred to as “Service Healing”, as it is a means of “healing” your service instances. In this case, Virtual Machines and the Hypervisor physical hosts are monitored and managed by the Fabric Controller. The Fabric Controller has the ability to detect failures.

It can perform the detection in two mode-Reactive and Proactive. If the FC detects failures in reactive mode (Heartbeats missing) or proactive mode (known situations leading to a failure) from a VM or a hypervisor host, it will initiate a recovery by either redeploying the VM on a healthy host (same host or another host) and mark the failed resource as unhealthy and remove it from the rotation for further diagnosis. This process is also known as Self-Healing or Auto Recovery.
With Above diagram we can see different layers of the system where faults can occur and the health checks that Azure performs to detect them

*auto-recovery mechanism is enabled and available on virtual machines across all the different VM sizes and offerings, across all Azure regions and datacenters.

Azure Storage - Account Monitoring

Azure supports storage account monitoring from management portal Azure Portal. While doing so via portal it uses Storage Analytics to track metrics for your account and log request data.

P.S. Storage accounts with a replication type of Zone-Redundant Storage (ZRS) do not have the metrics or logging capability enabled at this time.

How to: Configure monitoring for a storage account

1. In the Azure Portal, click Storage, and then click the storage account name to open the dashboard.
2. Click Configure, and scroll down to the monitoring settings for the blob, table, and queue services.
   
3. In monitoring, set the level of monitoring and the data retention policy for each service:
   • To set the monitoring level, select one of the following:
     Minimal - Collects metrics such as ingress/egress, availability, latency, and success percentages, which are aggregated for the blob, table, and queue services.
     Verbose - In addition to the minimal metrics, collects the same set of metrics for each storage operation in the Azure Storage Service API. Verbose metrics enable closer analysis of issues that occur during application operations.
     Off - Turns off monitoring. Existing monitoring data is persisted through the end of the retention period.

• To set the data retention policy, in Retention (in days), type the number of days of data to retain from 1 to 365 days. If you do not want to set a retention policy, enter zero. If there is no retention policy, it is up to you to delete the monitoring data. We recommend setting a retention policy based on how long you want to retain storage analytics data for your account so that old and unused analytics data can be deleted by system at no cost.

1. When you finish the monitoring configuration, click Save.

You should start seeing monitoring data on the dashboard and the Monitor page after about an hour.

Until you configure monitoring for a storage account, no monitoring data is collected, and the metrics charts on the dashboard and Monitor page are empty.

After you set the monitoring levels and retention policies, you can choose which of the available metrics to monitor in the Azure Portal, and which metrics to plot on metrics charts. A default set of metrics is displayed at each monitoring level. You can use Add Metrics to add or remove metrics from the metrics list.

Metrics are stored in the storage account in four tables named $MetricsTransactionsBlob, $MetricsTransactionsTable, $MetricsTransactionsQueue, and $MetricsCapacityBlob. For more information, see About Storage Analytics Metrics.

How to: Customize the dashboard for monitoring

On the dashboard, you can choose up to six metrics to plot on the metrics chart from nine available metrics. For each service (blob, table, and queue), the Availability, Success Percentage, and Total Requests metrics are available. The metrics available on the dashboard are the same for minimal or verbose monitoring.

1. In the Azure Portal, click Storage, and then click the name of the storage account to open the dashboard.
2. To change the metrics that are plotted on the chart, take one of the following actions:
   • To add a new metric to the chart, click the colored check box next to the metric header in the table below the chart.
   • To hide a metric that is plotted on the chart, clear the colored check box next to the metric header.
     
3. By default, the chart shows trends, displaying only the current value of each metric (the Relativeoption at the top of the chart). To display a Y axis so you can see absolute values, select Absolute.
4. To change the time range the metrics chart displays, select 6 hours, 24 hours, or 7 days at the top of the chart.

How to: Customize the Monitor page

On the Monitor page, you can view the full set of metrics for your storage account.

• If your storage account has minimal monitoring configured, metrics such as ingress/egress, availability, latency, and success percentages are aggregated from the blob, table, and queue services.
• If your storage account has verbose monitoring configured, the metrics are available at a finer resolution of individual storage operations in addition to the service-level aggregates.

Use the following procedures to choose which storage metrics to view in the metrics charts and table that are displayed on the Monitor page. These settings do not affect the collection, aggregation, and storage of monitoring data in the storage account.

How to: Add metrics to the metrics table

1. In the Azure Portal, click Storage, and then click the name of the storage account to open the dashboard.
2. Click Monitor.
   The Monitor page opens. By default, the metrics table displays a subset of the metrics that are available for monitoring. The illustration shows the default Monitor display for a storage account with verbose monitoring configured for all three services. Use Add Metrics to select the metrics you want to monitor from all available metrics.
   

   Note

   Consider costs when you select the metrics. There are transaction and egress costs associated with refreshing monitoring displays. For more information, see Storage Analytics and Billing.
3. Click Add Metrics.
   The aggregate metrics that are available in minimal monitoring are at the top of the list. If the check box is selected, the metric is displayed in the metrics list.
   
4. Hover over the right side of the dialog box to display a scrollbar that you can drag to scroll additional metrics into view.
   
5. Click the down arrow by a metric to expand a list of operations the metric is scoped to include. Select each operation that you want to view in the metrics table in the Azure Portal.
   In the following illustration, the AUTHORIZATION ERROR PERCENTAGE metric has been expanded.
   
6. After you select metrics for all services, click OK (checkmark) to update the monitoring configuration. The selected metrics are added to the metrics table.
7. To delete a metric from the table, click the metric to select it, and then click Delete Metric.
   

How to: Customize the metrics chart on the Monitor page

1. On the Monitor page for the storage account, in the metrics table, select up to 6 metrics to plot on the metrics chart. To select a metric, click the check box on its left side. To remove a metric from the chart, clear the check box.
2. To switch the chart between relative values (final value only displayed) and absolute values (Y axis displayed), select Relative or Absolute at the top of the chart.
3. To change the time range the metrics chart displays, select 6 hours, 24 hours, or 7 days at the top of the chart.

How to: Configure logging

For each of the storage services available with your storage account (blob, table, and queue), you can save diagnostics logs for Read Requests, Write Requests, and/or Delete Requests, and can set the data retention policy for each of the services.

1. In the Azure Portal, click Storage, and then click the name of the storage account to open the dashboard.
2. Click Configure, and use the Down arrow on the keyboard to scroll down to logging.
   
3. For each service (blob, table, and queue), configure the following:
   • The types of request to log: Read Requests, Write Requests, and Delete Requests.
   • The number of days to retain the logged data. Enter zero is if you do not want to set a retention policy. If you do not set a retention policy, it is up to you to delete the logs.
4. Click Save.

The diagnostics logs are saved in a blob container named $logs in your storage account. For information about accessing the $logs container, see About Storage Analytics Logging.

Purge Azure CDN Endpoint

Due to updates to your web application, or to quickly update assets that contain incorrect information.Sometimes you may wish to purge cached content from all edge nodes and force them all to retrieve new updated assets. 

To handle aforementioned scenario the term we use to describe this technology is Purging.Purging particular asset on Azure CDN deletes the cached version of that asset from all Edge servers on the Azure CDN. For example, if you have updated an image, or have made changes to your CSS file. Purging Azure CDN asset prevents an old version of an asset from being delivered to your clients.

Microsoft wasn't supporting the conventional way of purging the content, which intern was really difficult for customer to achieve.Under past procedure you had to contact Microsoft Support and create a ticket to purge the edge servers.Say for an example you have hosted the videos on Azure CDN and because the process was so clumsy with support tickets to force a purge on servers around the world, people would rather uploaded a file with a new name and changed the web page to point to the new file.Or best They will go with Short TTL as an Alternative to Purge.

Using Azure Portal

The support has recently been made available to Azure ARM Portal-

• In the Azure Portal, browse to the CDN profile containing the endpoint you wish to purge.
• From the CDN profile blade, click the purge button.

• On the Purge blade, select the service address you wish to purge from the URL dropdown.

• Select what assets you wish to purge from the edge nodes. If you wish to clear all assets, click the Purge all checkbox. Otherwise, type the full path of each asset you wish to purge (e.g., /pictures/test.png) in the Path textbox.
• Click the Purge button.

Using PowerShell

Get-Command -Module AzureRM.Cdn 
To invoke the purge make use of Unpublish-AzureRmCdnEndpointContent
Unpublish-AzureRmCdnEndpointContent `  
    -ResourceGroupName $ResourceGroupName `
    -ProfileName $ProfileName `
    -EndpointName $EndpointName `
    -PurgeContent = '/*'

With endpoint object in hand
$endpoint = Get-AzureRmCdnEndpoint -ResourceGroupName $ResourceGroupName `
    -ResourceGroupName $ResourceGroupName `
    -ProfileName $ProfileName `
    -EndpointName $EndpointName

$endpoint | Unpublish-AzureRmCdnEndpointContent -PurgeContent = '/*'

Related Articles : Solutions In Azure: Azure CDN

Part 1: Azure CDN For Content Delivery

Part 2: Azure CDN For Content Delivery

Best Practices for Creating Azure Resource Manager Templates -Part II

This post is second part of the series : Best Practices for Creating ARM Template, If you haven't already seen please go ahead and check the First part- Best Practices For Creating ARM template

11. Use ARM Template expression function such as trim(), toLower() as a data cleaning measure to ensure input consistency

12. ARM Termination Protection-For Critical Resource always implement Resource Lock

For resources where you have to constrain the ability to commit write actions and protect against accidental deletions, Azure Resource Lock can be a way to go.Check more on how to implement Resource Lock-

{
  "type": "extensions",
  "name": "Microsoft.EnterpriseCloud.Monitoring",
  "apiVersion": "[variables('apiVersion')]",
  "location": "[resourceGroup().location]",
  "dependsOn": [
    "[concat('Microsoft.Compute/virtualMachines/', variables('vmName'))]"
  ],
  "properties": {
    "publisher": "Microsoft.EnterpriseCloud.Monitoring",
    "type": "MicrosoftMonitoringAgent",
    "typeHandlerVersion": "1.0",
    "autoUpgradeMinorVersion": true,
    "settings": {
      "workspaceId": "[parameters('workspaceId')]"
    },
    "protectedSettings": {
      "workspaceKey": "[parameters('workspaceKey')]"
    }
  }
}

13. EnableLog Analytics To Azure VM's- Use OMS

If you are using VMs in Azure you should always opt for some monitoring and analyzing capabilities for them.OMS offers log analytic capabilities that enable you to collect and analyze machine data from virtually any source.Using data generated across devices, operating systems, workloads and user actions, OMS provides instant access to critical information through one integrated console. You can troubleshoot across multiple data sources and easily identify the root cause of operational and security issue.

Log Analytics (OMS) supports a majority of Windows and Linux variations available in Azure today,and because it's been pitched as IT management solution for the hybrid cloud you can manage any instance in any cloud, including on-premises, Azure, AWS, Windows Server, Linux, VMware, and OpenStack

{

  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",

  "contentVersion": "1.0.0.0",

  "parameters": {

    "lockedResource": {

      "type": "string"

    }

  },


  "resources": [

    {

      "name": "[concat(parameters('lockedResource'), '/Microsoft.Authorization/utLock')]",

      "type": "Microsoft.Storage/storageAccounts/providers/locks",

      "apiVersion": "2015-01-01",

      "properties": {

        "level": "CannotDelete"

      }

    }
  ]
} 

The OMS Workspace ID and OMS Workspace Key are required to onboard the OMS Service and can be found in the OMS Portal under Settings > Connected Sources

14. Tag the resources properly

Tagging provides you the ability to include metadata about your resource.Common use cases are to include references to environment types, team or division , individuals accountable.It provides subscription wide taxonomy The benefit of tags is that they can be pulled together in billing roll up or within a summary view.

"tags": { "
   "ProjectName": "[parameters(projectName)]",
   "EnvironmentType" :"[parameters('environmentType')]"
} ,

15. Have the habit for pre-deployment validation

Azure Resource Manager as it stand today has two-part validation: before and during deployment. The before deployment checks whether a template is syntactically correct and that it would be accepted by the Azure Resource Manager. Then the during deployment validation (runtime) checks for resource dependencies within your Azure subscription. The templates may seem valid, but until they are deployed as there is no fool-proof way to guarantee that it is a working template. However, as a good practice when deploying templates, make pre-deployment validation a habit as it will catch most obvious errors detected by Azure Resource Manager.

16. Don't Pass your Passwords as Plain-text:Use Azure key Vault With ARM Template

With template-based deployments that include passwords stored in parameter files (or other insecure stores), there is always the risk of leaking the password should someone access the particular file.

As an It guy we should always took to lock down environments so that only the right groups of users can login and modify the configurations.

Once you have stored your passwords as a Secret in a Key Vault, its ready to be accessed via ARM Template.

The way you can supply Key Vault Secrets to ARM templates is via use of parameters file.

"osProfile": {
    "computername": "[variables('vmName')]",
    "adminUsername": "[parameters('adminUsername')]",
    "adminPassword": "[parameters('adminPassword')]"
}

We can feed in the Secret we pushed to our Key Vault using the below snippet in our parameters file.

"adminPassword": {
     "reference": {
        "keyVault": {
   "id": "/subscriptions/{subscription-guid}/resourceGroups/{keyvault-rg}/providers/Microsoft.KeyVault/vaults/ProvisioningVault"
        },
        "secretName": "LocalAdminPass"
      }
  }

 Separation of Keys from Deployments
A best practice is to maintain separate ARM templates for-

Creation of vaults (which will contain the key material) 
Deployment of the VMs (with URI references to the keys contained in the vaults)

17. Include the Output Section
Whenever possible return some output to validate the Deployment.With Outputs section, users can specify values that are returned from deployment.For example,  you could return the URI to access a deployed resource.

"outputs":{
  "siteUri" : {
"type" : "string",
"value": "[concat('http://',reference(resourceId('Microsoft.Web/sites', parameters('siteName'))).hostNames[0])]"
   }
}

18. Use Audit Logs For Deployment Operations

Audit logs would be your first place for troubleshoot because it contains all write operations (PUT, POST, DELETE) performed on your resources.
It also helps you determine who made what changes to the subscription(s) and related Azure resources.In Future these logs can be processed for audit compliance purpose as well.

19. Disable Windows Automatic Updates  

To disable windows updates at provisioning time it self, you have to disable the same under "windowsConfiguration" Setting.This particular section comes under osProfile -

 "osProfile": {
          "computerName": "[variables('vmName')]",
          "adminUsername": "[parameters('adminUsername')]",
          "adminPassword": "[parameters('adminPassword')]",
          "windowsConfiguration": {
            "enableAutomaticUpdates": false
           }
          },

P.S.:IT's been written with - Azure Compute Provider schema version - 2015-08-01

20. Template Formatting 

• Pass your template through a JSON validator to remove extraneous commas, parenthesis, brackets that may cause an error during deployment. Try JSONlint or a linter package for your favorite editing environment (Visual Studio Code, Atom, Sublime Text, Visual Studio, etc.)
• Format your JSON for better readability. You can use a JSON formatter package for your local editor. In Visual Studio, format the document with Ctrl+K, Ctrl+D. In VS Code, use Alt+Shift+F. If your local editor doesn't format the document, you can use an online formatter.

ARM TEMPLATES KNOWN LIMITATIONS-

1. ARM Template file size can't exceed beyond 1 mb 
2. ARM Parameter Template file Size can't exceed beyong 64 kb 
3. If parameter name value contains "." character, the arm template deployment will fail in azure portal(bug).EG-My.Database 
4. Variables doesn't support ARM template expression function such as "reference()"

Related Articles : Best Practice For Creating ARM Template

Part 1: Best Practices For Creating ARM template

Part 2: Best Practices For Creating ARM template