Azure Virtual Machine – Best Practices

Provisioning a VM in Azure involves more components than  the VM itself. With adaptation of ARM Azure has decomposed the VM into set of services.There are compute, networking, storage and other peripheral elements. While provisioning the VM (either from Portal or Template) We should always consider these services as individuals and then provide their references where needed. For example Create the VNet and appropriate subnets and while  creating the VM provide  Subnet as reference.That way  it becomes easy to manage and change things on demand ex- You can detach & attach NIC cards  if in case your communication model changes.

Here is high level decomposition of services which constitute a Virtual Machine in Azure-

• Resource Group- A Resource group work as container for set of resources who shares same life cycle and management policies.As stated before VM is actually combination of multiple service glued togather, we do need Resource Group to hold them in one basket.

• Virtual Network-With ARM way of deployment each VM has to be deployed inside a Virtual Network to provide isolation/control on provisioned infrastructure.Once VNet in place we have to divide this further into subnet (VLAN concept in data center) and that is where individual VM will be provisioned.in nutshell it help you defines internal addressing.

• Virtual Machine- For IaaS VM has been primary construct. Now when you are migrating the VM from onprem, you can directly convert the machines into VHD's( Azure supports only VHD, support for VHDX is not yet facilitated).To achieve this we can make use of waagent( if your workload is Linux based)or sysprep ( on Windows front) . Once thats in-place we should port them to Azure storage.While provisioning the VM reference those VHD files from blob storage. On other hand if you are creating a fresh VM you can choose the same from the list of published images on azure gallery.

• OS Disk- It's VHD file backed by Azure storage to provide persistence operating disk for Azure VM's. Because of it's persistence it's immune from host machine going down.

• Temporary disk- This is ephemeral storage given to you free of cost by Azure.Any VM when gets created will always come with temporary disk (the D: drive on Windows). This disk is stored on a physical drive on the host machine. Because it's not backed by Azure storage it's tagged with Machines life cycle events.There is high chance of losing the data even during the reboot of VM. That limits the use of this storage only for the data which is temporary in nature.

• Data Disks- Just like OS disk this also gets stored on Azure storage as Page Blob. In contrast to temporary Disk this provides persistent storage for  application data. 

• Public IP - When provisioned, VM will get a private IP from the list of available ips in specified subnet. But that cant be used for public communication like RDP till you have VPN connectivity in place.To facilitate the same you can create one Public IP and associate that with aforementioned VM.

• NIC- NIC's(Network Interface Cards) are essential for any Virtual Machine being provisioned in Azure. It encapsulates all the network and communication related informations. With ARM you can associate multiple NIC's on given VM( selected tier has to support that).Given the circumstances you can detach the NIC from one VM and attach the same to another.

• NSG- NSG (Network Security Groups) are used to control the flow of traffic in and out of your VM. It's an extension to ACL's(Access Control Lists) which were used uding the classic model. Under this you can control the traffic(allow/deny) at different level, it presents the opportunity to get associated with subnet ,NIC or both. It's bi-directional in nature which means you can control ingress and egress both the traffic.

• Diagnostics- Diagnostic logging provides you appropriate logs which could latter be used for troubleshooting purpose.

Having discussed the major components, let's dive deep into some of the Best Practices-

1.Use Azure Managed Disks With VMs

When you choose this you don’t have to worry about placing the disks across multiple storage accounts in order to ensure you stay within the scalability limits for the storage accounts.Now it's Azure responsibility to do this management for you. - Azure handles that for you.Earlier to improve IOPS(input output operations per second) you have the responsibility to place each data disks into single storage account.Moreover it provides additional benefits like-Support for Azure backUp Service,simple and scalable VM deployment along with granular access control 

2.Preventing Charges For Azure VM When Not In Use

One thing about Azure worth considering is that it makes distinction between "stopped" and "deallocated" states of the VM. You are charged when the VM status is stopped, but not when the VM is deallocated.

If you are operating from Azure Portal, press of Stop button will always deallocates the VM. However, if you are shutting it down through the OS while logged in, the VM goes into stopped mode , while in stopped mode VM's will still be charged.

Same holds true in case of deletion - When you delete delete a VM, the VHDs that are associated with this will not automatically get deleted. That means you can safely delete the VM without losing data. However, you will still be charged for storage. To avoide unnecessary incur you have got to delete the file from blob storage.

Azure Virtual Machine - Architecture

Microsoft Azure is built on Microsoft’s definition of commodity infrastructure. The most intriguing part of Azure is its cloud operating system that is at its heart. During the initial days of azure when it started it stated using fork of windows as its underlying platform Back then they named it as red dog operating system & red dog hypervisor. If you go into the history of Azure the project which became azure was originally named as project red dog. David Cutler was the brain behind designing and developing the various Red Dog core components and it was he who gave this name.in his own words- the premises of Red Dog (RD) is being able to share a single compute node across several properties. This enables better utilization of compute resources and the flexibility to move capacity as properties are added, deleted, and need more or less compute power. This is turn drives down capital and operational expenses.

It was actually a custom version of windows and the driving reason for this customization was because hyper v during those didn’t had the features which was needed for Azure (particularly support for booting from VHD). if you try to understand the main components of its architecture we can count four pillars-

1. Fabric Controller
2. Storage
3. Integrated Development Tools and Emulated Execution Environment
4. OS and Hypervisor

Those were initial (early 2006) days of azure as it matured running a fork of an OS is not ideal (in terms of cost and complexity), so Azure team talked to the Windows team, and efforts were made to use Windows itself. As time passed windows eventually caught up and now Azure runs on Windows.

Azure Fabric Controller

Among there one component which contributed immensely in its success is fabric controller. The fabric controller owns all the resources in the entire cloud and runs on a subset of nodes in a durable cluster. It manages the placement, provisioning, updating, patching, capacity, load balancing, and scale out of nodes in the cloud all without any operational intervention.

Fabric Controller which still is backbone of azure compute is the kernel of the Microsoft Azure cloud operating system. Azure Fabric Controller regulates the creation, provisioning, de-provisioning and supervising of all the virtual machines and their back-end physical server. In other words It provisions, stores, delivers, monitors and commands the virtual machines (VMs) and physical servers that make up Azure. One added benefit is that It also detects and responds to both software and hardware failure automatically.

Patch Management

When we try to understand the underlying mechanism/workflow which Microsoft follows for patch management the common misconception is that it keeps updating all the nodes just like we do in our environment. But things in cloud is little different, AS Azure hosts are image-based (hosts boot from VHD) and it follows the image based deployment. So instead of just having patches delivered, azure roll out new VHD of the host operating system. Means they are not actually going and patching everyone but instead azure update at one place and because its orchestrated update it can use this image to update the whole environment.

This offers a major advantage in host maintenance as the volume itself can be replaced, enabling quick rollback. Host updates role out every few weeks (4-6 weeks), with an approach where updates are well-tested before they are rolled out broadly to the data centers. It’s the responsibility of Microsoft to ensure that each roll out is tested before updating the data center servers. To do so they start this implementation with few fabric controller stamps which could be called as pilot cluster and then once through they will gradually push the updated to production (Data Center) hosts. The underlying technology behind this is called Update Domain (UDs). When you create VM’s and put them in an availability set they get bucketed into update domain (by default you get 5 but there are provisions to increase them to 20). So, all the VMs part of availability set will get distributed equally among these UDs. With this the patching will take place in batches and Microsoft will ensure that at a time only single update domain should go for patching. You can call this as staged rollout. To understand this in more detail let’s see how Fabric controller manages the partitioning-

Partitioning

Under Azure’s Fabric Controller it has two types of partitions: Update Domains(UDs) and Fault Domains(FDs). These two are responsible for not only high availability for also for resiliency of infrastructure with this in place in empowers the Azure with ability to recover from failures and continue to function. It's not about avoiding failures, but responding to failures in a way that avoids downtime or data loss.

Update Domain: An Update Domain is used to upgrade a service’s role instances in groups. Azure deploys service instances into multiple update domains. For an in-place update, the FC brings down all the instances in one update domain, updates them, and then restarts them before moving to the next update domain. This approach prevents the entire service from being unavailable during the update process.

Fault Domain: Fault Domain defines potential points of hardware or network failure. For any role with more than one instance, the FC ensures that the instances are distributed across multiple fault domains, in order to prevent isolated hardware failures from disrupting service. All exposure to server and cluster failure in Azure is governed by fault domains.

Azure Compute Stamp

As in Azure, things gets divided into stamps where each stamp will have one fabric controller and this fabric controller is the one responsible for managing the VMs inside that stamp. In Azure, there are only two type of stamps, it could either be compute stamp or storage stamp. This Fabric controller is also not single; it has its distributed branches. Based on the available information, azure will have 5 replicas of the fabric controller where it uses synchronous mechanism to replicate the state. In this setup, there will be one primary and to which control pane will talk to. Now it’s the responsibility of this primary to act on the instruction (example- provision a VM) and also let other replicas know about it. And when at least 3 of them acknowledge the fact that this operation is going to happen then the operation take place (this is called quorum based approach).

VM Availability

Talking about Azure Virtual Machines there are three major components (Compute, Storage, Networking) which constitute Azure VM.While discussing Azure Virtual Machine (VM) resiliency with customers, they typically assume it is comparable to their on-prem VM architecture and as such, features from on-prem is expected in Azure. Well it is not the case, thus I wanted to put this together to provide more clarity on the VM construct in Azure to better understand how VM availability in Azure is typically more resilient then most on-prem configuration.

"Talking about Azure Virtual Machines there are three major components (Compute, Storage, Networking) which constitute Azure VM. So, when we talk about Virtual machine in Azure we must take two dependencies into consideration. Windows Azure Compute (to run the VM’s), and Windows Azure Storage (to persist the state of those VM’s). What this means is that you don’t have a single SLA, instead you actually have two SLA’s. And as such, they need to be aggregated since a failure in either, could render your service temporarily unavailable."

Under this article lets have our discussions on Compute(VM) and Storage components.

Azure Storage:You can check my other article where I have talked about this in great details, on how an Azure Storage Stamp is a cluster of servers hosted in Azure Datacenter. These Stamps follows layer architecture with built-in redundancy to provide High Availability. Under this multiple (most of the times 3) replicas of each file, referred as Extent, are maintained on multiple different servers partitioned between Update Domains and Fault Domains. Each write operations are performed Synchronously (till we are talking about intra Stamp replication) and control is returned only after the 3 copies completed the write, thus making the write operation strongly consistent.

Virtual Machine:

Microsoft Azure has provided a means to detect health of virtual machines running on the platform and to perform auto-recovery of those virtual machines should they ever fail. This process of auto-recovery is referred to as “Service Healing”, as it is a means of “healing” your service instances. In this case, Virtual Machines and the Hypervisor physical hosts are monitored and managed by the Fabric Controller. The Fabric Controller has the ability to detect failures.

It can perform the detection in two mode-Reactive and Proactive. If the FC detects failures in reactive mode (Heartbeats missing) or proactive mode (known situations leading to a failure) from a VM or a hypervisor host, it will initiate a recovery by either redeploying the VM on a healthy host (same host or another host) and mark the failed resource as unhealthy and remove it from the rotation for further diagnosis. This process is also known as Self-Healing or Auto Recovery.
With Above diagram we can see different layers of the system where faults can occur and the health checks that Azure performs to detect them

*auto-recovery mechanism is enabled and available on virtual machines across all the different VM sizes and offerings, across all Azure regions and datacenters.