Category Archives: Virtualization

Virtualize Windows Storage Replica cluster on vSphere

I have been testing Windows Storage Replica cluster (available in the upcoming Windows 10 Server) on VMware recently for the purposes of testing  geographically dispersed windows failover clusters without depending on shared SAN disk .

I created a two node Windows failover cluster with the following configuration.  Both VM’s are running in two separate datacenters with stretched VLANS, plus a windows fileshare witness is used for quorum located in a third witness site.  Both sites are using separate vCenter servers.

VM Configuration

  • ESXi 5.5 update 2
  • VMX-10
  • Guest OS – Set to Windows Server 2012

Networking

  • Production Network using VMXNET3
  • Cluster Heartbeat Network using E1000 – Separate physical network with no l3 routing

Storage

  • 1x LSI Logic SAS SCSI controller for guest OS VMDK
    • 1x thin provisioned VMDK for OS
  • 1x LSI Logic SAS SCSI controller for Storage Replica with Physical Bus sharing enabled.
    •  1x 40GB Eager Zero Thick VMDK Configured for Storage Replica Data
    • 1x 10GB Eager Zero Thick VMDK Configured for Log

VM config

One problem that I discovered, was that the virtualized SCSI controller must be set to physical bus sharing mode as storage replica uses SCSI3 persistant reservations to lock access to the disk on each node, even though replication happens in-guest using SMB3.  If SCSI bus sharing is not enabled, you will not be able to import the disks into cluster manager and therefore not be able to enable storage replica.

Note: If you have problemscreating an eager zero thick VMDK, enable fault tolerance on the VM.  It will inflate the disks automatically for you.

If the disk configuration is configured correctly as above, you should then be able to successfully import the disks to windows failover cluster manager, and configure storage replica between the data VMDK’s. (40GB in the below screenshot)The 10GB VMDK’s are used for storage replica log.

adddisks

To set the actual storage replica cluster up, I recommend following the excellent storage replica guide created by the Windows storage team located below http://go.microsoft.com/fwlink/?LinkID=514902

One thing I noticed was that the SOURCE replica disk must be a member of a cluster role before replication can be enabled, or Cluster Shared volumes must be enabled on the disk.

enablereplication

Once the replica is configured, you will notice the “replication role” in failover cluster manager becomes active. This specifies which disks are replication source and destination

replicationstatus

Initial testing looks good, task manager was showing 1.6Gb/s a second network throughput while the initial sync was run.

One other issue I discovered was I was unable to reverse replication by failing over the CSV from one node to another, however using the new “Set-SRpartnership” powershell cmdlets  to reverse replication worked fine.  This should be fixed in a later beta of Windows server

I see some really good use cases for Storage Replica in the future, one example would be for SQL Server clustering. Now, SQL clustering without shared SAN requires database replication using always on availability groups which are only available at a high cost with the enterprise edition of SQL server.

SQL Standard edition still supports clustering, but only with shared disk. So with storage replica, you could potentially create SQL failover clusters using the cheaper SQL standard edition and place the databases on storage replica volumes instead of using database replication.   SQL 2014 standard also supports cluster shared volumes which would add a few more good improvements when paired with storage replica.

Eliminating RDM complexity with storage replica in the next version of Windows Server

Recently, the new features available in the next version of Windows Server were announced along with a public preview.  One hot feature that caught my attention was storage replica. Storage replica enables block level synchronous or asynchronous replication with two storage agnostic volumes over SMB3.

If Synchronous replication is used, you can create Metro clusters using Windows Server Failover cluster manager.  You select two volumes that support SCSI3 persistent reservations, create the replica, and the volume will appear as a standard clustered disk resource in failover cluster manager which can be failed over to other nodes in the cluster.

Asynchronous replication can be used for scenarios such as data migration, as you can create replication partners between other servers or even to other volumes on the same server.  Since the replication is block and not file based, open files such as SQL databases are not a problem.

Many VMware customers, including myself, utilize in-guest virtualized metro clusters to create high availability across two or more datacenters for mission critical tier-1 applications.  These applications require four or more nines of availability, which cannot be dependent on a single VM for HA.

Unfortunately, not all applications that require high availability support application based replication and instead depend on shared clustered disk for this functionality.  So instead designs are based on SAN disk that is virtualized and replicated to two geo locations at the back end by products such as EMC VPLEX, and then presented to the guest as an RDM device.

You can create a cluster in a box scenario with a single shared VMDK, however unless the multi-writer flag is disabled you cannot run the two cluster VM’s across more than a single host.   Windows failover cluster requires SCSI persistent reservations to lock access to the disk, so unfortunately this solution what is common utilized for Oracle RAC also won’t work for Microsoft clusters.

So, in hindsight, the only way to create virtualized Windows based Metro clusters that require shared cluster disk is to use RDM devices across two or more guests.

I have the following issues with RDM’s used for in-guest clustering

  • They create operational dependencies between your virtualization and storage departments. To resize an RDM, it requires the virtualization administrator to coordinate with the storage administrator to resize the backend LUN.  This is difficult to automate without third party products.
  •  They create operational dependencies between application owners, OS administrators, and virtualization teams. RDM’s using SCSI bus sharing requires the virtualized SCSI adapter to be configured in physical bus sharing mode. If physical bus sharing is enabled, live vmotion is disabled.  Therefore any maintenance on the VMware hosts requires coordination between all these teams to agree on downtime as cluster resources are failed over.  Unfortunately, storage replica in synchronous mode still requires SCSI reservations, so one way around this is to use the Windows in-guest iSCSI initiator and target in loopback mode to get around this limitation.  Hopefully in future VMware versions we can vMotion with physical bus sharing enabled.
  • SAN migrations become more complex. Yes, with solutions like VPLEX you can present another SAN behind the VPLEX controller and migrate data on the fly, but what if you want to move to another vendor’s mirroring product entirely?  This requires potential downtime as data is manually copied in-guest from one array vendor to another, unless another third party block level replication software is used.  Clusters demand high uptime by design, so receiving the OK for these outage windows can take weeks of negotiation.
  • The 256 LUN limit per VMware host allows less consolidation of VM’s to host, and can cause you to reach this LUN limit faster. Especially if you use products like VERITAS storage foundation with in-guest mirroring, as this will require a minimum of two RDM’s per logical volume.
  • RDM’s are complex to manage. Unless this can be orchestrated in some way, it can be difficult to identify and select the correct RDM when adding disks to a new VM.

With storage replica, managing virtualized metro clusters are simplified as we can use VMDK’s the same as all other virtual machines. Replication dependency is moved away from the underlying hardware and closer to the application level, where it belongs.   I have demoed and automated the creation of virtualized metro clusters running on VMware in my lab, so I will share these guides in upcoming blog posts.   If you want to get started yourself, the following microsoft resources have good information.

Windows Server Technical Preview Storage Replica Guide –

http://go.microsoft.com/fwlink/?LinkID=514902

Whats New in Storage Services in Windows Server Technical Preview

http://technet.microsoft.com/en-us/library/dn765475.aspx