Out-of-band Initial Replication (OOB IR) and Deduplication

A recent conversation with a customer brought out the question:   What is the best way to create an entire Replica site from scratch? At the surface this seems simple enough – configure initial replication to send the data over the network for the VMs one after another in sequence. For this specific customer however, there were some additional constraints placed:

  1. The network bandwidth was less than 10Mbps and it primarily catered to their daily business needs (email etc…). Adding more network was not possible within their budget. This came as quite a surprise because despite the incredible download speeds that are encountered these days, there are still places in the world where it isn’t as cost effective to purchase those speeds.
  2. The VMs were of size between 150GB and 300GB each. This made it rather impractical to send the data over the wire. In the best case, it would have taken 34 hours for a single VM of size 150GB.

This left OOB IR as the only realistic way to transfer data. But at 300GB per VM, it is easy to exhaust a removable drive of 1TB. That left us thinking about deduplication – after all, deduplication is supported on the Replica site. So why not use it for deduplicating OOB IR data?

So I tested this out in my lab environment with a removable USB drive, and a bunch of VMs created out of the same Windows Server 2012 VHDX file. The expectation was that at least 20% to 40% of the data would be same in the VMs, and the overall deduplication rate would be quite high and we could fit a good number of VMs into the removable USB drive.

I started this experiment by attaching the removable drive to my server and attempted to enable deduplication on the associated volume in Server Manager.

Interesting discovery #1:  Deduplication is not allowed on volumes on removable disks

Whoops! This seems like a fundamental block to our scenario – how do you build deduplicated OOB IR, if the deduplication is not supported on removable media? This limitation is officially documented here: http://technet.microsoft.com/en-us/library/hh831700.aspx, and says “Volumes that are candidates for deduplication must conform to the following requirements:  Must be exposed to the operating system as non-removable drives. Remotely-mapped drives are not supported.”

Fortunately my colleague Paul Despe in the Windows Server Data Deduplication team came to the rescue. There is a (slightly) convoluted way to get the data on the removable drive and deduplicated. Here goes:

  • Create a dynamically expanding VHDX file. The size doesn’t matter too much as you can always start off with the default and expand if required.

image

  • Using Disk Management, bring the disk online, initialize it, create a single volume, and format it with NTFS. You should be able to see the new volume in your Explorer window. I used Y: as the drive letter.

image

  • Mount this VHDX on the server you are using to do the OOB IR process.
  • If you go to Server Manager and view this volume (Y:), you will see that it is backed by a fixed disk.

image

  • In the volume view, enable deduplication on this volume by right-clicking and selecting ‘Configure Data Deduplication’. Set the ‘Deduplicate files older than (in days)’ field to zero.

image

image

You can also enable deduplication in PowerShell with the following commandlets:

PS C:> Enable-DedupVolume Y: -UsageType HyperV

PS C:> Set-DedupVolume Y: -MinimumFileAgeDays 0

Now you are set to start the OOB IR process and take advantage of the deduplicated volume. This is what I saw after 1 VM was enabled for replication with OOB IR:

image

image

That’s about 32.6GB of storage used. Wait… shouldn’t there be a reduction in size because of deduplication?

Interesting discovery #2:  Deduplication doesn’t work on-the-fly

Ah… so if you were expecting that the VHD data would arrive into the volume in deduplicated form, this is going to be a bit of a surprise. At the first go, the VHD data will be present in the volume in its original size. Deduplication happens as post-facto as a job that crunches the data and reduces the size of the VHD after it has been fully copied as a part of the OOB IR process. This is because deduplication needs an exclusive handle on the file in order to go about doing its work.

The good part is that you can trigger the job on-demand and start the deduplication as soon as the first VHD is copied. You can do that by using the PowerShell commandlet provided:

PS C:> Start-DedupJob Y: -Type Optimization

There are other parameters provided by the commandlet that allow you to control the deduplication job. You can explore the various options in the TechNet documentation: http://technet.microsoft.com/en-us/library/hh848442.aspx.

This is what I got after the deduplication job completed:

image

That’s a 54% saving with just one VM – a very good start!

Deduplication rate with more virtual machines

After this I threw in a few more virtual machines with completely different applications installed and here is the observed savings after each step:

image

I think the excellent results speak for themselves! Smile Notice how between VM2 and VM3, almost all of the data (~9GB) has been absorbed by deduplication with an increase of only 300MB! As the deduplication team as published on TechNet, VDI VMs would have a high degree of similarity in their disks and would result in a much higher deduplication rate. A random mix of VMs yields surprisingly good results as well.

Final steps

Once you are done with the OOB IR and deduplication of your VMs, you need to do the following steps:

  1. Ensure that no deduplication job is running on the volume
  2. Eject the fixed disk – this should disconnect the VHD from the host
  3. Compact the VHD using the “Edit Virtual Hard Disk Wizard”. At the time I disconnected the VHD from the host, the size of the VHD was 36.38GB. After compacting it the size came down to 28.13GB… and this is more in line with the actual disk consumed that you see in the graph above
  4. Copy the VHD to the Replica site, mount it on the Replica host, and complete the OOB IR process!

 

Hope this blog post helps with setting up your own Hyper-V Replica sites from scratch using OOB IR! Try it out and let us know your feedback.

Microsoft acquires InMage: Better business continuity with Azure

I’m pleased to announce that Microsoft has acquired InMage, an innovator in the emerging area of cloud-based business continuity.

Our customers tell us that business continuity – the ability to backup, replicate and quickly recover data and applications in case of a system failure – is incredibly important. After all, revenue, supply chains, customer loyalty, employee productivity and more are on the line. It’s also very complicated and expensive to do. CIOs consistently rank business continuity as a top priority, but often don’t have the budgets or time to do it right.

As the productivity and platform company for the mobile-first, cloud-first world, Microsoft is committed to solving this challenge for customers. This acquisition will accelerate our strategy to provide hybrid cloud business continuity solutions for any customer IT environment, be it Windows or Linux, physical or virtualized on Hyper-V, VMware or others. This will make Azure the ideal destination for disaster recovery for virtually every enterprise server in the world. As VMware customers explore their options to permanently migrate their applications to the cloud, this will also provide a great onramp.

In January, Forrester Research identified the company as an enabling technology that plays a key role in delivery for top Disaster Recovery-as-a-Service providers HP and Sungard.* In May, the company won a Penton Media Best of TechEd award. Most importantly, customers across a variety of industries, including financial services, healthcare and government, rely on InMage technology to keep their organizations up and running. An IT manager at MCR Safety, a leading manufacturer of personal protective equipment, put it simply: “The technology is like a high-performance DVR to protect our entire IT infrastructure and business.”

With the completion of this acquisition, we are now working to integrate the InMage Scout technology into our Azure Site Recovery service in order to give customers a simple, cost-effective way to ensure business continuity with the power and scale of the Azure global cloud. The company already announced its plan to enable data migration to Azure with Scout. Existing customers can continue to use the InMage products and services they trust and, moving forward, customers will acquire Scout through Azure Site Recovery. We will also continue to work with InMage service provider partners, as well as new partners, to give our mutual customers a range of solution options.

As Laura DuBois, Research Vice President for IDC, said, “This is a great move to meet important customer needs with the cloud as a target for disaster recovery. InMage has distinguished itself in a poorly understood and underserved market.”

Enterprise customers are looking for the best, most valuable ways to take advantage of the cloud. Business continuity is often a great place to start, which is why we are very focused on delivering strong solutions in this area for our customers. It is a key element of our continued effort to deliver a consistent hybrid platform and a broad range of services that connect customer, partner and Microsoft clouds.

*The Forrester Wave™: Disaster-Recovery-As-A-Service Providers, Q1 2014, Forrester Research, Inc., January 17, 2014

Microsoft acquires InMage: Better business continuity with Azure

I’m pleased to announce that Microsoft has acquired InMage, an innovator in the emerging area of cloud-based business continuity.

Our customers tell us that business continuity – the ability to backup, replicate and quickly recover data and applications in case of a system failure – is incredibly important. After all, revenue, supply chains, customer loyalty, employee productivity and more are on the line. It’s also very complicated and expensive to do. CIOs consistently rank business continuity as a top priority, but often don’t have the budgets or time to do it right.

As the productivity and platform company for the mobile-first, cloud-first world, Microsoft is committed to solving this challenge for customers. This acquisition will accelerate our strategy to provide hybrid cloud business continuity solutions for any customer IT environment, be it Windows or Linux, physical or virtualized on Hyper-V, VMware or others. This will make Azure the ideal destination for disaster recovery for virtually every enterprise server in the world. As VMware customers explore their options to permanently migrate their applications to the cloud, this will also provide a great onramp.

In January, Forrester Research identified the company as an enabling technology that plays a key role in delivery for top Disaster Recovery-as-a-Service providers HP and Sungard.* In May, the company won a Penton Media Best of TechEd award. Most importantly, customers across a variety of industries, including financial services, healthcare and government, rely on InMage technology to keep their organizations up and running. An IT manager at MCR Safety, a leading manufacturer of personal protective equipment, put it simply: “The technology is like a high-performance DVR to protect our entire IT infrastructure and business.”

With the completion of this acquisition, we are now working to integrate the InMage Scout technology into our Azure Site Recovery service in order to give customers a simple, cost-effective way to ensure business continuity with the power and scale of the Azure global cloud. The company already announced its plan to enable data migration to Azure with Scout. Existing customers can continue to use the InMage products and services they trust and, moving forward, customers will acquire Scout through Azure Site Recovery. We will also continue to work with InMage service provider partners, as well as new partners, to give our mutual customers a range of solution options.

As Laura DuBois, Research Vice President for IDC, said, “This is a great move to meet important customer needs with the cloud as a target for disaster recovery. InMage has distinguished itself in a poorly understood and underserved market.” 

Enterprise customers are looking for the best, most valuable ways to take advantage of the cloud. Business continuity is often a great place to start, which is why we are very focused on delivering strong solutions in this area for our customers. It is a key element of our continued effort to deliver a consistent hybrid platform and a broad range of services that connect customer, partner and Microsoft clouds.

*The Forrester Wave™: Disaster-Recovery-As-A-Service Providers, Q1 2014, Forrester Research, Inc., January 17, 2014

Weekend Reading: July 11th Edition — A message from Satya Nadella, Lumia 930 on sale, Microsoft Azure StorSimple previewed

It may feel like the dog days of summer where you sit, but in this edition of Weekend Reading, we’re still going full steam ahead with an important message from CEO Satya Nadella and a busy July ahead.

Check out how summer sizzles at Microsoft in Washington state, Washington D.C. and beyond …

In a message to employees, Nadella kicked off what’s going to be a big One Microsoft month by highlighting the company’s charge to enable people to thrive in a mobile-first and cloud-first world, and harkening to what he said his first day as CEO: “Our industry does not respect tradition – it only respects innovation. … We must all understand and embrace what only Microsoft can contribute to the world and how we can once again change the world.”

The Worldwide Partner Conference starts Sunday in Washington D.C. and Phil Sorgen, Microsoft’s corporate vice president of the Worldwide Partner Channel has a preview that includes Microsoft Azure StorSimple, a hybrid cloud storage solution that makes managing a company’s growing volume of data more efficient and an Azure Machine Learning technology that partners can use to build advanced analytic cloud services for their customers, in minutes.

Did we say mobile first? The super sleek, brilliantly crafted Nokia Lumia 930 went on sale this week. It includes a 20-megapixel PureView camera with ZEISS optics and optical image stabilization and is the world‘s first smartphone that can capture and playback high-quality audio in Dolby Surround Sound.

WR_2

Also next week is the 2014 Microsoft Research Faculty Summit, and our annual glimpse into the future of computing. The event brings 350 elite academic researchers to Redmond, Washington. Watch the live stream beginning July 14.

In other news, Yammer Enterprise has been added to all Office 365 Midsize Business and Office 365 Education plans, which gives these customers Yammer Enterprise licenses and no longer requires them to purchase licenses for users on their external networks.

If you’re headed to the ballpark this weekend, bring some Windows Phone apps along to help you capture all the action. The Nokia Camera app lets you take a series of fast-action photos, on-the-go editor Adobe Photoshop Express will help enhance those images, and use Creative Studio’s Focus + Blur, Color Pop and Collage features to put on the finishing touches. Plus, this week, an update to the Twitter App for Windows Phone makes it possible to upload multiple photos and easier to tag your friends. On the game front, engage in the fantasy adventure of otherworldly combat with “Order & Chaos Duels,” based on a popular collectible trading card game. The game is new to the Windows Phone Store. Or, if you just want to chillax, crank up the OnAir Player app for Windows, which lets you access music from all your devices and stream it through your house using your tablet, PC or smartphone as a remote control.

WR_3

This week on the Microsoft Facebook page, we introduced you to two artists in Paris who created a mind-bending art installation with Kinect. Tell us what you’ve created using #ICreatedThis.

WR_4

Speaking of Nadella “firsts,” from Snaps, this week, we present: A Microsoft logo on display at Satya Nadella’s first event as CEO on March 27, 2014 and captured by Brian Smale.

WR_5

Back to your regularly scheduled dog days of summer. Join us here next week for more hot news.

Posted by Aimee Riordan
Microsoft News Center Staff