As Hurricane Florence continues its journey to the mainland, our thoughts are with those in its path. Please stay safe. We’re actively monitoring Azure infrastructure in the region. We at Microsoft have taken all precautions to protect our customers and our people.
Our datacenters (US East, US East 2, and US Gov Virginia) have been reviewed internally and externally to ensure that we are prepared for this weather event. Our onsite teams are prepared to switch to generators if utility power is unavailable or unreliable. All our emergency operating procedures have been reviewed by our team members across the datacenters, and we are ensuring that our personnel have all necessary supplies throughout the event.
As a best practice, all customers should consider their disaster recovery plans and all mission-critical applications should be taking advantage of geo-replication.
Rest assured that Microsoft is focused on the readiness and safety of our teams, as well as our customers’ business interests that rely on our datacenters.
You can reach our handle @AzureSupport on Twitter, we are online 24/7. Any business impact to customers will be communicated through Azure Service Health in Azure portal.
If there is any change to the situation, we will keep customers informed of Microsoft’s actions through this announcement.
For guidance on Disaster Recovery best practices see references below:
We’ve seen the early predictions about the 2018 Atlantic hurricane season. Global Weather Oscillations, citing the “strongest cycle in 70 years,” is projecting 18 storms with six making landfall in the United States and three or four classified as major storms rated Category 3.5 or higher. Scientists and climatologists blame hotter ocean temperatures for the increasing number and severity of these weather events, and this year’s Atlantic currents are running hot.
These storms are a big concern for businesses and homeowners living along the Atlantic and Gulf of Mexico coasts. But that’s only a fraction of the land mass and the population of this country. The Midwest may face a much lower hurricane threat, for example, but it isn’t in the clear when it comes to weather-related catastrophes, such as tornados. Whatever the location and threat, disaster avoidance has become top of mind for everyone.
Chris Hood, IT director at First Enterprise Bank, a four-branch community bank in Oklahoma City, Okla., recalls how he took his position as “the one-man IT department” for the bank only a short time after the infamous tornado outbreak of February 2009. In that historical event, six of 15 tornadoes across seven states touched down in Oklahoma on Feb. 10, including the most powerful tornado on record, an EF4. Since then, Hood noted, disaster preparedness has been front-of-mind.
Tape backup was the backbone of the bank’s primary disaster recovery plan in 2009. The plan was to recover to one of the larger branch offices if a storm or other disaster hit the bank’s headquarters. But the bank had migrated its workloads to virtual machines, and Hood wanted to take advantage of that to implement a more proactive approach that would elevate disaster avoidance to the forefront of its continuity plan.
Time to modernize
The bank was already virtualizing its servers using VMware hypervisor technology and making some forays into software-defined storage and with StarWind Software hyper-converged infrastructure storage with iSCSI connections to virtual workloads. But Hood wanted to update the infrastructure to capitalize on the latest technology advancements.
One goal was to support WAN-based cluster failover among the bank’s four locations. “We liked the idea of replicating data among offices across a 300 Mbps link,” Hood said. “That would enable us to fail over workloads should one of the offices become compromised.”
Hood looked at replacing StarWind with VMware vSAN, “but it was simply too expensive from the standpoint of licensing and hardware cost,” he said. His research uncovered StorMagic’s SvSAN with dual-way stretch clustering. Implementation went smoothly, and the cost was 87% less than the VMware product, which also required more link bandwidth and more equipment. The steps involved in recovering from an outage went from “five pages of small print … to one page and a couple of steps for failing over,” Hood said. Better yet, it put the bank into full compliance with federal regulations.
Synchronous mirroring in a server failover cluster is an increasingly popular high-availability model for customers, said Bruce Kornfeld, chief marketing officer at StorMagic.
“Smaller shops may have smaller equipment stacks and less demanding workloads,” Kornfeld said, “but their data is just as mission-critical as larger firms. We provide high-availability solutions to the folks whose entire storage infrastructure totals between 2 TB to 20 TB of capacity.”
That same strategy and StorMagic’s technology are also finding play at the edge of larger shops, where they’re used to support stretch clustering of selected applications to provide service continuity, Kornfeld said. StorMagic’s magic is linked to its significantly lower cost compared with competitors, he added.
To sweeten the deal, StorMagic recently announced Federal Information Processing Standard 140-2-compliant encryption of data at rest in its storage pool “without the need for self-encrypting media,” Kornfeld said. This feature also encrypts data in flight between clustered nodes running StorMagic, and it fully supports the Key Management Interoperability Protocol standard so any KMIP-compliant encryption key manager can be used with the setup.
Not for everyone
Is this disaster avoidance technology sufficient for the likes of Citibank? Probably not. But with failover clustering and Veeam Software backup and replication software and tape, which First Enterprise continues to use, Hood is confident he has taken every reasonable disaster avoidance and DR measure to avoid preventable disasters and facilitate the bank’s recovery from one.
This month’s column isn’t an endorsement of any particular strategy or product. Rather, it’s testimony to the idea that disaster preparedness can best be implemented as a built-in rather than a bolt-on. As you begin planning your next storage refresh, consider adding requirements for availability and continuity so that the infrastructure’s resiliency is built-in, instead of needing to be added in after the fact.
Thanks to Hood, First Enterprise has an above average chance of surviving an EF4 tornado. He accomplished that by considering disaster avoidance requirements as design goals when creating storage infrastructure for his production environment. That’s the way it should be done.
Roger Harland, a resident of the smallest of the U.S. Virgin Islands, St. John, is on medication for a recent liver transplant, and over the summer, with Hurricane Irma approaching, the family couldn’t get in touch. So his nephew began hatching a plan with his family to get to the island and make sure his uncle had the medication he needed.
Nicholas Harland, a senior manager at Microsoft, ran into a roadblock right away. St. John lacks its own airport, so the typical method of passage would be to fly to the island of St. Thomas and take a ferry over. But Irma decimated the St. Thomas airport, forcing Harland to find another path.
So he turned to the internet, finding a Facebook group of people trying to get supplies to those affected by the hurricane. There he connected with a restaurant owner cooking meals and sending them by boat to St. John from St. Croix, an island 45 minutes south that was spared the full brunt of Irma.
“I flew there, and these people that I didn’t know put me up in their house overnight and fed me and gave me an air-conditioned room,” said Harland. “And then I got on a boat the next morning and went over to St. John.”
A little more than a week after Irma hit, Harland reunited with his uncle and aunt Fran. It turned out Roger was well supplied with four months’ worth of medication, and Harland brought another two months’ worth.
An experienced backpacker, Harland brought everything he needed to survive for a couple days. He says he wanted to help, not turn himself into someone who needed aid.
That’s when his mission took on a larger purpose. He had brought with him $7,000 worth of satellite phones and other communication equipment to help Roger and Fran stay connected to the family.
Harland’s equipment stash included 12 Baofeng UV5-R radios, powerful handheld Ham Radios, to hand out to friends and neighbors. If they caught on, Harland said, he was ready to recruit folks who could set up radio towers to restore some form of communication to the island.
But it turned out there was a group already doing that. After getting a signal, Harland started asking around to find out who was working on re-connecting the island. He hooked up with a group of local IT pros working to restore internet service via a new WiFi network. Though cell service was down and the island was without power, the group was able use an undersea cable that still worked as the backbone of its network.
They set up the first WiFi hotspot on the island four days after Irma hit, before Harland arrived. They also managed to set up a point-to-point link at a National Park Service office where first responders from the Federal Emergency Management Agency were stationed and at a few businesses, so customers could run credit cards.
When Harland arrived, he met up with the team at a pizza shop that would serve as their home base. Together, they came up with a plan to provide WiFi to the entire island.
Harland was only supposed to stay a couple of days, help out his uncle, drop off the equipment he brought and then head out. But the second catastrophic storm to hit the area, Hurricane Maria, disabled the St. Croix airport, leaving him stranded.
Harland did eventually get off the island, but three months later, he is still making trips to St. John from Redmond, Wash., and is back there again between Christmas and New Year’s. Each time he has hauled with him thousands of dollars in gear to bolster the WiFi network. This effort is a first for Harland, taking his philanthropic giving to a new level of engagement.
“I give to charities, I have payroll deductions, but I’ve never actually been in a position where I had skills and experience that could really impact an entire community like that,” Harland said.
Harland and the local IT guys — Matt Gyuraki, Jason Monigold, Morgan Barlas, Rob Tutton, Pete Miazga — have since founded a nonprofit called Love City Community Network. The goal is to continue developing a plan for bringing WiFi to storm-stricken areas like St. John and serve as a resource for other groups that want to replicate that effort. Harland recruited a fellow engineer, Majdi Abbas, a former Yahoo engineer, to join the effort, and he has since become a critical team member.
Harland is in St. John again right now, his fourth trip there, and possibly his last for a while. This time around, he will be setting up more point-to-point equipment to increase the capacity of the network, as it is starting to run up against bandwidth issues.
And even if he can’t go back for a while, Harland will remain prominently involved in the effort. During an interview at his Microsoft office in Redmond, Wash., part of the network briefly went down, illustrating that the work didn’t end when the network launched. His office, where he is surrounded by a quartet of monitors and a poster showing Irma’s path in September, is a mix of his work life and his growing mission on St. John.
At his apartment, also in Redmond, Harland has been testing the equipment he brings to St. John. That gives him what he calls “probably one of the fastest WiFi networks around.”
Harland has always been an internet enthusiast. He got his first job with an internet service provider at the age of 15, back in an era when every small town had a dial-up internet service provider. He’s been with Microsoft for five years now, and today he works in the company’s Global Network Acquisition Group, which plans and manages Microsoft’s worldwide network of data centers. He knows a few things about managing a complex network.
Those skills came in handy when the group was hunkered down during Maria. Still without power, the team had to take down much of the infrastructure it had built because of the risk posed by the second storm. So they began building a plan to present to aid and governmental organization to get funding for their work and the equipment they needed. In putting together the bones of the new WiFi network, they went analog.
“We got out a map and some pins and strings, and pieced together sight lines to get point-to-point wireless all over the island,” Harland said.
Once the storm clouds cleared, the team that would become Love City Community Network got back to work. And so did other island residents. That included country music star Kenny Chesney, who has a home on the island that was destroyed by Irma. His charity Love for Love City aimed to identify the biggest needs and create an avenue for people to donate money and supplies to the rescue effort.
A friend of Harland’s in Boston bought some wireless equipment and sent it down on a private jet owned by a hedge fund manager on the island. With the local public airports still decimated, that same plane got Harland off the island nearly two weeks after he first arrived, along with some injured residents.
Wealthy and isolated, with 70 percent of the 20-square-mile island an undeveloped national park, the challenges on St. John were different from those in dense cities. With only about 5,000 people, a small proportion of the overall Virgin Islands population, Harland says St. John was not the top priority for government response. The team counted on their knowledge of the island, finding the right spots on the rugged terrain to put up their equipment
At the same time, other organizations on the ground were working to get things back up to speed. One was a group called Global Disaster Immediate Response Team, or Global DIRT. Founded to respond to the massive 2010 earthquake that devastated Haiti, the organization aims to fill in gaps in disaster response and work with both government organizations and locals participating in the recovery.
In the Virgin Islands and Puerto Rico, the organization has been working to get communications back online. Zac Clancy, director of IT at Global DIRT, told GeekWire that the locals on St. John have made a huge difference in the recovery — doing everything from getting communications back online to making meals to just walking around and asking people what they need.
As the internet and smartphones have proliferated around the world, getting the web back up has become almost as important as getting power again, and that’s why it’s among Global DIRT’s top priorities on the ground.
“When you have these larger bodies arriving, like your Navy, your Coast Guard, if you don’t have any sort of communications infrastructure laid out already, or if there isn’t a way to communicate anyways, things become very complicated,” Clancy said.
The first thing the Global DIRT team did when they arrived in St. John was hand out a bunch of satellite phones. And the large volume of equipment brought down by Harland, the local IT pros on the island and other volunteers sped things up. One of the local IT guys, Gyuraki, extrapolated what they did on St. John and installed a WiFi network working with Google, Facebook and the non-profit NetHope to set up an “air fiber” connection linking cell towers on a 19-square-mile Puerto Rican island.
Despite all this work, there remains much to do. It took almost two months for the first buildings to get power back on St. John, and Harland’s team estimates that still only about 60 percent of houses on the island have power. In Puerto Rico, devastated by Hurricane Maria, many remain without power.
Beyond the efforts of residents, aid organizations and government agencies, tech companies have stepped up to help the recovery mission. Tesla, Facebook, Google and many more all worked in the immediate aftermath of the storms to restore power and connectivity to the regions.
Microsoft has pitched in with both donations and technology. It gave $1 million for disaster relief in the immediate aftermath, and as of November it had donated more than $5 million.
Microsoft is teaming with NetHope and aid organizations to provide connectivity through TV white space technology. That involves tapping into unused blocks of broadcast spectrum between TV channels to deliver wireless broadband connections over great distances and difficult terrain.
The efforts of these tech companies has greatly aided recovery, said Clancy of Global DIRT. What helps most in terms of long-term recovery is manpower. Getting talented engineers down to fix problems is a key part of getting connections back up and keeping them functional. A long-term presence beyond just the initial shock of the storm makes it easier to maintain and manage new systems built during the recovery.
“It’s hard to gain perspective over a smaller period of time,” Clancy said. “One of the reasons we find ourselves so effective is because we are here for months, and we will be here months later. Because we’ve been here for that long and because we anticipate being here for that long, the decision-making of how we’re doing things becomes a bit easier because we don’t necessarily feel a crunch for time and we don’t necessarily feel like we need to make the most amount of impact in a couple days and hope it pans out.”
Beyond the effort on St. John, Harland hopes the Love City team group can help provide a model for re-connecting communities after disasters. Harland said initial recovery efforts from non-governmental organizations don’t often focus on getting businesses up and running so the economy can rev up again.
That was a big part of their work on St. John.
“When you lose your communication circuits and all of your power, all the banks close and you can’t use an ATM, and stores can’t process payment cards. So the entire economy became cash and what we realized, people can’t get more cash because banks the are closed, and the ferries weren’t running,” Harland said. “We deployed WiFi to the grocery store, the hardware and a pharmacy so that they could run credit cards again.”
Say the word “disaster” and what comes to mind? An earthquake, a drought, a flood, a tsunami, a hurricane? These are big and brutish events. They grab headlines, inspire people to donate, and trigger international relief efforts.
But what about the many micro-disasters that can, at any time, befall poor families across the developing world? For those who live on a perpetual economic knife edge, even a small misfortune or an unexpected turn of events can devastate their hopes and dreams.
Let’s turn to Thimi, a tiny village in the ancient valley of Bhaktapur in Nepal – a nation that sits in the shadow of the Himalayas and is among the world’s poorest. An overwhelming majority of its 30 million people rely on farming to subsist – often on fragmented, hilly and marginal land where weather and other conditions are subject to extremes. In this rural society, a family typically measures its wealth in the number of animals it keeps.
For years, Rajesh Ghimire and his wife, Sharadha, worked hard to build up a modest herd of 45 cows, goats, and buffaloes. The farm was generating enough income to raise their two children, support four other relatives, and even pay six workers to help out. The Ghimeres had their eyes fixed on better times ahead, and were saving to send their daughter, Ekta, to medical school.
Then, their own micro-disaster struck. A series of heatwaves triggered an outbreak of the disease, anthrax. Almost half of their animals were wiped out and, with that, most of their dreams. The money that had been put away for Ekta’s studies had to be used to save the farm. Seven years later, the family is still trying to claw back what it lost.
In light of the devastation that has taken place in Puerto Rico caused by Hurricane Maria—and the subsequent impact to the local communications infrastructure—domestic and international calls made via Skype to landlines and mobiles in the affected Puerto Rico regions are free of charge until further notice.* This is effective immediately; the list of area codes are below.
Our thoughts are with everyone impacted by Hurricane Maria. We hope this provides you with another alternative to reach loved ones, friends, and colleagues in Puerto Rico.
The free calling applies only to Skype calls made to landlines and mobiles in the following Puerto Rico area codes:
+1 787 (Puerto Rico) +1 939 (Puerto Rico)
To learn how to make Skype calls to landlines and mobiles, visit Skype Support.
Visit Skype.com to download Skype and feel free to reach out to us on Twitter.
Across the country, we watched as Hurricane Harvey hit Texas with devastating force last Friday night. As the crisis continues to unfold, our thoughts are with the millions of people in Houston and throughout the Gulf Coast who are impacted by this disaster, which has been described as unprecedented in scale and scope.
Since the weekend we’ve been in close contact with nonprofit partners and organizations leading the relief efforts in Texas to understand the specific needs they have. Based on their feedback, we have taken some immediate steps. We are contributing $2.5 million in cash, technology and services to support relief efforts and to help nonprofit and public-sector organizations as they assist the community in recovery. This includes an initial cash donation to the American Red Cross for its most pressing needs.
We have also harnessed the generosity of our employees in the U.S. through our employee giving program. Every dollar contributed by our employees to relief organizations working in the area will be matched dollar for dollar by Microsoft. To date, this effort has already raised $100,000 for organizations including the American Red Cross and Team Rubicon. As our employees continue to donate in the days and weeks ahead, we’ll continue to scale the impact of their giving by matching these donations.
To raise awareness and make it easier for people across the country to make donations toward relief and recovery efforts, we are using our company websites and social media channels —which together reach millions of people — to highlight the need and opportunity to help through the American Red Cross.
Beyond these immediate steps, our long-term goal is to help the communities across the region to recover and residents to rebuild their lives. As in other disasters, we will focus on working closely with the nonprofits and government agencies that play such a vital role in supporting the community. This model of providing immediate aid, followed by support for long-term recovery, is one we’ve deployed in response to 300+ disasters over the last 30 years.
We’re committed to continuing to play our part this week, and in the months and years ahead.