Tag Archives: vulnerabilities

Congress wants CVE program changes from DHS and MITRE

The House Energy and Commerce Committee completed its investigation of the Common Vulnerabilities and Exposures program this week and requested “significant changes to the very foundation of the CVE program.”

The investigation began in March of 2017 following media reports on extensive issues with the CVE tracking system, including long backlogs for assigning vulnerability scores. In letters to both the Department of Homeland Security (DHS) and MITRE Corporation — the two entities that manage the CVE program — members of the E&C Committee noted that changes have already been made to the CVE program, but said these changes didn’t address root issues with the program.

“The historical practices for managing the CVE program are clearly insufficient. Barring significant improvements, they will likely lead again to challenges that have direct, negative impacts on stakeholders across society,” Committee members wrote in the letters. “The Committee understands and appreciates that DHS and MITRE have already undertaken reforms to try and address the issues that prompted the Committee’s initial request. However, many of these reforms target symptoms that stem from what the Committee considers to be underlying root-causes — the contract-based nature of the program and the lack of oversight — which have yet to be addressed.”

During its investigation into the CVE program, the E&C Committee found red flags right from the start.

“Given the importance of the CVE program as critical cyberinfrastructure, the Committee expected to receive substantially more documentation in response to its request than was produced,” the Committee wrote in the letter to DHS. “[T]he Committee was surprised by the dearth of produced analyses, timelines, and other oversight materials documenting the year-over-year health of the program. The Committee finds the lack of documentation produced by DHS and MITRE to be revealing in and of itself.” 

The Committee members said the contract-based nature of the CVE program led to inconsistent funding, short-term planning and thousands of vulnerabilities per year that didn’t receive CVE numbers. The Committee suggested this be changed to make funding a PPA (Program, Project, or Activity) line item in the DHS budget in the hopes of forcing DHS and MITRE to take the program more seriously.

“The documentation produced to the Committee suggests that neither DHS nor MITRE fully recognize CVE’s status as critical cyberinfrastructure. Instead, both organizations continued to manage and fund the program through a series of contract which themselves were unstable,” the committee wrote. “This approach was perhaps to be expected given that neither organization, according to produced documentation, performed the lever of oversight needed to ensure the program continued to fulfill its purpose and meet stakeholder needs.”

The Committee also requested DHS and MITRE perform biennial reviews of the CVE program “to ensure its effectiveness and stability.”

“Since the CVE program’s inception, the nature of cybersecurity threats it is meant to address has drastically evolved. So, too, have stakeholders’ needs. Yet the scope and mission of the CVE program have not undergone similar transformation,” the Committee wrote. “By conducting regular reviews of the program, officials would be able to develop short, medium and long-term goals and then evaluate their progress at achieving those goals.”

However, even these changes to so-called “root-causes” of the CVE program’s issues weren’t enough for all experts. K. Reid Wightman, vulnerability analyst at Dragos Inc., said on Twitter the recommendations showed “the wildly inaccurate CVSS scores that accompany most CVEs was out of scope,” but added he would be “glad if some progress is made on assignments at least.”

DHS and MITRE have until Sept. 4 to respond to the recommendations made by the E&C Committee.

August Patch Tuesday closes CPU bug, two zero-day exploits

Microsoft closed two zero-day vulnerabilities and released a fix for a new exploit for Intel processors on August Patch Tuesday.

Microsoft released an advisory (ADV-180018) on the latest speculative execution side channel vulnerability in Intel Core and Xeon processors called L1 Terminal Fault. Dubbed Foreshadow by security researchers, the vulnerability lets an attacker read data as it passes between a host and a virtual machine and a hypervisor.

The earlier Spectre and Meltdown variants allowed process-to-process interactions, but this latest hardware exploit allows a guest system to retrieve data from another guest system, said Brian Secrist, content manager at Ivanti, based in South Jordan, Utah.  

Once again, we have a bunch of hoops to jump through to get to full remediation… 2018 is keeping us real busy.
Brian Secristcontent manager, Ivanti

Full protection from Foreshadow (CVE-2018-3615, CVE-2018-3620 and CVE-2018-3646) on Windows requires a registry change, Microsoft patch and Intel firmware update to close the vulnerability.

“Once again, we have a bunch of hoops to jump through to get to full remediation,” Secrist said. “2018 is keeping us real busy.”

Microsoft addresses two zero-day exploits

Microsoft also closed a pair of zero-day remote code execution vulnerabilities. The first (CVE-2018-8373), in the Microsoft Scripting Engine with known exploits that affect all versions of Internet Explorer, allows an attacker to run arbitrary code on unpatched machines in the context of users who visit a specially crafted website. Depending on the user’s rights, the attacker could install programs or view and delete data. The patch changes how the scripting engine handles objects in memory. This CVE is critical for Windows desktop systems and important for server versions.

Rated important, the second zero-day (CVE-2018-8414) uses a Windows Shell bug in Windows 10 and Windows Server SAC Server Core for remote-code execution attacks. This vulnerability requires the user to run a malicious file either from email or a web site, after which an attacker can run code at the privilege level of the current user. The patch makes Windows Shell validate file paths properly.

August Patch Tuesday closes more than 60 vulnerabilities

More than half of the 60 vulnerabilities disclosed in August Patch Tuesday affect browsers or the scripting engine. Administrators should prioritize patching workstations and servers for a critical remote code execution vulnerability (CVE-2018-8345) that triggers when viewed by a user. Microsoft resolved this exploit by correcting the processing of shortcut .LNK references.

“Because the user doesn’t have to click on the malicious .LNK file to actually exploit the vulnerability, compared to browser vulnerability, it’s more likely for a server admin to be browsing through files. If they see this shortcut and the system renders it, then that’s when the exploit runs,” said Jimmy Graham, director of product management at Qualys, based in Foster City, Calif.

Jimmy Graham, QualysJimmy Graham, Qualys

Almost every major third-party vendor released patches and updates between the July and August Patch Tuesday, said Secrist. Adobe released four updates, including fixes for Adobe Flash and Acrobat. Google Chrome released version 68, and Firefox released updates for Thunderbird.

“We haven’t seen any increase in attacks or anything, just an example of better research and better coverage of vulnerabilities,” Secrist said.

July Patch Tuesday issues anger IT workers

After the July Patch Tuesday releases, Microsoft warned customers of potential SQL Server startup problems on Windows desktop (7 and 8.1) and server (2008 R2 and 2012 R2) versions on July 26. The company released several hotfixes and recommended uninstalling the July patches. Such rollbacks of faulty Microsoft updates have become a recurring headache for administrators.

Microsoft security updates for July also caused problems for the .NET Framework. On July 16, Microsoft posted a blog that “encouraged” Exchange customers to delay applying the July 10 updates to avoid disruptions with mail delivery. Hotfixes for affected systems — all supported versions of Windows Server — did not arrive until July 17. Up until that point, the only remedy was to uninstall the .NET Framework 4.7.2 update.

“Clearly there is a quality assurance issue of some kind,” Secrist said. “There’s another .NET release this month. Hopefully they spend more time on this one. We always strongly recommend you run [patches] through a test group and make sure they are stable before you push them out.”

Jeff Guillet, CEO of EXPTA Consulting in Pacifica, Calif., reached out to the Exchange product group for more information when the disruptions first occurred and said it was a two-fold problem of “really bad patches and bad communication.”

“Nobody even acknowledged that there was a problem and then all of a sudden they said, ‘Oh, by the way, we fixed this.’ [Administrators] had to troubleshoot it themselves because there was no communication from Microsoft saying this was a problem,” said Guillet.

While the intent of Patch Tuesday is to protect systems from vulnerabilities, the recent spate of patching issues concerns some IT administrators.

“Everybody’s kind of come to terms with [monthly patching], but the expectation was that a patch isn’t going to break stuff,” said Guillet. “So if it’s going to start breaking things, now I need to worry about testing it and I don’t have time because the next patches are coming up next Tuesday.”

Critical Cisco vulnerabilities patched in Policy Suite

Cisco disclosed and patched a handful of critical and high-severity vulnerabilities in its products this week.

The company fixed four critical vulnerabilities in its Policy Suite: Two are flaws that enabled remote unauthenticated access to the Policy Builder interface; one flaw is in the Open Systems Gateway initiative (OSGi) interface; and the last is in the Cluster Manager.

A successful exploit of one of the critical Cisco vulnerabilities in Policy Builder — tracked as CVE-2018-0374 — gave attackers access to the database and the ability to change any data in that database. The other vulnerability in the Policy Builder interface — tracked as CVE-2018-0376 — could have enabled an attacker to change existing repositories and create new repositories through the interface.

The third critical vulnerability could have enabled an attacker to directly connect to the OSGi interface remotely and without authentication. Once exploited, an attacker could have accessed or changed any files accessible by the OSGi process.

The last of the critical Cisco vulnerabilities — CVE-2018-0375 — was in the Cluster Manager of Cisco Policy Suite. With this flaw, an attacker could have logged in to remotely use the root account, which has static default credentials, and execute arbitrary commands.

The Cisco Policy Suite manages policies and subscriber data for service providers by connecting to network routers and packet data gateways.

The Cisco vulnerabilities affected Policy Suite releases prior to 18.2.0. The Cisco Product Security Incident Response team has already patched the vulnerabilities and has not seen any exploits in the wild.

Cisco also disclosed and patched seven high-severity flaws in its software-defined WAN (SD-WAN) products, though only one of them can be exploited remotely and without authentication — unlike the four critical vulnerabilities. One vulnerability requires authentication and local access to successfully exploit, but the others only needed authentication to be successfully exploited.

The SD-WAN vulnerabilities gave attackers the ability to overwrite arbitrary files on the operating system and execute arbitrary commands. One was a zero-touch denial-of-service vulnerability, and there were four command injection vulnerabilities.

The company also patched a high-severity denial-of-service vulnerability in the Cisco Nexus 9000 Series Fabric Switches, as well as 16 other medium-severity issues in a variety of its other products.

In other news:

  • Venmo, the mobile payment app owned by PayPal, has its API set to public by default and is exposing user data. According to researcher Hang Do Thi Duc, if a Venmo user accepts the default settings on their account, their transaction details are publicly accessible through the API. “It’s incredibly easy to see what people are buying, who they’re sending money to, and why,” Do Thi Duc said in a blog post. She noted that she was able to gather data on cannabis retailers, lovers’ quarrels and the unhealthy eating habits of users — along with their identifying information. Do Thi Duc was able to gather all of this and more by perusing the public Venmo API and looking specifically at the 207,984,218 transactions left accessible to the public in 2017. “I think it’s problematic that there is a public feed which includes real names, their profile links (to access past transactions), possibly their Facebook IDs and essentially their network of friends they spend time with,” she wrote. “And all of this is so easy to access! I believe this could be designed better.”
  • Multinational telecommunications company Telefonica suffered a data breach that exposed the data of millions of customers. Spanish users of Telefonica’s Movistar telecommunication services may have had their personal and financial information exposed because of the breach, including phone numbers, full names, national ID numbers, addresses, banking information, and call and data records. The breach was discovered after a Movistar user reported it to FACUA, a Spanish consumer rights nonprofit. Because of a design flaw in the Movistar online portal, anyone with a Movistar account could access other users’ data. FACUA notified Telefonica of the breach, and the company responded the next day, at which point FACUA made a public disclosure.
  • Oracle’s July Critical Patch Update (CPU) patched 334 security vulnerabilities, including 61 critical flaws, across many of its products. The most vulnerable affected product is the Oracle Financial Services application, which has 56 vulnerabilities — 21 of which can be exploited over the network without authentication. The vulnerabilities with the highest severity ratings — with a CVSS score of 9.8 — are in Oracle’s Financial Services, Fusion Middleware, PeopleSoft, E-Business Suite, retail applications and others. Over 200 vulnerabilities noted in the Oracle CPU affected business-critical applications. This month’s CPU has the highest number of patches at 334; the runner-up was 308 patches in July 2017.

Microsoft launches identity bounty program, offers up to $100,000

Microsoft this week expanded its bug bounty program to include security vulnerabilities in its identity services.

The software giant launched the Microsoft Identity Bounty Program, which offers payouts between $500 and $100,000 for vulnerabilities reported in Microsoft’s identity services. The scope of the identity bounty includes both consumer and enterprise services — Microsoft Accounts and Azure Active Directory, respectively — as well as login tools such as login.live.com, account.windowsazure.com, portal.office.com and the Microsoft Authenticator for iOS and Android applications.

In addition, Microsoft said the identity bounty will be available for bugs reported in the company’s implementations of specific OpenID standards.

“If you are a security researcher and have discovered a security vulnerability in the Identity services, we appreciate your help in disclosing it to us privately and giving us an opportunity to fix it before publishing technical details,” wrote Phillip Misner, principal security group manager for the Microsoft Security Response Center, in a blog post. “Further in our commitment to the industry identity standards work that we have worked hard with the community to define, we are extending our bounty to cover those certified implementations of select OpenID standards.”

The expanded bug bounty program will pay up to $100,000 for the most serious vulnerabilities, including design vulnerabilities in identity standards and bypasses for multifactor authentication. Standards-based implementation flaws will pay a maximum of $75,000, while “significant” authentication bypasses will pay a maximum of $40,000.

The identity bounty program is the latest expansion of Microsoft’s bug bounty efforts. In 2015, the company announced a major expansion of its bug bounty program that included Microsoft’s Azure platform as well as specific vulnerabilities for its Hyper-V virtualization software.

July Patch Tuesday brings three public disclosures

Microsoft announced three public disclosures from the 54 vulnerabilities released in the July Patch Tuesday.

An elevation of privilege public disclosure (CVE-2018-8313) affects all OSes except Windows 7. Attackers could impersonate processes, cross-process communication or interrupt system functionality to elevate their privilege levels. The patch addresses this issue by ensuring that the Windows kernel API enforces permissions.

“The fact that there is some level of detailed description of how to take advantage of this out in the open, it’s a good chance an attacker will look to develop some exploit code around this,” said Chris Goettl, director of product management and security at Ivanti, based in South Jordan, Utah.

A similar elevation-of-privilege vulnerability (CVE-2018-8314) this July Patch Tuesday affects all OSes except Windows Server 2016. Attackers could escape a sandbox to elevate their privileges when Windows fails a check. If this vulnerability were exploited in conjunction with another vulnerability, the attacker could run arbitrary code. The update fixes how Windows’ file picker handles paths.

A spoofing vulnerability in the Microsoft Edge browser (CVE-2018-8278) tricks users into thinking they are on a legitimate website. The attacker could then extract additional code to remotely exploit the system. The patch fixes how Microsoft Edge handles HTML content.

“That type of enticing of a user, we know works,” Goettl said. “It’s not a matter of will they get someone to do it or not; it’s a matter of statistically you only need to entice so many people before somebody will do it.”

Out-of-band updates continue

Chris Goettl of IvantiChris Goettl

Before July Patch Tuesday, Microsoft announced a new side-channel attack called Lazy FP State Restore (CVE-2018-3665) — similar to the Spectre and Meltdown vulnerabilities — on supported versions of Windows. An attacker uses a different side-channel to pull information from other registers on Intel CPUs through speculative execution.

Jimmy Graham of QualysJimmy Graham

Microsoft also updated its Spectre and Meltdown advisory (ADV180012). It does not contain any new releases on the original three variants, but the company did update the Speculative Store Bypass, Variant 4 of the Spectre and Meltdown vulnerabilities. This completed coverage for Intel processors, and Microsoft is still working with AMD to mitigate its processors.

Microsoft released out-of-band patches between June and July Patch Tuesday for a third-party Oracle Outside In vulnerability (ADV180010) that affects all Exchange servers.

“We don’t have a lot of info on the exploitability,” said Jimmy Graham, director of product management at Qualys, based in Foster City, Calif. “It should be treated as critical for Exchange servers.”

New Windows Server 2008 R2 servicing model on its way

Alongside its June Patch Tuesday, Microsoft announced plans to switch the updating system for Windows Server 2008 SP2 to a rollup model. The new monthly model will more closely match the servicing model used for older Windows versions, enabling administrators to simplify their servicing process. This will include a security-only quality update, a security monthly quality rollup and a preview of the monthly quality rollup.

“The 2008 Server users out there now need to adopt the same strategy, where they had the luxury of being able to do one or two updates if they chose to and not the rest,” Goettl said.

The new model will preview on Aug. 21, 2018. Administrators will still receive extended support for Windows Server 2008 SP2 until January 2020. After that, only companies that pay for Premium Assurance will have an additional six years of support.

For more information about the remaining security bulletins for July Patch Tuesday, visit Microsoft’s Security Update Guide.

WebAssembly updates may cancel out Meltdown and Spectre fixes

Impending WebAssembly updates may render mitigations for the Meltdown and Spectre vulnerabilities ineffective.

According to John Bergbom, Forcepoint Security Labs’ senior security researcher, once the WebAssembly updates go through the mitigations for Meltdown and Spectre that were put in place by web browsers will no longer work.

The WebAssembly, or Wasm, standard was released in March 2017 and is a compact binary language meant to improve the speed of delivery and performance of JavaScript code. Currently, all major browsers — including Chrome, Edge, Firefox and Safari — support WebAssembly. One of the benefits of WebAssembly is that programs written in languages such as C and C++ can be compiled into it and run inside the browser.

An unintended consequence of WebAssembly is that there are some potential abuses of the standard. One of these, according to Forcepoint, is the exploitation of hardware bugs, including the CPU vulnerabilities Meltdown and Spectre, which were discovered in January 2018.

“This family of CPU vulnerabilities was mitigated in browsers by lowering the precision of timers in JavaScript,” Bergbom explained in a blog post. “However, once Wasm gets support for threads with shared memory (which is already on the Wasm roadmap) very accurate timers can be created. That may render browser mitigations of certain CPU side channel attacks non-working.”

Meltdown and Spectre have been wreaking havoc on processors from Intel, AMD and ARM since early this year. Both exploit vulnerabilities in CPUs to steal sensitive data stored in memory. After these vulnerabilities were disclosed, most major vendors released patches.

Researchers previously proved that attackers could exploit Meltdown and Spectre remotely using JavaScript code that runs in browsers. In response, the major browsers released updates that affected the accuracy of the attack codes. The WebAssembly updates will effectively negate those browser mitigations.

“Like with many new technologies there are potential security issues which need to be considered,” Bergbom wrote. “Collectively, these present new opportunities for malicious actors. Much as with JavaScript, the possibilities with [WebAssembly] are — if not quite endless — very broad.”

In other news

  • U.S. cybersecurity company FireEye has denied claims that it hacked a Chinese nation-state cyberespionage group. The claims about FireEye spread over social media last week after a book by The New York Times national security journalist, David Sanger, was published. In the book, The Perfect Weapon: War, Sabotage, and Fear in the Cyber Age, Sanger said that FireEye’s 2013 report “APT1, Exposing One of China’s Cyber Espionage Units” was so detailed about the activities of Chinese hackers because FireEye, then Mandiant, obtained the information through hacking back — which is illegal in the U.S. FireEye has since released a statement denying any hacking back efforts. “Mr. Sanger’s description of how Mandiant obtained some of the evidence underlying APT1 has resulted in a serious mischaracterization of our investigative efforts,” FireEye wrote. “To state this unequivocally, Mandiant did not employ ‘hack back’ techniques as part of our investigation of APT1, does not ‘hack back’ in our incident response practice, and does not endorse the practice of ‘hacking back.'”
  • Reality Winner, the former National Security Agency contractor who admitted to leaking classified information as part of a plea deal this week, was previously a linguist with the Air Force and, while working as an NSA contractor, shared a classified report about alleged Russian interference in the 2016 U.S. election with the news outlet The Intercept. Winner, now 26 years old, was arrested in June 2017 and has been in jail since. The plea agreement she reached with federal prosecutors will give her 63 months in prison in exchange for her pleading guilty to one felony count under the Espionage Act. “All of my actions I did willfully, meaning I did so of my own free will,” Winner said in court this week, according to The New York Times. After Winner sent the classified documents to The Intercept, the news outlet published the report, which described two cyberattacks by the Russian government on U.S. elections.
  • Sophos SafeGuard security software is vulnerable to seven privilege escalation flaws. SafeGuard Enterprise Client, SafeGuard Easy and SafeGuard LAN Crypt client are all vulnerable to a flaw disclosed by security researcher Kyriakos Economou from Nettitude, a cybersecurity company headquartered in New York City. “Exploitation of those vulnerabilities requires running malicious code on the target machine and can result in privilege escalation,” the alert from Sophos said, noting that the vulnerability is at least not remotely exploitable. Some of the flaw could also enable an attacker to create an input/output control and modify token privileges. Then the attacker could run commands with system privileges on any computer running Windows and Sophos SafeGuard. Economou first discovered the vulnerabilities in December 2017, notified Sophos in January 2018, and the fix was complete in April.

Spectre v4 fix and Windows DNS patch in June Patch Tuesday

The June 2018 Patch Tuesday release addressed a total of 51 vulnerabilities, 11 of which were deemed critical, but the headline fix was a Windows DNS patch.

Experts uniformly pointed to the Windows DNS patch (CVE-2018-8225) as the most interesting fix of the month and the one that should take priority for most enterprises. Microsoft described the Windows DNS patch as addressing a remote code execution (RCE) vulnerability that affects Windows desktop versions 7 through 10 and Windows Server 2008 and newer.

Microsoft wrote in the advisory for the Windows DNS patch that if an attacker used a malicious DNS server to send corrupted DNS responses to the target, the exploit could allow for running arbitrary code in the context of the local user permissions.

Craig Young, security researcher for Tripwire’s Vulnerability and Exposure Research Team, said the full impact of the Windows DNS vulnerability “is not entirely clear.”

“Microsoft describes it as a problem processing DNS responses. Normally, I would expect that to mean that the attacker must be in a position to respond to DNS requests from a victim. This would mean that the victim is either making a DNS request to a server the attacker controls or that the attacker has a privileged network position allowing them to spoof responses from a legitimate server,” Young wrote via email. “In this case, however, Microsoft’s CVSS v3 score indicates that there is no user interaction required to trigger the vulnerability. It could be that Microsoft did not score the vulnerability properly, but it could also mean that there are circumstances where a vulnerable system will process unsolicited responses.”

Jimmy Graham, director of product management at Qualys Inc., based in Redwood City, Calif., added in a blog post that “mobile workstations that may connect to untrusted Wi-Fi are at high risk” and the Windows DNS patch should be a priority for those users.

Spectre v4 gets OS fixes

Beyond the Windows DNS patch, another highlight of the June Patch Tuesday was an update to Microsoft’s advisory regarding Spectre v4 — the latest Spectre attack method discovered in May 2018.

According to Microsoft’s updated advisory, Windows now supports Speculative Store Bypass Disable (SSBD) in Intel processors, but this in itself will not protect against Spectre v4 and will require microcode patches from Intel to fully remediate.

Microsoft couldn’t provide a timetable for when those microcode updates would be available, but it did warn users that “in testing Microsoft has seen some performance impact when SSBD is turned on. However, the actual performance impact will depend on multiple factors, such as the specific chipset in your physical host and the workloads that are running.”

Another flaw to watch

The final critical flaw for enterprises to prioritize was CVE-2018-8267, a scripting engine memory corruption vulnerability in Internet Explorer. This patch should take priority because although there have not been any attacks seen in the wild, this flaw was publicly disclosed.

According to Microsoft, this RCE vulnerability could allow an attacker to run code in the context of the current user either by luring a target to a malicious website or by embedding a malicious ActiveX control in a Microsoft Office document.

Meltdown and Spectre malware discovered in the wild

Chip makers have said they’ve seen no evidence the Meltdown and Spectre vulnerabilities have been exploited to steal customer data, but those days of relative comfort may be coming to an end.

Researchers at AV-TEST, an independent organization that tests antimalware and security software, announced this week they had discovered 139 samples of malware that “appear to be related to recently reported CPU vulnerabilities.” While the good news is that most of the malware samples appear to be based on previously published proof-of-concepts from security researchers, the bad news is that AV-TEST’s latest findings show the number of unique samples has risen sharply in recent weeks.  

The organization had previously reported the discovery of 77 unique samples of Meltdown and Spectre malware on January 17. At that time, AV-TEST said via Twitter that all identified samples were “original or modified PoC code” and that the majority of the samples were for Spectre rather than Meltdown. AV-TEST posted another update on Jan. 23 showing the unique malware samples had risen to 119.

After analyzing most of those samples, Fortinet’s FortiGuard Labs published a report Tuesday saying it was “concerned” about the potential of Meltdown and Spectre malware attacking users and enterprises.

“FortiGuard Labs has analyzed all of the publicly available samples, representing about 83 percent of all the samples that have been collected [by AV-TEST], and determined that they were all based on proof of concept code,” the research team wrote. “The other 17 percent may have not been shared publicly because they were either under NDA or were unavailable for reasons unknown to us.”

Fortinet also released several antivirus signatures to help users defend against the Meltdown and Spectre malware samples. But detecting other exploits related to these chip vulnerabilities could prove extremely difficult. While Intel and AMD have said there is no evidence the flaws have been exploited in the wild, the researchers who discovered the chip vulnerabilities say it’s “probably not” possible for organizations or users to tell whether Meltdown and Spectre have been used against them.

“The exploitation does not leave any traces in traditional log files,” according to an FAQ on the Meltdown and Spectre research site.

Defending against possible Meltdown and Spectre malware has been further complicated by patch issues. Intel recently announced it was pulling its microcode updates for the chip vulnerabilities because of reboot problems on systems running Intel’s Broadwell and Haswell processors. Microsoft later issued an out-of-band patch that disabled Intel’s update for variant 2 of the Spectre vulnerability, which involves branch target injection.

The Actual Performance Impact of Spectre/Meltdown Hyper-V Updates

The Spectre and Meltdown vulnerabilities have brought a fair amount of panic and havoc to the IT industry, and for good reason. Hardware manufacturers and operating system authors have been issuing microcode updates and patches in a hurry. As administrators, we need to concern ourselves with three things: the risks of running unpatched systems, the performance hit from patching, and quality control problems with the patches. If you’re skimming, then please pay attention to the section on ensuring that you get the update — not everyone will automatically receive the patches. Below I’ll run down what you need to know to ensure you’re protected plus a benchmark analysis of the performance impact of the recently released update patches.

What are Spectre and Meltdown?

Both Spectre and Meltdown are hardware attacks. Your operating system or hypervisor choice does not affect your vulnerability. It might affect fix availability.

Spectre is a category of assault that exploits a CPU’s “branch prediction” optimizations. A CPU looks at an instruction series that contains a decision point (if condition x then continue along path a, else jump to b) and guesses in advance whether it will follow the “else” code branch or continue along without deviation. The Wikipedia article that I linked contains further links to more details for those interested. Because the Spectre vulnerability encompasses multiple attack vectors, the predictor has more than one vulnerability. All of the ones that I’m aware of involve fooling the CPU into retrieving data from a memory location other than the code’s intended target. Spectre affects almost every computing device in existence. Fixes will affect CPU performance out of necessity.

Meltdown affects Intel and some ARM processors. AMD processors appear to be immune (AMD claims immunity; I do not know of any verified tests with contrary results). As with Spectre, Meltdown exploits target the CPU’s optimization capabilities. You can read the linked Wikipedia article for more information. For a less technical description, Luke posted a great analogy on our MSP blog. Whether you read more or not, you should assume that any running process can read the memory of any other running process. The fix for this problem also includes a performance hit. That hit is potentially worse than Spectre’s — an estimated 5 percent to as much as 30 percent. I’ll revisit that later.

What are the Risks of Running without Patches?

Properly addressing Spectre and Meltdown will require more administrative effort than most other problems. Admins in large and complex environments will need to plan deployments. With unpredictable performance impacts, some will want to wait as long as possible before doing anything. The more avoidant of us will want to just ignore it as much as possible and hope that it just sorts itself out. Eventually, we’ll all have new chips, new operating systems, and redesigned applications that won’t be susceptible to these problems. That won’t be ubiquitous for some time, though. We need more immediate solutions.

In one sense, an unpatched system will be fairly easy to assault. Attacks can be carried out via simple Javascript. Unprivileged user code can access privileged kernel memory. The processor cache is an open book for many of these vulnerabilities. It looks to me as though some attack vectors could be used to read essentially any location in memory, although some other accounts dispute that. These are hardware problems, so in the hypervisor world, they transcend the normal boundaries between virtual machines as well as the management operating system. These threats are serious.

However, they cannot be exploited remotely. Attacking code must be executed directly on the target system. That separates the Spectre and Meltdown vulnerabilities from other nefarious vectors, such as Heartbleed. Furthermore, Spectre-class attacks are a bit of a gamble from the attacker’s side. Even if they could read any location in memory, it will be mostly without context. So, something may look like a password, but a password to what? The clues might be alongside the password or they might not.

So, the risk to your systems is extremely high, but the effort-to-reward ratio for an attacker is also high. I do know of one concern: LSASS keeps the passwords for all logged-on users in memory, in clear text. You should assume that a successful Spectre or Meltdown attack might be able to access those passwords. Also, attacks can come from Javascript in compromised websites. You shouldn’t be browsing from a server at all, so that could help. But what about remote desktop sessions? Do you keep those users off of the general Internet? If not, then any of them could unwittingly risk everyone else on the same host.

I cannot outline your risk profile for you, but I would counsel you to work from the assumption that a compromise of an unpatched system is inevitable.

Should I Patch My Hyper-V Hosts, Guests, or Both?

One of my primary research goals was to determine the effects of only applying OS patches, only applying firmware updates, and doing both. I did not find full, definitive answers. However, it does appear that some kernel protections only require OS updates. Others indicated that they would require firmware updates, but they did not make it clear whether or not a firmware update alone would address the problem or if a combination of firmware and OS updates were necessary. However, there is no question that the best protection only comes from updating your hardware and software.

Also, the guests’ kernels must be made aware of the changes to the physical layer in order to fully utilize their mitigation techniques, which can only be done if both the host and the guest are updated.

Therefore, for the fullest protection, you must:

  • Patch the host
  • Perform host configuration changes
  • Update the firmware on the host
  • Patch the guests
  • Cold boot the guests

Also, be aware that “host” means any Hyper-V host. Windows Server, Hyper-V Server, or desktop Windows — patch them all.

We’ll go over the how-to in a bit. Let’s knock out some other questions first.

What Happens if a Hyper-V VM Updates Before Its Host?

Well, the host isn’t patched, so that’s a problem. Aside from that, nothing bad will happen (generally speaking — outliers always exist). But the guest will not be aware of the changed capabilities of the host if it is not cold booted after the host is updated. Until all of the above has been done, the guest will not have the fullest protection.

What If No BIOS Update is Available?

If your hardware does not have an available update, then you still need to apply all software patches. That will mitigate several kernel attacks.

If you are running Windows/Hyper-V Server 2012 R2 or earlier as your management operating system, I’m sad to say that you cannot do much else. Practice defense-in-depth: if no attacker can access your systems, then they cannot execute an attack. Remember that the patches are still applicable and will help — but don’t lose sight of the hardware-based nature of the problems. If possible, upgrade to 2016.

For 2016, you can restrict which processors and cores that the management operating system and guests use. That helps to confine attacks. The relevant features are not well-documented. I had planned on performing some research and writing an article on them, but none of that has happened yet. In the meantime, you can read Microsoft’s article on using these features to mitigate Meltdown and Spectre attacks on systems that do not have an available hardware update. It contains links to instructions on using those features.

What About Non-OS Software?

Software plays an enormous role in all of this. Application developers can take precautions to protect their cached information. Interpreters and compilers can implement their own mitigations. The January updates included patches for IE and Edge to help block Spectre and Meltdown attacks via their Javascript interpreter. Microsoft has released an update to their C++ compiler that will help prevent Spectre attacks.

Expect your software vendors to be releasing updates. Remember that the nature of the attacks means that otherwise benign software can now be an attack vector.

Quality Control Problems with Spectre and Meltdown Patches

I am aware of three problems in the deployment pipeline for the Spectre and Meltdown fixes from Microsoft.

  1. Older AMD processors (Athlon series) reacted very badly to the initial patch group that contained the Spectre and Meltdown fix. In response, Microsoft stopped offering them while working on a fix. I have a Turion x2 system with Hyper-V that was offered the patch after that, so I believe that Microsoft has now issued a new fix.
  2. Many antivirus applications caused blue screen errors after the fixes were applied. To prevent that, Microsoft implemented a technique that will ensure that no system will receive the update until it has been deliberately flagged as being ready. Pay special attention, as that affects every system!
  3. Some systems simply will not take the update. I wish that I could give you some solid guidance on what to do. I do not know of any tips or tricks particular to these patches. I had one installation problem that I was able to rectify by disabling the “Give me updates for other Microsoft products when I update Windows” option. However, that particular system seemed to have problems of its own apart from the patches, so I cannot say for certain if it was anything special with this patch. You may need to open a ticket with Microsoft support.

How to Ensure that You Receive Microsoft’s Spectre and Meltdown Patches

These patches were deployed in the January 2018 security patch bundle. My system showed it as “2018-01 Security Monthly Rollup for OS Name” with a follow-up Quality Rollup. That deployment vehicle might lead you to believe that just running Windows Update will fix everything. However, that’s not necessarily the case. In order to avoid blue screens caused by incompatible antivirus applications, the updater looks for a particular registry key. If it doesn’t exist, then the updater assumes that the local antivirus is not prepared and will decline to offer the update. Since Microsoft now issues security patches in cumulative roll-up packages, it will decline every subsequent security update as well.

If Your System Runs Antivirus

In this case, “system” means host or guest. If you run an antivirus application, it is up to the antivirus program to update the registry key once it has received an update known to be compatible with the Spectre/Meltdown fixes. If you have an antivirus package and the updater will not offer the January 2018 security pack, contact your antivirus vendor. Do not attempt to force the update. Switch antivirus vendors if you must. Your antivirus situation needs to be squared away before you can move forward.

If Your System Does Not Run Antivirus

Personally, I prefer to run antivirus on my Hyper-V host. I recommend that everyone else run antivirus on their Hyper-V hosts. I’m aware that this is a debate topic and I’m aware of the counter-arguments. I’ll save that for another day.

If you do not run antivirus on your Hyper-V host (or any Windows system), then you will never receive the January 2018 security roll-up or any security roll-up thereafter without taking manual action. Microsoft has left the task of updating the related registry key to the antivirus vendors. No antivirus, no registry update. No registry update, no security patches.

In your system’s registry, navigate to HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindowsCurrentVersion. If it doesn’t already exist, create a new key named “QualityCompat”. Inside that key, create a new DWORD key-value pair named “cadca5fe-87d3-4b96-b7fb-a231484277cc”. Leave it at its default value of 0x0. You do not need to reboot. The updates will be offered on the next run of Windows Update (pursuant to any applicable group policy/WSUS approvals).

For more information, you can read Microsoft’s official documentation: https://support.microsoft.com/en-us/help/4072699/january-3-2018-windows-security-updates-and-antivirus-software.

How to Apply, Configure, and Verify the Spectre and Meltdown Fixes for Hyper-V and Windows

Microsoft has already published solid walk-throughs:

Things to absolutely not miss in all of that:

  • Without the post-install registry changes, your Hyper-V hosts will not have the strongest protection
  • Depending on your hypervisor and guest configurations, one of the additional registry settings may be necessary in order to
  • Each virtual machine must be COLD-booted after the host has been updated! A simple restart of the guest OS will not be sufficient.
  • Live Migrations between patched/unpatched hosts will likely fail. Keep this in mind when you’re attempting to update clusters.

The Performance Impact of the Spectre and Meltdown Patches on a Hyper-V Host

We’ve all seen the estimations that the Meltdown patch might affect performance in the range of 5 to 30 percent. What we haven’t seen is a reliable data set indicating what happens in a real-world environment. The patches are too new for us to have any meaningful levels of historical data to show. I can tell you that my usage trends are not showing any increase, but that is anecdotal information at best. I also know of another installation (using a radically different hardware, hypervisor, and software set) that has experienced crippling slowdowns.

So, for the time being, we’re going to need to take some benchmarks. For anyone who missed the not-so-subtle point: performance bottlenecks found by benchmarks do not automatically translate to performance bottlenecks in production.

Impact on Storage

From what I’ve seen so far, these patches have the greatest detriment on storage performance. Since storage already tends to be the slowest part of our systems, it makes sense that we’d find the greatest impact there.

First up is Storage Space Direct. Ben Thomas has already provided some benchmarks that show a marked slow down. Pay close attention to the differences, as they are substantial. However, most of us would be more than happy with the “slow” number of 1.3 million IOPS. Those numbers are difficult to read in comparison with builds that would be normative in a small or medium-sized business.

I also found an article detailing the storage performance impact on an Ubuntu system using ZFS in a VMware client. This helps reinforce the point that these problems are not caused by or related to your choice of hypervisor or operating system.

I don’t have an S2D environment of my own to do any comparison. However, I will perform some storage benchmarking on a standard Hyper-V build with local storage and remote Storage Spaces (not S2D) storage.

Performance Benchmarks

I ran a number of performance checks to try to help predict the impact of these updates.

The Build Environment and Methodology

I used the build outlined in our eBook on building a Hyper-V cluster for under $2500. To save you a click, the CPUs are Intel Xeon E3-1225 (Sandy Bridge). We should expect to see some of the more “significant” impacts that Microsoft and Intel warn about.

The tests were performed on one of the compute nodes. Some of the storage tests involved the storage system.

The host runs Windows Server 2016. It has multiple guests with multiple operating systems. I chose three representatives: 1 Windows 2012 R2, 1 Windows 10, and 1 Windows Server 2016. All run GUI Windows because Passmark won’t generate a full CPU report under Core.

None of these systems have the patch yet.

I took the following steps:

  1. Collected CPU and storage benchmarks from the host, a Windows 10 guest, the WS2012R2 guest, and the WS2016 guest. Tests were run separately.
  2. Ran a CPU stress utility on some guests while testing others.
  3. Ran a storage stress utility on some guests while testing others.
  4. I applied available firmware updates to the host and the OS patch to the management OS and all guests.
  5. Repeated steps 1-3 for comparison purposes.

I used these tools:

Testing Phase 1: The Pre-Patch Raw Results

To begin, I ran benchmarks against the host and some of the guests separately. This establishes a baseline.

Pre-Patch CPU Benchmark
Host Windows 10 Guest 2012 R2 Guest 2016 Guest
CPU Mark 7404 4198 4055 4197
Integer Math 8226 4209 4077 4212
Prime Numbers 33 24 21 24
Compression 8208 4084 4028 4103
Physics 492 303 295 315
CPU Single Threaded 1905 1882 1847 1886
Floating Point Math 7371 3738 3646 3522
Extended Instructions (SSE) 283 144 138 131
Encryption 1260 632 592 631
Sorting 5165 2590 2541 2608
Pre-Patch Disk Benchmark
Host (internal) Windows 10 Guest (Remote SS) 2012 R2 Guest (internal) 2016 Guest (Remote SS)
Disk Mark 416 505 355 629
Disk Sequential Read 55 124 44 153
Disk Random Seek +RW 6 4 6 4
Disk Sequential Write 53 11 47 15

Phase 1 Analysis

The primary purpose of phase 1 was to establish a performance baseline. The numbers themselves mean: “the systems can do this when not under load”.

The host has four physical cores. Each virtual machine has two vCPU. As expected, each VM’s CPU performance comes out to about half of the host’s metrics.

Testing Phase 2: Pre-Patch Under CPU Load

Next, I needed to have some idea of how a load affects performance. This system serves no real-world purpose, so I needed to fake a load.

Hyper-V is architected so that loads in the management operating system take priority over the guests (I didn’t always know that, in case you find something that I wrote to the contrary). Therefore, I didn’t feel that simulating a load in the management operating system would provide any value. It will drown out the guests and that’s that.

To test, I ran Prime95 on four virtual machines. I set the torture test to use small FFTs because we’re interested in CPU burn, not memory:

melt-prime95

For whatever reason. that only appeared to run the CPUs up to about 25% actual usage:

melt-CPUusage

The guests all believe that they’re burning full-bore, so this might be due to some optimizations. I thought the load would be higher, but we should not be overly surprised; hypervisors should do a decent job at managing their guests’ CPU usage. No matter what, this generates a load that the host distributed across all physical cores, so it works for what we’re trying to do.

While Prime95 ran in those four VMs, I tested each of the originals’ CPU and disk performance separately:

Pre-Patch CPU Benchmark during CPU Load
Host Windows 10 Guest 2012 R2 Guest 2016 Guest
CPU Mark not tested 1553 1339 1261
Integer Math not tested 1549 1356 1335
Prime Numbers not tested 7 5 5
Compression not tested 1570 1345 1308
Physics not tested 108 96 77
CPU Single Threaded not tested 753 646 564
Floating Point Math not tested 1385 1211 1109
Extended Instructions (SSE) not tested 52 45 44
Encryption not tested 232 205 201
Sorting not tested 985 841 817
Pre-Patch Disk Benchmark during CPU Load
Host (internal) Windows 10 Guest (Remote SS) 2012 R2 Guest (internal) 2016 Guest (Remote SS)
Disk Mark 416 349 385 481
Disk Sequential Read 55 90 51 111
Disk Random Seek +RW 6 3 6 4
Disk Sequential Write> 53 2 48 16

Testing Phase 3: Pre-Patch Under Disk Load

Phase 3 is like phase 2 except that we’re simulating heavy disk I/O load. It’s a little different though because the guests are spread out between local and remote storage.

I used DiskSpd.exe with the following command line:

That’s the default from the documentation with the exception of -d600 and #0. The -d600 sets the duration. I used a five-minute timeout so that I would have enough time to start the stress tool on all of the guests and run the CPU and disk benchmarks. The #0 selects a disk that the VMs have.

Pre-Patch CPU Benchmark during I/O Load
Host Windows 10 Guest 2012 R2 Guest 2016 Guest
CPU Mark not tested 3696 3814 3747
Integer Math not tested 3934 3927 3856
Prime Numbers not tested 13 18 16
Compression not tested 3715 3759 3838
Physics not tested 259 249 267
CPU Single Threaded not tested 1760 1852 1775
Floating Point Math not tested 3415 3476 3375
Extended Instructions (SSE) not tested 129 136 131
Encryption not tested 565 577 544
Sorting not tested 2399 2430 2375
Pre-Patch Disk Benchmark during I/O Load
Host (internal) Windows 10 Guest (Remote SS) 2012 R2 Guest (internal) 2016 Guest (Remote SS)
Disk Mark not tested 76 385 94
Disk Sequential Read not tested 12 57 17
Disk Random Seek +RW not tested 2 6 3
Disk Sequential Write> not tested 5 43 5

Testing Phase 4: Post-Patch Raw Results

Now we’re simply going to repeat the earlier processes on our newly-updated systems. Phase 4 aligns with Phase 1: a straight benchmark of the host and representative guests.

Post-Patch CPU Benchmark
Host Windows 10 Guest 2012 R2 Guest 2016 Guest
CPU Mark 7263 3989 3841 3705
Integer Math 8191 4052 3966 3827
Prime Numbers 34 21 17 18
Compression 8015 3890 3919 3336
Physics 489 303 301 273
CPU Single Threaded 1860 1712 1756 1747
Floating Point Math 7205 3464 3528 3281
Extended Instructions (SSE) 249 135 105 121
Encryption 1188 612 598 560
Sorting 5021 2469 2210 2448
Post-Patch Disk Benchmark
Host (internal) Windows 10 Guest (Remote SS) 2012 R2 Guest (internal) 2016 Guest (Remote SS)
Disk Mark 348 505 269 373
Disk Sequential Read 46 124 33 88
Disk Random Seek +RW 5 4 4 3
Disk Sequential Write> 44 11 36 10

Phase 4 Analysis

We don’t see that much CPU performance slowdown, but we do see some reduction in storage speeds.

Testing Phase 5: Post-Patch Under CPU Load

This test series compares against phase 2. Several VMs

Post-Patch CPU Benchmark during CPU Load
Host Windows 10 Guest 2012 R2 Guest 2016 Guest
CPU Mark not tested 1241 1267 1518
Integer Math not tested 1203 1288 1525
Prime Numbers not tested 5 5 7
Compression not tested 1280 1253 1480
Physics not tested 91 94 116
CPU Single Threaded not tested 603 620 700
Floating Point Math not tested 1087 1118 1325
Extended Instructions (SSE) not tested 43 44 45
Encryption not tested 192 192 239
Sorting not tested 811 802 941
Post-Patch Disk Benchmark during CPU Load
Host (internal) Windows 10 Guest (Remote SS) 2012 R2 Guest (internal) 2016 Guest (Remote SS)
Disk Mark 416 349 325 366
Disk Sequential Read 55 90 41 85
Disk Random Seek +RW 6 3 6 4
Disk Sequential Write> 53 2 42 12

Testing Phase 6: Post-Patch Under Disk Load

This is the post-patch equivalent of phase 3. We’ll test performance while simulating high disk I/O.

Again, I used DiskSpd.exe on several VMs with the following command line:

Post-Patch CPU Benchmark during I/O Load
Host Windows 10 Guest 2012 R2 Guest 2016 Guest
CPU Mark not tested 3152 3104 2920
Integer Math not tested 3192 3268 3106
Prime Numbers not tested 14 14 13
Compression not tested 3163 2654 2361
Physics not tested 215 219 211
CPU Single Threaded not tested 1667 1774 1614
Floating Point Math not tested 2678 2885 2672
Extended Instructions (SSE) not tested 105 95 100
Encryption not tested 476 452 449
Sorting not tested 2015 2086 1999
Post-Patch Disk Benchmark during I/O Load
Host (internal) Windows 10 Guest (Remote SS) 2012 R2 Guest (internal) 2016 Guest (Remote SS)
Disk Mark not tested 40 360 54
Disk Sequential Read not tested 5 50 7
Disk Random Seek +RW not tested 1 5 3
Disk Sequential Write not tested 3 44 4

Consolidated Charts

These charts reorganize the above data to align each result with its system rather than its condition.

Host CPU
Pre Post
CPU Mark 7404 7263
Integer Math 8226 8191
Prime Numbers 33 34
Compression 8208 8015
Physics 492 489
CPU Single Threaded 1905 1860
Floating Point Math 7371 7205
Extended Instructions (SSE) 283 249
Encryption 1260 1188
Sorting 5165 5021
Host Disk (Internal)
Pre Post
Disk Mark 416 348
Disk Sequential Read 55 46
Disk Random Seek +RW 6 5
Disk Sequential Write 53 44
Windows 10 Guest CPU
Pre Pre+CPU Pre+I/O Post Post+CPU Post+I/O
CPU Mark 4198 1553 3696 3989 1241 3152
Integer Math 4209 1549 3934 4052 1203 3192
Prime Numbers 24 7 13 21 5 14
Compression 4084 1570 3715 3890 1280 3163
Physics 303 108 259 303 91 215
CPU Single Threaded 1882 753 1760 1712 603 1667
Floating Point Math 3738 1385 3415 3464 1087 2678
Extended Instructions (SSE) 144 52 129 135 43 105
Encryption 632 232 565 612 192 476
Sorting 2590 985 2399 2469 811 2015
Windows 10 Guest Disk (Remote Storage Spaces)
Pre Pre+CPU Pre+I/O Post Post+CPU Post+I/O
Disk Mark 505 349 76 505 349 40
Disk Sequential Read 124 90 12 124 90 5
Disk Random Seek +RW 4 3 2 4 3 1
Disk Sequential Write 11 2 5 11 2 3
Windows 2012 R2 Guest CPU
Pre Pre+CPU Pre+I/O Post Post+CPU Post+I/O
CPU Mark 4055 1339 3814 3841 1267 3104
Integer Math 4077 1356 3927 3966 1288 3268
Prime Numbers 21 5 18 17 5 14
Compression 4028 1345 3759 3919 1253 2654
Physics 295 96 249 301 94 219
CPU Single Threaded 1847 646 1852 1756 620 1774
Floating Point Math 3646 1211 3476 3528 1118 2885
Extended Instructions (SSE) 138 45 136 105 44 95
Encryption 592 205 577 598 192 452
Sorting 2541 841 2430 2210 802 2086
Windows 2012 R2 Guest Disk (Internal)
Pre Pre+CPU Pre+I/O Post Post+CPU Post+I/O
Disk Mark 355 385 385 269 325 360
Disk Sequential Read 44 51 57 33 41 50
Disk Random Seek +RW 6 6 6 4 6 5
Disk Sequential Write 47 48 43 36 42 44
Windows 2016 Guest CPU
Pre Pre+CPU Pre+I/O Post Post+CPU Post+I/O
CPU Mark 4197 1261 3747 3705 1518 2920
Integer Math 4212 1335 3856 3827 1525 3106
Prime Numbers 24 5 16 18 7 13
Compression 4103 1308 3838 3336 1480 2361
Physics 315 77 267 273 116 211
CPU Single Threaded 1886 564 1775 1747 700 1614
Floating Point Math 3522 1109 3375 3281 1325 2672
Extended Instructions (SSE) 131 44 131 121 45 100
Encryption 631 201 544 560 239 449
Sorting 2608 817 2375 2448 941 1999
Windows 2016 Guest Disk (Remote Storage Spaces)
Pre Pre+CPU Pre+I/O Post Post+CPU Post+I/O
Disk Mark 629 481 94 373 366 54
Disk Sequential Read 153 111 17 88 85 7
Disk Random Seek +RW 4 4 3 3 4 3
Disk Sequential Write 15 16 5 10 12 4

Data Analysis

The analysis has a fairly simple conclusion: some CPU functions degraded noticeably, others did not, and not much changed for I/O. The Windows 2016 guest does have a wide disparity in its disk statistics. Based on the other outcomes, I suspect an environmental difference or that I made some error when running the traces more than I believe that the raw speed degraded by 50%.

Conclusion

What we learn from these charts is that the full range of updates causes CPU performance degradation for some functions but leaves others unaffected. Normal storage I/O does not suffer as much. I imagine that the higher impacts that you see in the Storages Spaces Direct benchmarks shared by Ben Thomas are because of their extremely high speed. Since most of us don’t have storage subsystems that can challenge a normal CPU, we aren’t affected as strongly.

Overall, I think that the results show what could be reasonably inferred without benchmarks: high utilization systems will see a greater impact. As far as the exact impact on your systems, that appears to largely depend on what CPU functions they require most. If your pre-patch conditions were not demanding high performance, I suspect that you’ll suffer very little. If you have higher demand workloads, I’d recommend that you try a test environment if possible. If not, see if your software vendors have any knowledge or strategies for keeping their applications running acceptably.

Tell Me Your Experiences

I’ve given a breakdown of the benchmarking from my set-up but I’m curious to know if others have experienced a similar performance impact from the patches. Furthermore, if you still have any lingering doubts about the whole issue I’m ready to answer any questions you have about Meltdown, Spectre or anything else related to these security vulnerabilities and their respective patches. Contact me through the comments section below!

Meltdown and Spectre fixes eyed for SQL Server performance issues

The headlines about Meltdown and Spectre microprocessor vulnerabilities have somewhat subsided, but the patching goes on in IT shops both big and small — creating possible SQL Server performance issues for Microsoft users.

Databases, including SQL Server, could be affected by the architectural flaws in chips widely reported at the start of the year. At risk are processors from Intel and others that use creative design to boost system performance.

As described by Google Project Zero, both Meltdown and Spectre access on-chip cache memory to create vulnerable side-channels of communication. Spectre can inject commands that divulge data. Meltdown, using simpler operations, can monitor data in memory.

In both cases, malicious code would exploit chip-level speculative code execution techniques — ones used in many types of systems, including relational databases.

Early indications are software patches that counter Meltdown can incur SQL Server performance issues. There are no signs of actual hacks yet, but database administrators (DBAs) have been advised to update server-side software. The patches may lead to processing overhead, however.

Problems are by no means limited to on-premises databases. Reports indicated some users of SQL Server cloud versions were first to feel the impact of Meltdown protection, when Microsoft Azure cloud patching activity caused brief blips in operations.

Patches and overhead

Spectre logo

When DBAs apply updates to guard against Meltdown and Spectre vulnerabilities, they will have to judge for themselves how performance overhead of patches may affect their workloads. Performance degradation covering a variety of database, virtual machine, operating system and hardware combinations has been cited in some user web blog entries.

But early estimates should be considered critically, according to Thomas LaRock, who serves as “head geek” at technology infrastructure management software provider SolarWinds, based in Austin, Texas. It is still early in terms of finding clarity when it comes to judging SQL Server performance issues that may be incurred by recent Microsoft patches to counter Meltdown.

“When you factor in the number of patches involved with Meltdown [and] Spectre, it is easy to understand why some people may be reporting a 30% performance hit. You could find hundreds of such claims on Reddit right now, many of them without any understanding of why such a performance hit might have been possible,” LaRock said.

Calling ‘Captain Edgecase’

Meltdown logo

Workloads, hardware, applications and code are among the variables that contribute to different performance measures. This is not to mention the human element.

“There is always one ‘Captain Edgecase’ in the crowd that wants everyone to know they found something different than anyone else,” LaRock mused.

While waiting for chipmakers to create their own patches for Meltdown and Spectre, IT pros will have to look to patch applications not just at the database level, but at the operating system and browser level, too.

La Rock said the basic message boils down to this: “Update all things.”

Meltdown and Spectre lesson: Assess patch risk

Still, it is important for everyone to be able to assess their risk properly before applying patches, according to LaRock, who is a Microsoft MVP.

“To me, any risk is too much risk, and I would want to patch. But I wouldn’t do so without knowing the impact, especially for mission-critical servers,” he said.

The advice here is to, in LaRock’s words, “test, test, test.” Ned Bellavance, who is director of cloud solutions at Anexinet in Blue Bell, Pa., and is also a Microsoft MVP, agreed.

If you didn’t follow best practices for high availability, you may have had a performance hit.
Ned Bellavancedirector of cloud solutions at Anexinet

“There are always going to be vulnerabilities and news about vulnerabilities. So, you have to have an environment for testing patches before rolling out changes into production,” he said.

Bellavance said moving databases to the cloud does not relieve an admin’s responsibility for faults in system settings. That is especially true in the face of vulnerabilities like Meltdown, which can exploit cloud environments that share resources like databases across virtual machines. The cloud provider can be expected to roll out fixes, but databases have to be configured to anticipate such disruptions in the status quo.

Microsoft was quick to roll out patches for SQL Server databases on its cloud, but some users experienced downtime, Bellavance said.

“People were impacted because Microsoft had to do cloud maintenance. If you didn’t follow best practices for high availability, you may have had a performance hit,” he said.

Now is as good a time as any to review such practices, Bellavance advised.