Tag Archives: vulnerability

Hyper-V HyperClear Mitigation for L1 Terminal Fault

Introduction

A new speculative execution side channel vulnerability was announced recently that affects a range of Intel Core and Intel Xeon processors. This vulnerability, referred to as L1 Terminal Fault (L1TF) and assigned CVE 2018-3646 for hypervisors, can be used for a range of attacks across isolation boundaries, including intra-OS attacks from user-mode to kernel-mode as well as inter-VM attacks. Due to the nature of this vulnerability, creating a robust, inter-VM mitigation that doesn’t significantly degrade performance is particularly challenging.

For Hyper-V, we have developed a comprehensive mitigation to this attack that we call HyperClear. This mitigation is in-use by Microsoft Azure and is available in Windows Server 2016 and later. The HyperClear mitigation continues to allow for safe use of SMT (hyper-threading) with VMs and, based on our observations of deploying this mitigation in Microsoft Azure, HyperClear has shown to have relatively negligible performance impact.

We have already shared the details of HyperClear with industry partners. Since we have received questions as to how we are able to mitigate the L1TF vulnerability without compromising performance, we wanted to broadly share a technical overview of the HyperClear mitigation and how it mitigates L1TF speculative execution side channel attacks across VMs.

Overview of L1TF Impact to VM Isolation

As documented here, the fundamental premise of the L1TF vulnerability is that it allows a virtual machine running on a processor core to observe any data in the L1 data cache on that core.

Normally, the Hyper-V hypervisor isolates what data a virtual machine can access by leveraging the memory address translation capabilities provided by the processor. In the case of Intel processors, the Extended Page Tables (EPT) feature of Intel VT-x is used to restrict the system physical memory addresses that a virtual machine can access.

Under normal execution, the hypervisor leverages the EPT feature to restrict what physical memory can be accessed by a VM’s virtual processor while it is running. This also restricts what data the virtual processor can access in the cache, as the physical processor enforces that a virtual processor can only access data in the cache corresponding to system physical addresses made accessible via the virtual processor’s EPT configuration.

By successfully exploiting the L1TF vulnerability, the EPT configuration for a virtual processor can be bypassed during the speculative execution associated with this vulnerability. This means that a virtual processor in a VM can speculatively access anything in the L1 data cache, regardless of the memory protections configured by the processor’s EPT configuration.

Intel’s Hyper-Threading (HT) technology is a form of Simultaneous MultiThreading (SMT). With SMT, a core has multiple SMT threads (also known as logical processors), and these logical processors (LPs) can execute simultaneously on a core. SMT further complicates this vulnerability, as the L1 data cache is shared between sibling SMT threads of the same core. Thus, a virtual processor for a VM running on a SMT thread can speculatively access anything brought into the L1 data cache by its sibling SMT threads. This can make it inherently unsafe to run multiple isolation contexts on the same core. For example, if one logical processor of a SMT core is running a virtual processor from VM A and another logical processor of the core is running a virtual processor from VM B, sensitive data from VM B could be seen by VM A (and vice-versa).

Similarly, if one logical processor of a SMT core is running a virtual processor for a VM and the other logical processor of the SMT core is running in the hypervisor context, the guest VM could speculatively access sensitive data brought into the cache by the hypervisor.

Basic Inter-VM Mitigation

To mitigate the L1TF vulnerability in the context of inter-VM isolation, the most straightforward mitigation involves two key components:

  1. Flush L1 Data Cache On Guest VM Entry – Every time the hypervisor switches a processor thread (logical processor) to execute in the context of a guest virtual processor, the hypervisor can first flush the L1 data cache. This ensures that no sensitive data from the hypervisor or previously running guest virtual processors remains in the cache. To enable the hypervisor to flush the L1 data cache, Intel has released updated microcode that provides an architectural facility for flushing the L1 data cache.
  2. Disable SMT – Even with flushing the L1 data cache on guest VM entry, there is still the risk that a sibling SMT thread can bring sensitive data into the cache from a different security context. To mitigate this, SMT can be disabled, which ensures that only one thread ever executes on a processor core.

The L1TF mitigation for Hyper-V prior to Windows Server 2016 employs a mitigation based on these components. However, this basic mitigation has the major downside that SMT must be disabled, which can significantly reduce the overall performance of a system. Furthermore, this mitigation can result in a very high rate of L1 data cache flushes since the hypervisor may switch a thread between the guest and hypervisor contexts many thousands of times a second. These frequent cache flushes can also degrade the performance of the system.

HyperClear Inter-VM Mitigation

To address the downsides of the basic L1TF Inter-VM mitigation, we developed the HyperClear mitigation. The HyperClear mitigation relies on three key components to ensure strong Inter-VM isolation:

  1. Core Scheduler
  2. Virtual-Processor Address Space Isolation
  3. Sensitive Data Scrubbing

Core Scheduler

The traditional Hyper-V scheduler operates at the level of individual SMT threads (logical processors). When making scheduling decisions, the Hyper-V scheduler would schedule a virtual processor onto a SMT thread, without regards to what the sibling SMT threads of the same core were doing. Thus, a single physical core could be running virtual processors from different VMs simultaneously.

Starting in Windows Server 2016, Hyper-V introduced a new scheduler implementation for SMT systems known as the “Core Scheduler“. When the Core Scheduler is enabled, Hyper-V schedules virtual cores onto physical cores. Thus, when a virtual core for a VM is scheduled, it gets exclusive use of a physical core, and a VM will never share a physical core with another VM.

With the Core Scheduler, a VM can safely take advantage of SMT (Hyper-Threading). When a VM is using SMT, the hypervisor scheduling allows the VM to use all the SMT threads of a core at the same time.

Thus, the Core Scheduler provides the essential protection that a VM’s data won’t be directly disclosed across sibling SMT threads. It protects against cross-thread data exposure of a VM since two different VMs never run simultaneously on different threads of the same core.

However, the Core Scheduler alone is not sufficient to protect against all forms of sensitive data leakage across SMT threads. There is still the risk that hypervisor data could be leaked across sibling SMT threads.

Virtual-Processor Address Space Isolation

SMT Threads on a core can independently enter and exit the hypervisor context based on their activity. For example, events like interrupts can cause a SMT thread to switch out of running the guest virtual processor context and begin executing the hypervisor context. This can happen independently for each SMT thread, so one SMT thread may be executing in the hypervisor context while its sibling SMT thread is still running a VM’s guest virtual processor context. An attacker running code in the less trusted guest VM virtual processor context on one SMT thread can then use the L1TF side channel vulnerability to potentially observe sensitive data from the hypervisor context running on the sibling SMT thread.

One potential mitigation to this problem is to coordinate hypervisor entry and exit across SMT threads of the same core. While this is effective in mitigating the information disclosure risk, this can significantly degrade performance.

Instead of coordinating hypervisor entry and exits across SMT threads, Hyper-V employs strong data isolation in the hypervisor to protect against a malicious guest VM leveraging the L1TF vulnerability to observe sensitive hypervisor data. The Hyper-V hypervisor achieves this isolation by maintaining separate virtual address spaces in the hypervisor for each guest SMT thread (virtual processor). When the hypervisor context is entered on a specific SMT thread, the only data that is addressable by the hypervisor is data associated with the guest virtual processor associated with that SMT thread. This is enforced through the hypervisor’s page table selectively mapping only the memory associated with the guest virtual processor. No data for any other guest virtual processor is addressable, and thus, the only data that can be brought into the L1 data cache by the hypervisor is data associated with that current guest virtual processor.

Thus, regardless of whether a given virtual processor is running in the guest VM virtual processor context or in the hypervisor context, the only data that can be brought into the cache is data associated with the active guest virtual processor. No additional privileged hypervisor secrets or data from other guest virtual processors can be brought into the L1 data cache.

This strong address space isolation provides two distinct benefits:

  1. The hypervisor does not need to coordinate entry and exits into the hypervisor across sibling SMT threads. So, SMT threads can enter and exit the hypervisor context independently without any additional performance overhead.
  2. The hypervisor does not need to flush the L1 data cache when entering the guest VP context from the hypervisor context. Since the only data that can be brought into the cache while executing in the hypervisor context is data associated with the guest virtual processor, there is no risk of privileged/private state in the cache that needs to be protected from the guest. Thus, with this strong address space isolation, the hypervisor only needs to flush the L1 data cache when switching between virtual cores on a physical core. This is much less frequent than the switches between the hypervisor and guest VP contexts.

Sensitive Data Scrubbing

There are cases where virtual processor address space isolation is insufficient to ensure isolation of sensitive data. Specifically, in the case of nested virtualization, a single virtual processor may itself run multiple guest virtual processors. Consider the case of a L1 guest VM running a nested hypervisor (L1 hypervisor). In this case, a virtual processor in this L1 guest may be used to run nested virtual processors for L2 VMs being managed by the L1 nested hypervisor.

In this case, the nested L1 guest hypervisor will be context switching between each of these nested L2 guests (VM A and VM B) and the nested L1 guest hypervisor. Thus, a virtual processor for the L1 VM being maintained by the L0 hypervisor can run multiple different security domains – a nested L1 hypervisor context and one or more L2 guest virtual machine contexts. Since the L0 hypervisor maintains a single address space for the L1 VM’s virtual processor, this address space could contain data for the nested L1 guest hypervisor and L2 guests VMs.

To ensure a strong isolation boundary between these different security domains, the L0 hypervisor relies on a technique we refer to as state scrubbing when nested virtualization is in-use. With state scrubbing, the L0 hypervisor will avoid caching any sensitive guest state in its data structures. If the L0 hypervisor must read guest data, like register contents, into its private memory to complete an operation, the L0 hypervisor will overwrite this memory with 0’s prior to exiting the L0 hypervisor context. This ensures that any sensitive L1 guest hypervisor or L2 guest virtual processor state is not resident in the cache when switching between security domains in the L1 guest VM.

For example, if the L1 guest hypervisor accesses an I/O port that is emulated by the L0 hypervisor, the L0 hypervisor context will become active. To properly emulate the I/O port access, the L0 hypervisor will have to read the current guest register contents for the L1 guest hypervisor context, and these register contents will be copied to internal L0 hypervisor memory. When the L0 hypervisor has completed emulation of the I/O port access, the L0 hypervisor will overwrite any L0 hypervisor memory that contains register contents for the L1 guest hypervisor context. After clearing out its internal memory, the L0 hypervisor will resume the L1 guest hypervisor context. This ensures that no sensitive data stays in the L0 hypervisor’s internal memory across invocations of the L0 hypervisor context. Thus, in the above example, there will not be any sensitive L1 guest hypervisor state in the L0 hypervisor’s private memory. This mitigates the risk that sensitive L1 guest hypervisor state will be brought into the data cache the next time the L0 hypervisor context becomes active.

As described above, this state scrubbing model does involve some extra processing when nested virtualization is in-use. To minimize this processing, the L0 hypervisor is very careful in tracking when it needs to scrub its memory, so it can do this with minimal overhead. The overhead of this extra processing is negligible in the nested virtualization scenarios we have measured.

Finally, the L0 hypervisor state scrubbing ensures that the L0 hypervisor can efficiently and safely provide nested virtualization to L1 guest virtual machines. However, to fully mitigate inter-VM attacks between L2 guest virtual machines, the nested L1 guest hypervisor must implement a mitigation for the L1TF vulnerability. This means the L1 guest hypervisor needs to appropriately manage the L1 data cache to ensure isolation of sensitive data across the L2 guest virtual machine security boundaries. The Hyper-V L0 hypervisor exposes the appropriate capabilities to L1 guest hypervisors to allow L1 guest hypervisors to perform L1 data cache flushes.

Conclusion

By using a combination of core scheduling, address space isolation, and data clearing, Hyper-V HyperClear is able to mitigate the L1TF speculative execution side channel attack across VMs with negligible performance impact and with full support of SMT.

Disclose.io launches vulnerability disclosure ‘safe harbor’

Disclose.io is a new project that promotes a framework for the standardization of norms for vulnerability disclosure with the intent to remove the threat of criminal or civil prosecution of cybersecurity researchers, a long-standing obstacle to more open research and sharing of vulnerabilities by independent experts.

Describing itself as “a collaborative and vendor-agnostic project to standardize best practices around safe harbor for good faith security research,” Disclose.io was jointly announced this week by bug bounty company Bugcrowd and Amit Elazari, a University of California, Berkeley, doctoral candidate and bug bounty legal expert. The project addresses the lack of consistency in policies on vulnerability  disclosure, the need to keep researchers safe from legal action by companies with vulnerabilities and a framework to provide researchers with a “safe harbor” from prosecution under the Computer Fraud and Abuse Act (CFAA) or the Digital Millennium Copyright Act (DMCA).

So far, Elazari has listed 21 organizations — up from 18 when the announcement was first made — that have adopted language in their bug bounty programs that follow Department of Justice guidelines for protecting bug bounty participants from prosecution under the CFAA and that also address DMCA issues.

“With growing attention to this issue and increasing adoption of bug bounties in general, as well as the emergence of best practices, I hope adoption within big players will rise,” Elazari wrote by email. “Hackers are also becoming more aware to this issue and with time safe harbor will hopefully become a competitive feature of the program — a way to get more professional eyeballs on your code. This trend will continue as long as the law continues to be murky — and that is the case especially with the CFAA.”

The Disclose.io framework builds on the Open Source Vulnerability Disclosure Framework from Bugcrowd and tech-focused law firm CipherLaw, as well as Elazari’s own #legalbugbounty standardization project, both of which provide guidance on ways to keep participants safe from prosecution under the CFAA or the DMCA for companies setting up their own vulnerability disclosure programs.

Organizations that have adopted safe harbor terms in their bug bounty or vulnerability disclosure programs include Bugcrowd, as well as Dropbox, HackerOne and Mozilla.

Risks of vulnerability disclosure

The Disclose.io project comes from the intent to protect both cybersecurity researchers from the risk of legal proceedings as a result of them disclosing vulnerabilities, as well as to protect program owners from individuals who discover vulnerabilities and act in bad faith; for example, some individuals may have ulterior motives and use bug bounty programs to gain unauthorized access to the program owner’s resources.

Amit Elazari, doctoral candidate at University of California, Berkeley, and bug bounty legal expertAmit Elazari

However, some organizations attempt to shift some of the risks of bug bounty hunting to the bug hunters, especially when bug bounty participants are not explicitly granted full authorization to all relevant assets.

“Not providing authorization is shifting the legal risk to the hacker. Since these are take-it-or-leave-it contracts, lawyers might be inclined to protect their own organization interests. The main practical barrier for adoption of safe harbor is it actually requires obtaining the rights to authorization in all assets and careful scoping and policy drafting,” Elazari wrote. “When you are authorizing access you are clarifying that one must follow the guidelines to get it, and that’s why it works well for both parties because it signals to the hacker what are the rules. If you intentionally violate the rules — you don’t get the safe harbor.”

In other news

  • Facebook security chief Alex Stamos is leaving the social networking giant and starting a research and teaching role as an adjunct professor at Stanford University’s Freeman-Spogli Institute for International Studies (FSI). His last day at Facebook is Aug. 17, almost precisely five months after The New York Times reported that his “impending exit” was set for August. Prior to his stint at Facebook, Stamos was CISO at Yahoo. In a message posted on his Facebook page, Stamos wrote that he would be continuing his work on “understanding and preventing the misuse of technology,” and would be launching “a course teaching hands-on offensive and defensive techniques and to contribute to the new cybersecurity master’s specialty” at FSI.
  • Congress passed a bill this week that will force tech companies to disclose to the Pentagon if they have allowed foreign governments to examine their software if it was sold to the U.S. military. The legislation was included in the Pentagon spending bill, which was approved by an 87-to-10 vote in the Senate after having passed in the House of Representatives last week; President Trump is expected to sign the bill into law. The new law was drafted after an investigation by Reuters discovered that companies, including Hewlett Packard, SAP and McAfee had allowed Russian agencies to examine their software products as a precondition for sale in Russia. The legislation, included in the fiscal 2019 National Defense Authorization Act, was drafted by Senator Jeanne Shaheen (D-NH), who told Reuters that the new rules would help secure the government’s technology acquisition process.
  • The CA/Browser Forum has changed its rules for how certificate authorities (CAs) are allowed to validate claims of domain ownership for issuance of trusted certificates as of Aug. 1, removing two methods of validation that have been exploited by malicious actors seeking legitimacy through domain certificates. The CA/B Forum Baseline Requirements no longer permit CAs to use the first validation method, which compared the domain certificate applicant’s contact information with domain contact information listed on domain name registrar databases. Until now, CAs could validate an applicant’s contact information with domain contact information returned by a “whois” query to the domain registrar. Also deprecated was the fifth validation method, which “allowed lawyers to write letters asserting ownership of domain names, a subject they are generally not qualified to evaluate,” wrote Timothy Hollebeek, industry and standards technical strategist at DigiCert, in a blog post announcing the move. “Neither of these methods were particularly secure, and we led the effort to get them removed, as part of an overall focus on improving validation standards.”

Adobe zero-day fix precedes June Patch Tuesday

An Adobe zero-day vulnerability in Flash Player that was actively exploited stirred up excitement for admins in the week leading up to June Patch Tuesday.

Adobe released a fix for the zero-day (CVE-2018-5002) and three other vulnerabilities for the Windows client operating system on June 7.

The zero-day exploit launched its attacks from Excel documents sent via email. Users who open these infected Excel attachments on unpatched systems could allow the execution of arbitrary code under the exploited user account.

Chris Goettl, director of product management, IvantiChris Goettl

After the Adobe zero-day issue, the patching workload for administrators is lighter than usual for June Patch Tuesday, with about 50 unique vulnerabilities to correct — including 11 rated critical.

“Our recommendation is the Flash patch — if it already hasn’t been pushed out, [give that] high priority,” said Chris Goettl, director of product management at Ivanti, based in South Jordan, Utah.

June Patch Tuesday closes about 50 vulnerabilities

Microsoft released an update for the only publicly disclosed vulnerability (CVE-2018-8267) for June Patch Tuesday, which affects the Microsoft scripting engine on all supported versions of Internet Explorer. Attacks can exploit this flaw through a compromised website, or user-contributed ads or content, to take control of the target machine.

On an unpatched system, attackers could execute arbitrary code as the hacked user. Organizations that follow least-privilege rules that restrict the use of higher full permissions will reduce the damage from a breach.

Jimmy Graham, director of product management at QualysJimmy Graham

Microsoft’s June Patch Tuesday fixes also closed a remote code execution vulnerability (CVE-2018-8225) that affects all supported versions of Windows. This vulnerability could allow an attacker to compromise systems through a domain name system (DNS) server.

“That would be higher risk for mobile workstations, where it’s likely the system will be accessing an untrusted DNS server through public Wi-Fi,” said Jimmy Graham, director of product management at Qualys, based in Redwood City, Calif.

A memory corruption vulnerability (CVE-2018-8229) in the Edge browser’s Chakra scripting engine would let an attacker exploit an unpatched system through specially crafted websites or user-provided content. The effects depend on the level of privilege on the system.

Spectre vulnerabilities continue

Just when it seemed the Meltdown and Spectre vulnerabilities were winding down, security researchers uncovered another CPU bug. The vulnerability, called Spectre variant 4, is similar to the other speculative execution side-channel vulnerabilities disclosed in January, but they are rated with moderate severity.  

Jann Horn, a security researcher at Google’s Project Zero, and Ken Johnson, of the Microsoft Security Response Center, discovered Spectre variant 4 (CVE-2018-3639). This exploit enables malicious actors to read privileged data across trust boundaries.

Microsoft released its ADV180012 advisory in January to assist administrators with closing the exploits from the speculative execution side-channel vulnerabilities. The company continues to update the site, and it added further mitigation instructions to address Spectre variant 4. There are still no active attacks on Meltdown or Spectre, but administrators should install the patches and microcode updates when the CPU manufacturers release them.

For more information about the remaining security bulletins for June Patch Tuesday, visit Microsoft’s Security Update Guide.

Electron framework flaw puts popular desktop apps at risk

A new vulnerability found in an app development tool has caused popular desktop apps made with the tool to inherit a risky flaw.

The Electron framework uses node.js and Chromium to build desktop apps for popular web services — including Slack, Skype, WordPress.com, Twitch, GitHub, and many more — while using web code like JavaScript, HTML and CSS. Electron announced that a remote code execution vulnerability in the Electron framework (CVE-

2018-1000006
) was inherited by an unknown number of apps.

Zeke Sikelianos, a designer and developer who works at Electron, wrote in a blog post that only apps built for “Windows that register themselves as the default handler for a protocol … are vulnerable,” while apps for macOS and Linux are not at risk.

Amit Serper, principal security researcher at Cybereason, said a flaw like the one found in the Electron framework “is pretty dangerous since it allows arbitrary command execution by a simple social engineering trick.”

A flaw like this is pretty dangerous since it allows arbitrary command execution by a simple social engineering trick.
Amit Serperprincipal security researcher at Cybereason

“Electron apps have the ability to register a protocol handler to make it easier to automate processes for the Electron apps themselves (for example, if you’ll click a link that starts with slack:// then Slack will launch. It makes it easier to automate the process of joining a Slack group,” Serper told SearchSecurity by email. “The vulnerability is in the way that the protocol handler is being processed by the Electron app, which allows an attacker to create a malicious link to an Electron app which will execute whatever command that the attacker wanted to run.”

Sikelianos urged developers to update apps to the most recent version of Electron as soon as possible.

There are more than 460 apps that have been built using the flawed Electron framework, but it is unclear how many of those apps are at risk and experts noted that code reviews could take a while.  

Security audits

Lane Thames, senior security researcher at Tripwire, said mechanisms for code reuse like software libraries, open source code, and the Electron framework “are some of the best things going for modern software development. However, they are also some of its worst enemies in terms of security.”

“Anytime a code base is in use across many products, havoc will ensue when (not if) a vulnerability is discovered. This is inevitable. Therefore, developers should ensure that mechanisms are in place for updating downstream applications that are impacted by the vulnerabilities in the upstream components,” Thames told SearchSecurity. “This is not an easy task and requires lots of coordination between various stakeholders. In a perfect world, code that gets used by many other projects should undergo security assessments with every release. Implementing a secure coding practice where every commit is evaluated at least with a security-focused code review would be even better.”

Serper said developers need to “always audit their code and be mindful to security.”

“However, in today’s software engineering ecosystem, where there is a lot of use of third party libraries it is very hard to audit the code that you are using since many developers today use modules and code that was written by other people, completely unrelated to their own project,” Serper said. “These are vast amounts of code and auditing third party code in addition to auditing your own code could take a lot of time.”

Justin Jett, director of audit and compliance at Plixer International Inc., a network analysis company based in Kennebunk, Maine, said the Electron framework flaw was significant, given that “affected applications like Skype, Slack, and WordPress are used by organizations to host and share their most critical information”

“If these applications were to be compromised, the impact could be devastating. Developers that use third-party frameworks, like Electron, should audit their code on a regular basis, ideally quarterly, to ensure they are using an up-to-date version of the framework that works with their application and has resolved any security issues from previous releases,” Jett told SearchSecurity. “Additionally, platform developers, like Electron, should complete routine audits on their software to ensure that the developers taking advantage of their platform don’t expose users to security vulnerabilities — vulnerabilities which, left unresolved, could cause profound damage to businesses that rely on these applications.”

Chip bugs hit cloud computing usage less than first feared

In the aftermath of one of the largest compute vulnerability disclosures in years, it turns out that cloud computing usage won’t suffer greatly after all.

Public clouds were potentially among the most imperiled architectures from the Spectre and Meltdown chip vulnerabilities. But at least from the initial patches, the impact to these platforms’ security and performance appears to be less dire than predicted.

Many industry observers expressed concern that these chip-level vulnerabilities would make the multitenant cloud model a conspicuous target for hackers to gain access to data in other users’ accounts on the same shared host. But major cloud vendors’ quick responses – in some cases months ago — have largely addressed those issues.

Customers must still update systems that live on top of the cloud, but with the underlying patches, cloud environments are well-positioned to address the initial concerns about data theft. And cloud customers have far less to do than a company that owns its own data center and needs to update its hardware, microcode, hypervisor and perhaps management instances.

“The sky is not falling; just relax,” said Chris Gardner, an analyst with Forrester Research. “They’re probably the most critical CPU bugs we’ve seen in quite some time, but the mitigations help and the chip manufacturers are already working on long-term solutions.”

In some ways, vendors’ rapid response to install fixes to the Meltdown and Spectre vulnerabilities also illustrates their centralization and automation chops.

“We couldn’t have worked with hardware vendors and open source projects like Linux at the pace they were able to do to patch project,” said Joe Kinsella, CTO and founder of CloudHealth, a cloud managed service provider in Boston. “The end result is a testament to the centralization of ability to actually go and respond.”

Security experts say there are no known exploits in the wild for the Meltdown and the two-pronged Spectre vulnerabilities. The execution of a hack through these vulnerabilities, especially Spectre, is beyond the scope of the average hacker, who is far more likely to find a path of less resistance, they say.

In fact, the real impact from Meltdown and Spectre vulnerabilities so far has been the patching process itself. Microsoft, in particular, riled some of its Azure customers with forced, unscheduled reboots, after reports about Meltdown and Spectre surfaced before the embargo on the disclosure was to be lifted. Google, for its part, said it avoided reboots by live migrating all its customers.

And while Amazon Web Services (AWS), Microsoft, Google and others could quietly get ahead of the problem to varying degrees, smaller cloud companies were often left scrambling.

AMD and Intel have worked on firmware updates to further mitigate the problem, but early versions of these have caused issues of their own.  Updated patches are supposedly imminent, but it’s unclear if they will require another round of cloud provider reboots.

The initial patches to Meltdown and Spectre are stopgap measures — it may take years to redesign chips in a way that doesn’t rely on speculative execution, an optimization technique at the root of these vulnerabilities. It’s also possible that any fundamental redesign of these chips could ultimately benefit cloud vendors, which swap out hardware more frequently than traditional enterprises and thus could jump on the new processors faster.

I can’t imagine a chief risk officer or chief security officer saying this is inconsequential to what we’re going to do in the future.
Marty PuranikCEO, Atlantic.Net

These flaws could cause potential customers to rein in their cloud computing usage, or do additional due diligence before they transition out of their own data centers. This is particularly true in the financial sector and other heavily regulated industries that have just begun to warm to the public cloud.

“If you [are] starting a new project, there’s this question mark that wasn’t there before,” said Marty Puranik, CEO of Atlantic.Net, a cloud hosting provider in Orlando, Fla. “I can’t imagine a chief risk officer or chief security officer saying this is inconsequential to what we’re going to do in the future.”

Performance hits not as bad as first predicted

The other potential fallout from Spectre and Meltdown is how the patches will impact performance. Initial predictions were up to a 30% slowdown, and frustrated customers took to the Internet to highlight major performance hits. Cloud vendors have pushed back on those estimates, however, and multiple managed service providers that oversee thousands of servers on behalf of their clients said that the vast majority of workloads were unaffected.

While it remains to be seen if performance issues will start to emerge over time, IT pros seem to corroborate the providers’ claims. More than a dozen sources — many of whom requested anonymity because of the situation’s sensitive and fluid nature — told SearchCloudComputing that they saw almost no impact from the patches.

The reality is that the number of impacted systems is fairly small and the performance impact is highly variable, said Kinsella. “If it was really 30% I think we’d be having a different conversation because that’s like rolling back a couple years of Moore’s Law,” he said.

Zendesk, based in San Francisco, suspected something was up with its cloud environment following an uptick in reboot notices from AWS toward the end of 2017, said Steve Loyd, vice president of technology operations at Zendesk. Those reboots weren’t exactly welcome, but were better than the alternative, and the company hasn’t seen a big impact from testing patches so far, he said.

Google said it has seen no reports of notable impacts for its cloud customers, while Microsoft and AWS initially said they expected a minority of customers to see performance degradation. It’s unclear how Microsoft has mitigated these issues for those customers, though it has recommended customers switch to a faster networking service that just became generally available. AWS said in a statement that, since installing its patches, it has worked with impacted customers to optimize workloads and “in almost every case, prevent significant changes to their cost.”

The biggest potential exception to these negligible impacts on cloud computing usage would be anything that uses the OS kernel extensively, such as distributed databases or caching systems. Of course, the same type of workload on premises would presumably face the same problem, but even a small impact adds up at scale.

“If any single system doesn’t appear to have more than 1% impact, it’s almost immeasurable,” said Eric Wright, chief evangelist at Turbonomic, a Boston-based hybrid cloud management provider. “But if you have that across 100 systems, you have to add one new virtual system to your load, so no matter how you slice it, there’s some kind of impact.”

Cloud providers also could take more of a hit with customers simply because of their pricing schemes. A company that owns its own data center could just throw some underused servers at the problem. But cloud vendors charge based on CPU cycles, and slower workloads there could have a more pronounced impact, said Pete Lindstrom, an analyst at IDC.

“It’s impressionistic stuff but that’s how security works,” he said. “Really, the question will be what does the monthly bill look like, and is the impact actually there?”

The biggest beneficiary from performance impacts could be abstracted services, such as serverless or platform as a service products. In those scenarios, all the patches are the responsibility of the provider and analysts believe that, to the customer, these services will appear unaltered.

ACI Information Group, a news and social media aggregator, patched its AWS EC2 instances, base AMIs and Docker images. So far the company hasn’t noticed any huge issues, but employees did take note that its serverless workloads required no work on their part to address the problem and the performance was unaffected, said Chris Moyer, vice president of technology at ACI and a TechTarget contributor.

“We have about 40% of our workload on serverless now, so that’s a big win for us too, and another reason to complete our migration entirely,” he said.

Trevor Jones is a senior news writer with SearchCloudComputing and SearchAWS. Contact him at tjones@techtarget.com.

Apple High Sierra patch undone by macOS update

A critical patch for a vulnerability in Apple’s macOS High Sierra may not be properly applied if a user also updates the system software.

The vulnerability, which was made public on Nov. 28, could allow a malicious user to bypass authentication dialogs and even potentially acquire root system privileges. Apple released the High Sierra patch the following day, but users have reported the patch being undone depending on system updates that were applied.

According many users on Twitter — and first reported by Wired — if the Apple system was running macOS 10.13.0 and not the newer 10.13.1 version, the High Sierra patch would be undone after the system update was applied. Additionally, re-installing the High Sierra patch after the system update would require a reboot to properly apply the fix, but users were not getting the notification that a restart was necessary.

Apple has since updated its patch notes to include these issues: “If you recently updated from macOS High Sierra 10.13 to 10.13.1, reboot your Mac to make sure the Security Update is applied properly.”

MacLemon, a Mac sysadmin and independent security researcher, said the system update downgrading the High Sierra patch shouldn’t be surprising.

It’s part of Apple growing carelessness for the Mac in general.
MacLemona Mac sysadmin and independent security researcher

“It’s mostly expected that an older updated installed over a newer system downgrades components. The failure here is that Apple doesn’t show the Security Update 2017-001 again after reinstalling 10.13.1,” MacLemon told SearchSecurity via Twitter Direct Message. “It’s part of Apple growing carelessness for the Mac in general. Since they changed the development process to release on time instead of when done Mac OS X/OS X/macOS quality and stability has been in steady decline. Banana software shipped green that ripens at the customer.”

Because of the confusion surrounding the High Sierra patch and the macOS update, users may not know if the patch was applied properly and whether or not they are protected against the root password flaw, as Marc Rogers, head of SecOps for DefCon and head of infosec for Cloudflare, said on Twitter.

Experts suggested checking for software updates and ensuring systems have been rebooted.

Root passwords and the High Sierra patch

When the High Sierra root flaw was first announced, an early suggestion from experts was to create a password for the root user. However, MacLemon noted this could cause security issues as well.

Additionally, Adam Nichols, principal of software security at Grimm, said creating this password would not be a full fix anyway.

DUHK attack puts random number generators at risk

Researchers have discovered a vulnerability that affects some legacy security devices, including Fortinet’s FortiGate devices.

The vulnerability has been dubbed DUHK, which stands for Don’t Use Hard-coded Keys, and affects devices that use the ANSI X9.31 Random Number Generator (RNG) and a hardcoded seed key. Researchers Nadia Heninger and Shaanan Cohney from the University of Pennsylvania, along with cryptographer Matthew Green at Johns Hopkins University, studied the Federal Information Processing Standards (FIPS) certified products that use the ANSI X9.31 RNG algorithm and found 12 that are vulnerable to DUHK.

“DUHK allows attackers to recover secret encryption keys from vulnerable implementations and decrypt and read communications passing over VPN connections or encrypted web sessions,” the researchers explained in a blog post. “The encrypted data could include sensitive business data, login credentials, credit card data and other confidential content.”

Heninger, Cohney and Green were only able to gain access to the firmware of one product — a Fortinet firewall — so their detailed research paper mostly focuses on the affected Fortinet devices, specifically the FortiGate VPN gateways.

“Traffic from any VPN using FortiOS 4.3.0 to FortiOS 4.3.18 can be decrypted by a passive network adversary who can observe the encrypted handshake traffic,” they explained. “Other key recovery attacks on different protocols may also be possible.”

The full list of affected vendors is in the research paper and includes Fortinet, Becrypt, Cisco, DeltaCrypt Technologies, MRV Communications, NeoScale Systems, Neopost Technologies, Renesas Technology America, TechGuard Security, Tendyron Corp., ViaSat and Vocera Communications.

The ANSI X9.31 RNG algorithm lost its FIPS certification in January 2016, so the researchers noted that many vendors have since published software updates to remove it.

Devices have to meet four requirements in order to be vulnerable to DUHK, according to Heninger, Cohney and Green:

  • A device must use the X9.31 RNG.
  • A seed key is hardcoded into the implementation.
  • The output from the RNG is used to generate crypto keys.
  • “At least some of the random numbers before or after those used to make the keys are transmitted unencrypted. This is typically the case for SSL/TLS and IPsec.”

The researchers recommended anyone who develops cryptographic software should stop using the X9.31 RNG and not use a hardcoded key.

The research team also warned that this vulnerability is the key to an easy and practical attack, though there’s no evidence it’s being actively exploited by attackers.

“Our attack against [the] FortiGate device can be carried out on a modern computer in about four minutes,” they noted.

In other news:

  • FBI Director Christopher Wray spoke earlier this week about the FBI’s continuous battle with mobile device encryption. Speaking at the International Association of Chiefs of Police conference in Philadelphia, Wray said the FBI was unable to access more than 6,900 mobile devices so far this year. “To put it mildly, this is a huge, huge problem,” Wray said. “It impacts investigations across the board — narcotics, human trafficking, counterterrorism, counterintelligence, gangs, organized crime [and] child exploitation.” The FBI has warred with vendors and the security community in recent years over encryption in mobile devices, arguing that law enforcement needs backdoors through encryption to access devices during investigations. Vendors such as Apple and security experts argue that a backdoor cannot exist for law enforcement without it being accessible by malicious actors, as well, and thus putting user privacy at risk. Wray’s comment follows the U.S. Department of Justice’s call for “responsible encryption.”
  • A group of senators and congressmen have introduced a bipartisan bill that would create a new legal framework that would allow law enforcement to access U.S. electronic communications stored on servers located in other countries. The group includes Rep. Doug Collins (R-Ga.), Rep. Hakeem Jeffries (D-N.Y.), Sen. Orrin Hatch (R-Utah), Sen. Chris Coons (D-Del.), and Sen. Dean Heller (R-Nev.). They are calling on Congress to pass the bill, called the International Communications Privacy Act, and are supported by organizations such as Americans for Tax Reform and the R Street Institute, which penned a letter to the Congress pushing for the bill. With this new bill, the group of senators and representatives aims to update the Electronic Communications Privacy Act of 1986, which they argued is outdated. The International Communications Privacy Act would require law enforcement to obtain a warrant for all electronic data on U.S. citizens and allow law enforcement to access data on foreign nationals.
  • Serious security flaws have been discovered in the way the Presidential Advisory Commission on Election Integrity, which is investigating voter fraud, handles the personal data of millions of voters. Illinois-based advocacy group Indivisible Chicago requested public records from Illinois and Florida on the Interstate Voter Registration Crosscheck Program. Crosscheck aims to identify people who are registered and voting in more than one state. Indivisible Chicago received emails and other documents from election officials, which showed several security issues with Crosscheck, including the freely available usernames and passwords. “The primary problem here is not that we have these passwords, but that every official and IT department involved in this process sends usernames, login passwords, and decryption passwords in clear text in email — sometimes with up to eighty recipients,” Indivisible Chicago wrote. “Anyone could have these passwords and could have had them at a time they could have been used while the ISBE would have been none the wiser.”

KRACK WPA2 vulnerability might be more hype than risk

Researchers found a vulnerability in the WPA2 protocol that could affect many Wi-Fi devices in the wild, but some experts said early reports overstated the danger the flaw.

The WPA2 vulnerability was discovered by Mathy Vanhoef, a network security and applied crypto post-doctoral candidate, and Frank Piessens, a computer science professor at the University of Leuven in Flanders, Belgium. According to Vanhoef, the WPA2 vulnerability is “in the Wi-Fi standard itself, and not in individual products or implementations. Therefore, any correct implementation of WPA2 is likely affected.”

“An attacker within range of a victim can exploit these weaknesses using key reinstallation attacks (KRACKs). Concretely, attackers can use this novel attack technique to read information that was previously assumed to be safely encrypted,” Vanhoef wrote in his report detailing the flaw. “This can be abused to steal sensitive information such as credit card numbers, passwords, chat messages, emails, photos, and so on. The attack works against all modern protected Wi-Fi networks. Depending on the network configuration, it is also possible to inject and manipulate data.”

Rob Graham, owner of Errata Security, described the issue as “reliable enough that people should be afraid.”

“When a client connects to the network, the access-point will at some point send random key data to use for encryption. Because this packet may be lost in transmission, it can be repeated many times. What the hacker does is just repeatedly sends this packet, potentially hours later. Each time it does so, it resets the keystream back to the starting conditions,” Graham wrote in a blog post. “At this point, the protocol bug becomes a crypto bug. We know how to break crypto when we have two keystreams from the same starting position.”

Vanhoef and Piessens said that the biggest potential risk with the KRACK WPA2 vulnerability would be in Linux and Android devices where it could more easily be exploited. On Android, Graham said attackers could even create “a fake WiFi access-point and man-in-the-middle all traffic.”

Questions surrounding the KRACK WPA2 vulnerability risk

Other experts were not as convinced that KRACK was as big a danger because of the difficulty in successfully exploiting the flaw.

Martijn Grooten, security researcher for Virus Bulletin, was impressed with the research but not sold on the impact.

You would need an incredibly high skill set and to be at the Wi-Fi base station to attack this.
Kevin Beaumontsecurity architect based in the U.K.

Kevin Beaumont, security architect based in the U.K., said on Twitter that the KRACK WPA2 vulnerability was “very difficult to exploit.”

“The attack realistically doesn’t work against Windows or iOS devices. The Group vulnerability is there, but it’s not near enough to actually do anything of interest,” Beaumont added in a blog post detailing the flaw. “There is currently no publicly available code out there to attack this in the real world – you would need an incredibly high skill set and to be at the Wi-Fi base station to attack this.”

Remediating the KRACK WPA2 vulnerability

While some experts said traffic running over HTTPS could help to mitigate the risks of data interception by an attacker exploiting KRACK, most said patching was the best option and praised Vanhoef and Piessens for their responsible disclosure and waiting for major manufacturers to create patches.

Vanhoef and Piessens said patching either the client or access point can mitigate the risk of the KRACK WPA2 vulnerability.

“Implementations can be patched in a backwards-compatible manner. This means a patched client can still communicate with an unpatched access point, and vice versa. In other words, a patched client or access point sends exactly the same handshake messages as before, and at exactly the same moment in time,” Vanhoef wrote. “However, the security updates will assure a key is only installed once, preventing our attack.”

Experts said that although patches may be available, another trouble with the KRACK WPA2 vulnerability is scale. Currently, Android versions 6.0 and higher are affected — approximately 41% of devices, according to Google’s stats — but as more devices are updated to newer Android versions or users buy new smartphones, that number could rise.

Apple added patches for the WPA2 vulnerability in the latest beta version of iOS, but it has not been pushed to most users as of this post. Beaumont said patches were already available for Linux, but he admitted that Android could be a problem because security patches often don’t get to users or get patched when they are pushed out.

Paul Martini, CEO and co-founder of iboss, said the most concerning part of this exploit “is that it affects every device that connects to a Wi-Fi network.”

“This is another reminder that even if your security team does everything in its power to keep your devices and network secure, there will always be incidents like this that put your information at risk,” Martini told SearchSecurity. “This vulnerability is particularly worrisome for distributed organizations that often have mobile and remote employees connecting via Wi-Fi. Even secure Wi-Fi networks are now at risk and require additional protection at the web gateway level to help prevent data loss.”

Apache Struts vulnerability affects versions since 2008

A security researcher discovered an Apache Struts vulnerability that affects versions of the web application development framework going back to 2008.

Man Yue Mo, researcher at the open source software project LGTM.com run by software analytics firm Semmle, Inc., headquartered in San Francisco, disclosed the remotely executable Apache Struts vulnerability, which he said was “a result of unsafe deserialization in Java” and could lead to arbitrary code execution. Mo originally disclosed the issue to Apache on July 17, 2017.  

Mo publicly disclosed the Apache Struts vulnerability on Sept. 5 and the Apache Struts group released patches the same day, but by the morning of Sept. 6 Mo updated his post because “multiple working exploits [were] observed on various places on the internet.”

“I verified that it was a genuine remote code execution [RCE] vulnerability before reporting it to the Struts security team. They have been very quick and responsive in working out a solution even though it is a fairly non-trivial task that requires API changes,” Mo wrote in a blog post. “Due to the severity of this finding I will not disclose more details at this stage. Rather, I will update this blog post in a couple of weeks’ time with more information.”

Mo’s discovery is the latest in string of serious Apache Struts vulnerabilities that have been disclosed recently. In March, for example, an RCE vulnerability was patched after being actively exploited by attackers.

Boris Chen, vice president of engineering and co-founder of tCell, Inc., a web applications security company headquartered in San Francisco, said “serialization exploits resulting in RCE are one of the most serious yet underreported vulnerabilities that applications face today, and it doesn’t seem to be waning. For Apache Struts alone, this is the fourth RCE vulnerability this year.”

The newly discovered Apache Struts vulnerability is a stark reminder that while websites represent the front-line for most organizations, they can also become the front door for attackers.
Brian Robisonsenior director of security technology, Cylance Inc.

Michael Patterson, CEO of Plixer International, Inc., a network traffic analysis company based in Kennebunk, Me., said that this Apache Struts vulnerability “is a significant finding given that the majority of our largest companies are using Apache Struts.”  

“Although a patch for the vulnerability has since been released, given that many companies don’t stay on top of patches, there still could be plenty of time for malicious code writers to exploit it,” Patterson told SearchSecurity. “Most organizations are aware that there is absolutely no way to prevent being compromised.”

Brian Robison, senior director of security technology at Cylance Inc., said attacks like this are not new but should be a wake-up call.

“The newly discovered Apache Struts vulnerability is a stark reminder that while websites represent the front-line for most organizations, they can also become the front door for attackers. Many organizations develop layers of security to protect their public facing websites, but in some cases, those layers can’t stop something that looks like normal behavior,” Robinson told SearchSecurity. “No matter whether someone is using Apache, IIS or any other web server, it is critical that they keep up with patches and security feeds. A web server that is left idle while the company focuses on building the content can quickly become ground-zero for a wide spread attack.”