Tag Archives: discovered

No need to rush network patching for Spectre and Meltdown

The recently discovered security threat in CPUs from nearly a dozen manufacturers poses a low risk to corporate networking gear, so operators have time to test vendors’ patches thoroughly.

That’s the take of security experts contacted by SearchNetworking following the discovery last week of the Spectre and Meltdown vulnerabilities that affect Intel, AMD and ARM chips. In response, Cisco and Juniper Networks have released patches rated medium and low risk, respectively, for a variety of products.

The low risk of Spectre and Meltdown to switches and routers means network managers have the time to thoroughly test the patches to minimize their impact on hardware performance, experts said.

“If you’re getting a firmware update, you need to patch,” said Rob Westervelt, analyst at IDC. “[But] the issue is whether you just deploy the patch or test it thoroughly and make sure you don’t break any applications or anything else.”

Roughly 20 CSOs and IT security professionals interviewed by IDC were taking a methodical approach to applying Spectre and Meltdown fixes across all systems.

“While it is top of mind, it’s not something that they’re immediately jumping on to patch,” Westervelt said. “They are using established best practices and testing those patches first.”

Network performance at risk

Westervelt warned there is the possibility network performance will suffer. “In some cases, it could be very costly.”

If you’re getting a firmware update, you need to patch.
Rob Westerveltanalyst at IDC

Indeed, Microsoft reported in a blog post patches for the PC and server versions of Windows would range from minor to significant, depending on the age of the operating system and the CPU. “I think we can expect a similar variety of performance impacts across other [vendors’] products,” said Jake Miller, a senior security analyst at IT consulting firm Bishop Fox, based in Tempe, Ariz.

Security pros expect hackers sophisticated enough to exploit the hard-to-reach vulnerabilities to target mostly servers in large data centers that host cloud computing environments. Because of the level of expertise needed to take advantage of the flaws, hackers working for nation states are the most likely attackers, experts said.

Exploiting the CPU holes would involve crafting code that takes advantage of how some processors anticipate features computer users will request next. In preparation for those requests, processors will load into memory valuable data and instructions that hackers can steal.

“The threat is significant, but currently is limited to highly sophisticated attackers and hacking groups with the means to carry out multi-staged targeted attacks,” IDC said in a research note. “Financially motivated cybercriminals are more likely to continue to use more accessible, time-tested methods to retrieve passwords and sensitive data.”

Nevertheless, even a low risk to networking gear is worth the time needed for fixing. “It’s better to be safe than sorry,” said Jonathan Valamehr, COO and co-founder of cybersecurity company Tortuga Logic Inc.

Return of Bleichenbacher: ROBOT attack means trouble for TLS

A team of security researchers discovered eight leading vendors and open source projects whose implementations of the Transport Layer Security protocol are vulnerable to the Bleichenbacher oracle attack, a well-known flaw that was first described in 1998.

The Bleichenbacher attack has been referenced in all IETF specifications for the Transport Layer Security (TLS) protocol since version 1.0 in 1999, and implementers of TLS versions through 1.2 were warned to take steps to avoid the Bleichenbacher attack. However, the researchers noted that, based on the ease with which they were able to exploit the vulnerability, it appears that many implementers ignored the warnings.

The attack is named after its discoverer, Daniel Bleichenbacher, a Swiss cryptographer who was working for Bell Laboratories in 1998 when his research on the vulnerability was first published. The TLS protocol, which was meant to replace the Secure Sockets Layer, is widely used for encryption and the authentication of web servers.

The research team  included Hanno Bock, information security researcher; Juraj Somorovsky, research associate at the Horst Görtz Institute for IT Security at the Ruhr-Universität Bochum in Germany; and Craig Young, , computer security researcher with Tripwire’s Vulnerability and Exposures Research Team (VERT). “Perhaps the most surprising fact about our research is that it was very straightforward,” the researchers wrote. “We took a very old and widely known attack and were able to perform it with very minor modifications on current implementations. One might assume that vendors test their TLS stacks for known vulnerabilities. However, as our research shows in the case of Bleichenbacher attacks, several vendors have not done this.”

The researchers said many web hosts are still vulnerable to the ROBOT attack and that nearly a third of the top 100 sites in the Alexa Top 1 Million list are vulnerable. The team identified vulnerable products from F5, Citrix, Radware, Cisco, Erlang, and others, and “demonstrated practical exploitation by signing a message with the private key of facebook.com’s HTTPS certificate.”

The researchers described their work as the “Return Of Bleichenbacher’s Oracle Threat” (ROBOT) and published it in a paper of the same title, as well as on a branded vulnerability website. The team also published a capture the flag contest, posting an encrypted message and challenging the public to decrypt the message using the strategies described in the paper.

TLS protocol designers at fault

The researchers placed the blame for the ease of their exploits squarely on the shoulders of TLS protocol designers. The ROBOT attack is made possible by the behavior of servers implementing TLS using the RSA Public-Key Cryptography Standards (PKCS) #1 v1.5 specification; the issues that enable the Bleichenbacher attack are fixed in later versions of PKCS. TLS 1.3, which is expected to be finalized soon, deprecates the use of PKCS #1 v1.5 and specifies use of PKCS #1 v2.2.

The TLS protocol designers absolutely should have been more proactive about replacing PKCS#1 v1.5.
Craig Youngcomputer security researcher, Tripwire VERT

“The TLS protocol designers absolutely should have been more proactive about replacing PKCS#1 v1.5. There is an unfortunate trend in TLS protocol design to continue using technology after it should have been deprecated,” Young told SearchSecurity by email. He added that vendors also “should have been having their code audited by firms who specialize in breaking cryptography since most software companies do not have in-house expertise for doing so.”

TLS as currently deployed ignores improperly formatted data, and as described in 1999 in RFC 2246. “The TLS Protocol Version 1.0,” the original specification for TLS 1.0, the ROBOT attack “takes advantage of the fact that by failing in different ways, a TLS server can be coerced into revealing whether a particular message, when decrypted, is properly PKCS #1 formatted or not,” the RFC 2246 document states.

The solution proposed in that specification for avoiding “vulnerability to this attack is to treat incorrectly formatted messages in a manner indistinguishable from correctly formatted RSA blocks. Thus, when it receives an incorrectly formatted RSA block, a server should generate a random 48-byte value and proceed using it as the premaster secret. Thus, the server will act identically whether the received RSA block is correctly encoded or not.”

Potential for attacks, detection and remediation

The researchers noted in the paper that the ROBOT flaw could lead to very serious attacks. “For hosts that are vulnerable and only support RSA encryption key exchanges it’s pretty bad. It means an attacker can passively record traffic and later decrypt it,” the team wrote on the ROBOT website, adding that “For hosts that usually use forward secrecy, but still support a vulnerable RSA encryption key exchange the risk depends on how fast an attacker is able to perform the attack. We believe that a server impersonation or man in the middle attack is possible, but it is more challenging.”

Young said that it might be possible to detect attempts to abuse the Bleichenbacher vulnerability, but it would not be easy. “This attack definitely triggers identifiable traffic patterns. Servers would observe a high volume of failed connections as well as a smaller number of connections with successful handshakes and then little to no data on the connection,” he told SearchSecurity. “Unfortunately, I am unaware of anybody actually doing this. Logging the information needed to detect this can be cumbersome and for a site receiving a billion connections a second, it could be quite difficult to notice 10-100 thousand failed connections.”

As for other, ongoing risks, Young said that while “PKCS#1 v1.5 is not being used in TLS 1.3 but it is still used in other systems like XML encryption. Whether or not it can be disabled through configuration is highly application specific.”

DUHK attack puts random number generators at risk

Researchers have discovered a vulnerability that affects some legacy security devices, including Fortinet’s FortiGate devices.

The vulnerability has been dubbed DUHK, which stands for Don’t Use Hard-coded Keys, and affects devices that use the ANSI X9.31 Random Number Generator (RNG) and a hardcoded seed key. Researchers Nadia Heninger and Shaanan Cohney from the University of Pennsylvania, along with cryptographer Matthew Green at Johns Hopkins University, studied the Federal Information Processing Standards (FIPS) certified products that use the ANSI X9.31 RNG algorithm and found 12 that are vulnerable to DUHK.

“DUHK allows attackers to recover secret encryption keys from vulnerable implementations and decrypt and read communications passing over VPN connections or encrypted web sessions,” the researchers explained in a blog post. “The encrypted data could include sensitive business data, login credentials, credit card data and other confidential content.”

Heninger, Cohney and Green were only able to gain access to the firmware of one product — a Fortinet firewall — so their detailed research paper mostly focuses on the affected Fortinet devices, specifically the FortiGate VPN gateways.

“Traffic from any VPN using FortiOS 4.3.0 to FortiOS 4.3.18 can be decrypted by a passive network adversary who can observe the encrypted handshake traffic,” they explained. “Other key recovery attacks on different protocols may also be possible.”

The full list of affected vendors is in the research paper and includes Fortinet, Becrypt, Cisco, DeltaCrypt Technologies, MRV Communications, NeoScale Systems, Neopost Technologies, Renesas Technology America, TechGuard Security, Tendyron Corp., ViaSat and Vocera Communications.

The ANSI X9.31 RNG algorithm lost its FIPS certification in January 2016, so the researchers noted that many vendors have since published software updates to remove it.

Devices have to meet four requirements in order to be vulnerable to DUHK, according to Heninger, Cohney and Green:

  • A device must use the X9.31 RNG.
  • A seed key is hardcoded into the implementation.
  • The output from the RNG is used to generate crypto keys.
  • “At least some of the random numbers before or after those used to make the keys are transmitted unencrypted. This is typically the case for SSL/TLS and IPsec.”

The researchers recommended anyone who develops cryptographic software should stop using the X9.31 RNG and not use a hardcoded key.

The research team also warned that this vulnerability is the key to an easy and practical attack, though there’s no evidence it’s being actively exploited by attackers.

“Our attack against [the] FortiGate device can be carried out on a modern computer in about four minutes,” they noted.

In other news:

  • FBI Director Christopher Wray spoke earlier this week about the FBI’s continuous battle with mobile device encryption. Speaking at the International Association of Chiefs of Police conference in Philadelphia, Wray said the FBI was unable to access more than 6,900 mobile devices so far this year. “To put it mildly, this is a huge, huge problem,” Wray said. “It impacts investigations across the board — narcotics, human trafficking, counterterrorism, counterintelligence, gangs, organized crime [and] child exploitation.” The FBI has warred with vendors and the security community in recent years over encryption in mobile devices, arguing that law enforcement needs backdoors through encryption to access devices during investigations. Vendors such as Apple and security experts argue that a backdoor cannot exist for law enforcement without it being accessible by malicious actors, as well, and thus putting user privacy at risk. Wray’s comment follows the U.S. Department of Justice’s call for “responsible encryption.”
  • A group of senators and congressmen have introduced a bipartisan bill that would create a new legal framework that would allow law enforcement to access U.S. electronic communications stored on servers located in other countries. The group includes Rep. Doug Collins (R-Ga.), Rep. Hakeem Jeffries (D-N.Y.), Sen. Orrin Hatch (R-Utah), Sen. Chris Coons (D-Del.), and Sen. Dean Heller (R-Nev.). They are calling on Congress to pass the bill, called the International Communications Privacy Act, and are supported by organizations such as Americans for Tax Reform and the R Street Institute, which penned a letter to the Congress pushing for the bill. With this new bill, the group of senators and representatives aims to update the Electronic Communications Privacy Act of 1986, which they argued is outdated. The International Communications Privacy Act would require law enforcement to obtain a warrant for all electronic data on U.S. citizens and allow law enforcement to access data on foreign nationals.
  • Serious security flaws have been discovered in the way the Presidential Advisory Commission on Election Integrity, which is investigating voter fraud, handles the personal data of millions of voters. Illinois-based advocacy group Indivisible Chicago requested public records from Illinois and Florida on the Interstate Voter Registration Crosscheck Program. Crosscheck aims to identify people who are registered and voting in more than one state. Indivisible Chicago received emails and other documents from election officials, which showed several security issues with Crosscheck, including the freely available usernames and passwords. “The primary problem here is not that we have these passwords, but that every official and IT department involved in this process sends usernames, login passwords, and decryption passwords in clear text in email — sometimes with up to eighty recipients,” Indivisible Chicago wrote. “Anyone could have these passwords and could have had them at a time they could have been used while the ISBE would have been none the wiser.”

CCleaner malware spread via supply chain attack

Researchers discovered a popular system maintenance tool was the victim of a supply chain attack that put potentially millions of users at risk of downloading a malicious update.

CCleaner is a tool designed to help consumers perform basic PC maintenance functions like removing cached files, browsing data and defragmenting hard drives. CCleaner is made by Piriform Ltd., a UK-based software maker that was acquired by antivirus company Avast Software in July. The compromised update of the tool was first discovered by Israeli endpoint security firm Morphisec following an investigation that began on Sept. 11th, but the company claims it began blocking the CCleaner malware at customer sites on Aug. 20th.

“A backdoor transplanted into a security product through its production chain presents a new unseen threat level which poses a great risk and shakes customers’ trust,” wrote Michael Gorelik, vice president of research and development at Morphisec in a blog post. “As such, we immediately, as part of our responsible disclosure policy, contacted Avast and shared all the information required for them to resolve the issue promptly. Customers safety is our top concern.”

The CCleaner malware gathered information about systems and transmitted it to a command and control (C&C) server; it was reportedly downloaded by users for close to one month from August 15 to September 12, according to Morphisec. However, Avast noted that the CCleaner malware was limited to running on 32-bit systems and would only run if the affected user profile had administrator privileges.

Avast said CCleaner claims to have more than 2 billion downloads and adds new users at a rate of 5 million per week, but because only the 32-bit and cloud versions of CCleaner were compromised, the company estimated just 2.27 million users were affected.

Impact of the CCleaner malware

A team of researchers at Cisco Talos, which included Edmund Brumaghin, threat researcher, Ross Gibb, senior information security analyst, Warren Mercer, technical leader, Matthew Molyett, research engineer, and Craig Williams, senior technical leader, discovered and analyzed the CCleaner malware soon after Morphisec. According to the Cisco Talos team, Avast unwittingly distributed legitimate signed versions of CCleaner and CCleaner Cloud which “contained a multi-stage malware payload that rode on top of the installation.”

“This is a prime example of the extent that attackers are willing to go through in their attempt to distribute malware to organizations and individuals around the world. By exploiting the trust relationship between software vendors and the users of their software, attackers can benefit from users’ inherent trust in the files and web servers used to distribute updates,” Talos researchers wrote in their analysis. “In many organizations data received from commonly software vendors rarely receives the same level of scrutiny as that which is applied to what is perceived as untrusted sources. Attackers have shown that they are willing to leverage this trust to distribute malware while remaining undetected.”

What makes this attack particularly worrying is the volume of downloads this software receives leaving a huge number of users exposed.
James MaudeSenior security engineer for Avecto

James Maude, senior security engineer for Avecto, a privilege management software maker, said it was especially concerning that the CCleaner malware included the official code signature from Avast.

“Given that CCleaner is designed to be installed by a user with admin rights, and the malware was not only embedded within it but also signed by the developers own code signing certificate (giving it a high level of trust), this is pretty dangerous,” Maude told SearchSecurity via email. “This means that the malware, and therefore the attacker, would have complete control of the system and the ability to access almost anything they wanted. What makes this attack particularly worrying is the volume of downloads this software receives leaving a huge number of users exposed.”

Itsik Mantin, director of security research at security software company Imperva, said the CCleaner malware incident shows “there’s not much users can do when the vendor gets infected.”

“This hack creates a new reality where users need to assume that their desktops, laptops and smartphones are infected, which has been the reality for security officers at organizations in the last years,” Mantin told SearchSecurity. “For organizations, this does not really matter as security officers are accustomed to the reality that they should always assume the attackers are in, are looking for ways to spread the infection within the organization and are searching for business sensitive data to steal or corrupt.”

Avast response to the CCleaner malware incident

Vince Steckler, CEO of Avast Software, and Ondřej Vlček, executive vice president and general manager of the consumer business unit, released a statement saying the company remediated the issue within 72 hours of becoming aware of the problem by releasing an clean update without the malware. They also stated Avast is working with law enforcement to shut down the CCleaner malware C&C server on Sept. 15th.

The Avast execs downplayed their company’s involvement by saying they “strongly suspect that Piriform was being targeted while they were operating as a standalone company, prior to the Avast acquisition,” and that the compromise “may have started on July 3rd,” two weeks before Avast’s acquisition of Piriform was complete. Avast also claimed the compromised update took four weeks to discover due to “the sophistication of the attack.”

Avast asserted users “should upgrade even though they are not at risk as the malware has been disabled on the server side,” and claimed it was unnecessary to follow the suggestions by Talos and other experts to restore systems to a date before Aug. 15, 2017 to ensure removal of the CCleaner malware.

“Based on the analysis of this data, we believe that the second stage payload never activated, i.e. the only malicious code present on customer machines was the one embedded in the ccleaner.exe binary,” Steckler and Vlček wrote. “Therefore, we consider restoring the affected machines to the pre-August 15 state unnecessary. By similar logic, security companies are not usually advising customers to reformat their machines after a remote code execution vulnerability is identified on their computer.”

Supply chain attacks

Experts said the CCleaner malware incident should be a reminder of the dangers of supply chain attacks.

Marco Cova, senior security researcher at malware protection vendor Lastline, said the recent NotPetya attacks were another case of a supply chain attack “where an otherwise trusted software vendor gets compromised and the update mechanism of the programs they distribute is leveraged to distribute malware.”

“This is sort of a holy grail for malware authors because they can efficiently distribute their malware, hide it in a trusted channel, and reach a potentially large number of users,” Cova told SearchSecurity. “It appears that the build process of CCleaner itself was compromised: that is, attackers had access to the infrastructure used to build the software itself. This is very troublesome because it indicates that attackers were able to control a critical piece of the infrastructure used by the vendor.”

Jonathan Cran, vice president of product at Bugcrowd, told SearchSecurity the CCleaner malware issue appeared to be “less of a traditional supply chain attack and more of a case of poor vendor security. Given that the affected installer was signed as a verified safe binary by Piriform, this indicates that they didn’t realize at the time of release and that the corporate network of Piriform was likely compromised.”

Justin Fier, director for cyber intelligence and analysis at threat detection company Darktrace, said this “should come as yet another wake-up call that corporations must have visibility into how their suppliers interact with their systems, as well as a real-time assessment of their suppliers’ cyber risk.”

“The risk that companies inherit from their suppliers is a pervasive problem for cybersecurity. Quite simply, companies with a supply chain cannot avoid compromises — supply chain breaches are inevitable,” Fier told SearchSecurity. “The assessment of potential supply chain partners is often a rushed process in terms of evaluating their cyber security level, and is rarely as in-depth as it should be. While we can’t change the security posture of our supply chains, we can have a transparent relationship when it comes to cyber risk.”

Apache Struts vulnerability affects versions since 2008

A security researcher discovered an Apache Struts vulnerability that affects versions of the web application development framework going back to 2008.

Man Yue Mo, researcher at the open source software project LGTM.com run by software analytics firm Semmle, Inc., headquartered in San Francisco, disclosed the remotely executable Apache Struts vulnerability, which he said was “a result of unsafe deserialization in Java” and could lead to arbitrary code execution. Mo originally disclosed the issue to Apache on July 17, 2017.  

Mo publicly disclosed the Apache Struts vulnerability on Sept. 5 and the Apache Struts group released patches the same day, but by the morning of Sept. 6 Mo updated his post because “multiple working exploits [were] observed on various places on the internet.”

“I verified that it was a genuine remote code execution [RCE] vulnerability before reporting it to the Struts security team. They have been very quick and responsive in working out a solution even though it is a fairly non-trivial task that requires API changes,” Mo wrote in a blog post. “Due to the severity of this finding I will not disclose more details at this stage. Rather, I will update this blog post in a couple of weeks’ time with more information.”

Mo’s discovery is the latest in string of serious Apache Struts vulnerabilities that have been disclosed recently. In March, for example, an RCE vulnerability was patched after being actively exploited by attackers.

Boris Chen, vice president of engineering and co-founder of tCell, Inc., a web applications security company headquartered in San Francisco, said “serialization exploits resulting in RCE are one of the most serious yet underreported vulnerabilities that applications face today, and it doesn’t seem to be waning. For Apache Struts alone, this is the fourth RCE vulnerability this year.”

The newly discovered Apache Struts vulnerability is a stark reminder that while websites represent the front-line for most organizations, they can also become the front door for attackers.
Brian Robisonsenior director of security technology, Cylance Inc.

Michael Patterson, CEO of Plixer International, Inc., a network traffic analysis company based in Kennebunk, Me., said that this Apache Struts vulnerability “is a significant finding given that the majority of our largest companies are using Apache Struts.”  

“Although a patch for the vulnerability has since been released, given that many companies don’t stay on top of patches, there still could be plenty of time for malicious code writers to exploit it,” Patterson told SearchSecurity. “Most organizations are aware that there is absolutely no way to prevent being compromised.”

Brian Robison, senior director of security technology at Cylance Inc., said attacks like this are not new but should be a wake-up call.

“The newly discovered Apache Struts vulnerability is a stark reminder that while websites represent the front-line for most organizations, they can also become the front door for attackers. Many organizations develop layers of security to protect their public facing websites, but in some cases, those layers can’t stop something that looks like normal behavior,” Robinson told SearchSecurity. “No matter whether someone is using Apache, IIS or any other web server, it is critical that they keep up with patches and security feeds. A web server that is left idle while the company focuses on building the content can quickly become ground-zero for a wide spread attack.”

Intel kill switch code indicates connection to NSA

Security researchers studying the Intel Management Engine discovered an undocumented kill switch in the code as well as references to an National Security Agency program.

Dmitry Sklyarov, Mark Ermolov and Maxim Goryachy, security researchers for Positive Technologies, based in Framingham, Mass., found the Intel kill switch that has the ability to disable the controversial Intel Management Engine (ME).

Experts have been wary of the Intel ME because it is an embedded subsystem on every chip that essentially functions as a separate CPU with deep access to system processes and could be active even if the system were hibernating or shut off.

Lamar Bailey, director of security research and development at Tripwire, said the Intel ME is “an out of band remote management interface” that is not uncommon in hardware.

“The problem happens when there are vulnerabilities in these interfaces or weak authentication issues. The remote management interface has the ability to take over and modify a system, so to many, they are seen as security risks and they are often the target of research and hackers,” Bailey told SearchSecurity. “Many organizations, both commercial and federal, disable these features due to security concerns.”

Finding the Intel kill switch

It was previously thought that the Intel ME was impossible to access or disable because, as the Positive Technologies researchers noted in their analysis, “the executable modules are compressed by Huffman codes with unknown tables,” but the researchers found a way around this.

When inspecting the Intel ME code, the researchers found a field labeled “High Assurance Platform (HAP) enable,” which is a reference to “a multi-year NSA program with the vision to define a framework for the development of the ‘next generation’ of secure computing platforms,” according to the Trusted Computing Group.

The researchers said this was essentially an Intel kill switch for the Management Engine because once that feature was enabled, “quick checks showed that ME did not respond to commands or react to requests from the operating system.” And, because the HAP feature disabled Intel ME at such an early stage of system boot, it won’t cause the ME to crash. However, the researchers couldn’t find a way to disable the Intel kill switch.

Intel did not respond to SearchSecurity’s requests for comment on this story, but a company representative did confirm the Intel kill switch was introduced under request by the U.S. government and the HAP program, but noted the “modifications underwent a limited validation cycle and are not an officially supported configuration.”

Reactions to the Intel kill switch

Bailey said any customer big enough could make a vendor consider implementing a feature like the kill switch, “no matter if they are commercial or federal.”

“If I were using these in a highly classified area or even a secure data center, I would demand these features be turned just like we disable external port like USB,” Bailey said. “It’s just another lock on the system as companies and organizations secure their data and information.”

Satya Gupta, co-founder and CTO at application security vendor Virsec, said the Intel kill switch “at the chip level may sound nefarious, it’s almost inevitable for any technology to have a reboot function if all else fails.”

“Technology backdoors are always problematic and a very slippery slope. We’ve seen this with the encryption debate — if there’s a backdoor, it will almost inevitably get in the wrong hands and become a huge liability,” Gupta told SearchSecurity. “And if the U.S. has a backdoor, should this be shared with allies? Will China demand their own backdoors to allow access to their markets?”

Philip Lieberman, president of Lieberman Software Corp., said the design of the processor “may have flaws that can be exploited by high capability attack teams, but it is doubtful that backdoors have been implemented by design.” 

“The management engine has been a work in process that deserves criticism for its lack of transparency and it has not exhibited consistent quality. I attribute lack of security and potential kill switches to poor engineering quality by Intel rather than collaboration with intelligence agencies,” Lieberman told SearchSecurity via email. “In reality, government agencies may very well be helping Intel close security holes they have inserted by mistake (the U.S. government agencies might not be evil or conniving as some might believe).

Hyper-V Key-Value Pair Data Exchange Part 3: Linux

15 Aug 2017 by Eric Siron
   
0    
Hyper-V Articles

Some time ago, I discovered uses for Hyper-V Key-Value Pair Data Exchange services and began exploiting them on my Windows guests. Now that I’ve started building Linux guests, I need similar functionality. This article covers the differences in the Linux implementation and includes version 1.0 of a program that allows you to receive, send, and delete KVPs.

For a primer on Hyper-V KVP Exchange, start with this article: Hyper-V Key-Value Pair Data Exchange Part 1: Explanation.

The second part of that series presented PowerShell scripts for interacting with Hyper-V KVP Exchange from both the host and the guest sides. The guest script won’t be as useful in the context of Linux. Even if you install PowerShell on Linux, the script won’t work because it reads and writes registry keys. It might still spark some implementation ideas, I suppose.

What is Hyper-V Key-Value Pair Data Exchange?

To save you a few clicks and other reading, I’ll give a quick summary of Hyper-V KVP Exchange.

Virtual machines are intended to be “walled gardens”. The host and guest should have limited ability to interact with each other. That distance sometimes causes inconvenience, but the stronger the separation, the stronger the security. Hyper-V’s KVP Exchange provides one method for moving data across the wall without introducing a crippling security hazard. Either “side” (host or guest) can “send” a message at any time. The other side can receive it — or ignore it. Essentially, they pass notes by leaving them stuck in slots in the “wall” of the “walled garden”.

KVP stands for “key-value pair”. Each of these messages consists of one text key and one text value. The value can be completely empty.

How is Hyper-V KVP Exchange Different on Linux?

On Windows guests, a service runs (Hyper-V Data Exchange Service) that monitors the “wall”. When the host leaves a message, this service copies the information into the guest’s Windows registry. To send a message to the host, you (or an application) create or modify a KVP within a different key in the Windows registry. The service then places that “note” in the “wall” where the host can pick it up. More details can be found in the first article in this series.

Linux runs a daemon that is the analog to the Windows service. It has slightly different names on different platforms, but I’ve been able to locate it on all of my distributions with
sudo service statusall | grep kvp. It may not always be running; more on that in a bit.

Linux doesn’t have a native analog to the Windows registry. Instead, the daemon maintains a set of files. It receives inbound messages from the host and places them in particular files that you can read (or ignore). You can write to one of the files. The daemon will transfer those messages up to the host.

On Windows, I’m not entirely certain of any special limits on KVP sizes. A registry key can be 16,384 characters and there is no hard-coded limit on value size. I have not tested how KVP Exchange handles these extents on Windows. However, the Linux daemon has much tighter constraints. A key can be no longer than 512 bytes. A value can be no longer than 2,048 bytes.

The keys are case sensitive on the host and on Linux guests. So, key “LinuxKey” is not the same as key “linuxkey”. Windows guests just get confused by that, but Linux handles it easily.

How does Hyper-V KVP Exchange Function on Linux?

As with Windows guests, Data Exchange must be enabled on the virtual machine’s properties:

Hyper-V KVP Exchange on Linux

The daemon must also be installed and running within the guest. Currently-supported versions of the Linux kernel contain the Hyper-V KVP framework natively, so several distributions ship with it enabled. As mentioned in the previous section, the exact name of the daemon varies. You should be able to find it with:
sudo service statusall | grep kvp. If it’s not installed, check your distribution’s instruction page on TechNet.

All of the files that the daemon uses for Hyper-V KVP exchange can be found in the /var/lib/hyperv folder. They are hidden, but you can view them with ls‘s -a parameter:

Hyper-V KVP exchange

Anyone can read any of these files. Only the root account has write permissions, but that can be misleading. Writing to any of the files that are intended to carry data from the host to the guest has no real effect. The daemon is always monitoring them and only it can carry information from the host side.

What is the Purpose of Each Hyper-V KVP Exchange File?

Each of the files is used for a different purpose.

  • .kvp_pool_0: When an administrative user or an application in the host sends data to the guest, the daemon writes the message to this file. It is the equivalent of HKLMSOFTWAREMicrosoftVirtual MachineExternal on Windows guests. From the host side, the related commands are ModifyKvpItems, AddKvpItems, and RemoveKvpItems. The guest can read this file. Changing it has no useful effect.
  • .kvp_pool_1: The root account can write to this file from within the guest. It is the equivalent of HKLMSOFTWAREMicrosoftVirtual MachineGuest on Windows guests. The daemon will transfer messages up to the host. From the host side, its messages can be retrieved from the GuestExchangeItems field of the WMI object.
  • .kvp_pool_2: The daemon will automatically write information about the Linux guest into this file. However, you never see any of the information from the guest side. The host can retrieve it through the GuestIntrinsicExchangeItems field of the WMI object. It is the equivalent of the HKLMSOFTWAREMicrosoftVirtual MachineAuto key on Windows guests. You can’t do anything useful with the file on Linux.
  • .kvp_pool_3: The host will automatically send information about itself and the virtual machine through this file. You can read the contents of this file, but changing it does nothing useful. It is the equivalent of the HKLMSOFTWAREMicrosoftVirtual MachineGuestParameter key on Windows guests.
  • .kvp_pool_4: I have no idea what this file does or what it is for.

What is the Format of the Hyper-V KVP Exchange File on Linux?

Each file uses the same format.

One KVP entry is built like this:

  • 512 bytes for the key. The key is a sequence of non-null bytes, typically interpreted as
    char. According to the documentation, it will be processed as using UTF8 encoding. After the characters for the key, the remainder of the 512 bytes is padded with null characters.
  • 2,048 bytes for the value. As with the key, these are non-null bytes typically interpreted as
    char. After the characters for the value, the remainder of the 2,048 bytes is padded with null characters.

KVP entries are written end-to-end in the file with no spacing, headers, or footers.

For the most part, you’ll treat these as text strings, but that’s not strictly necessary. I’ve been on this rant before, but the difference between “text” data and “binary” data is 100% semantics, no matter how much code we write to enforce artificial distinctions. From now until the point when computers can process something other than low voltage/high voltage (0s and 1s), there will never be anything but binary data and binary files. On the Linux side, you have 512 bytes for the key and 2,048 bytes for the value. Do with them as you see fit. However, on the host side, you’ll still need to get through the WMI processing. I haven’t pushed that very far.

How Do I Use Hyper-V KVP Exchange for Linux?

This is the part where it gets fun. Microsoft only goes so far as to supply the daemon. If you want to push or pull data, that’s all up to you. Or third parties.

But really, all you need to do is read to and/or write from files. The trick is, you need to be able to do it using the binary format that I mentioned above. If you just use a tool that writes simple strings, it will improperly pad the fields, resulting in mangled transfers. So, you’ll need a bit of proficiency in whatever tool you use. The tool itself doesn’t matter, though. Perl, Python, bash scripts,… anything will do. Just remember these guidelines:

  • Writing to files _0, _2, _3, and _4 just wastes time. The host will never see it, it will break KVP clients, and the files’ contents will be reset when the daemon restarts.
  • You do not need special permission to read from any of the files.
  • _1 is the only file that it’s useful to write to. You can, of course, read from it.
    • Deleting the existing contents deletes those KVPs. You probably want to update existing or append new.
    • The host only receives the LAST time that a KVP is set. Meaning that if you write a KVP with key “NewKey” twice in the _1 file, the host will only receive the second one.
    • Delete a KVP by zeroing its fields.
  • If the byte lengths are not honored properly, you will damage that KVP and every KVP following.

Source Code for a Hyper-V KVP Exchange Utility on Linux

I’ve built a small utility that can be used to read, write, and delete Hyper-V KVPs on Linux. I wrote it in C++ so that it can be compiled into a single, neat executable.

The long-term goal is to move this to my Github account. I have a number of things on my to-do list before that can happen. However, any updates will be published there. The listing on this article will be forever locked in a 1.0 state.

Compile Instructions

Each file is set so that they all live in the same directory. Use
make to build the sources and
sudo make install to put the executable into the /bin folder.

Install Instructions

Paste the contents of all of these files into accordingly-named files. File names are in the matching section header and in the code preview bar.

Transfer all of the files to your Linux system. It doesn’t really matter where. They just need to be in the same folder.

Run:

Usage Instructions

Get help with:

  • hvkvp –help
  • hvkvp read –help
  • hvkvp write –help
  • hvkvp delete –help

Each includes the related keys for that command and some examples.

Code Listing

The file list:

  • makefile
  • main.cpp
  • hvkvp.h
  • hvkvp.cpp
  • hvkvpfile.h
  • hvkvpfile.cpp
  • hvkvpreader.h
  • hvkvpreader.cpp
  • hvkvpremover.h
  • hvkvpremover.cpp
  • hvkvpwriter.h
  • hvkvpwriter.cpp

makefile

main.cpp

 

hvkvp.h

 

hvkvp.cpp

 

hvkvpfile.h

 

hvkvpfile.cpp

 

hvkvpreader.h

 

hvkvpreader.cpp

 

hvkvpremover.h

 

hvkvpremover.cpp

 

hvkvpwriter.h

 

hvkvpwriter.cpp

More in this series:

Part 1: Explanation

Part 2: Implementation

Have any questions or feedback?

Leave a comment below!