Tag Archives: matured

DevOps security shifts left, but miles to go to pass hackers

DevOps security processes have matured within enterprises over the last year, but IT shops still have far to go to stem the tide of data breaches.

DevOps teams have built good security habits almost by default as they have increased the frequency of application releases and adopted infrastructure and security automation to improve software development. More frequent, smaller, automated app deployments are less risky and less prone to manual error than large and infrequent ones.

Microservices management and release automation demand tools such as infrastructure as code and configuration management software to manage infrastructure, which similarly cut down on human error. Wrapped up into a streamlined GitOps process, Agile and DevOps techniques automate the path to production while locking down access to it — a win for both security and IT efficiency.

However, the first six months of 2019 saw such a flood of high-profile data breaches that at least one security research firm called it the worst year on record. And while cybersecurity experts aren’t certain how trustworthy that measurement is — there could just be more awareness of breaches than there used to be, or more digital services to attack than in past years — they feel strongly that DevOps security teams still aren’t staying ahead of attackers, who have also learned to automate and optimize what they do.

Adrian Sanabria, advocate at Thinkst Applied ResearchAdrian Sanabria

“The attackers have innovated, and that’s one of the problems with our industry — we’re at least five years behind the attackers,” said Adrian Sanabria, advocate at Thinkst Applied Research, a cybersecurity research and software firm based in South Africa. “We’re in a mode where we’re convinced, with all this VC money and money spent on marketing, that we have to wait for a product to be available to solve these problems … and they’re never going to be ready in time.”

DevOps security tools aren’t enough

A cybersecurity tool is only as good as how it’s used, Sanabria said, citing the example of a Target breach in 2013, where security software detected potentially malicious activity, but IT staff didn’t act on its warnings. In part, this was attributed to alert fatigue, as IT teams increasingly deal with a fire hose of alerts from various monitoring systems. But it also has to do with IT training, Sanabria said.

“In the breach research I’ve done, generally everyone owned [the tools] they needed to own,” he said. “They either didn’t know how to use it, hadn’t set it up correctly, or they had some kind of process issue where the [tools] did try to stop the attacks or warn them of it, [but] they either didn’t see the alert or didn’t act on the alert.”

The attackers have innovated, and that’s one of the problems with our industry — we’re at least five years behind the attackers.
Adrian SanabriaAdvocate, Thinkst Applied Research

DevOps security, or DevSecOps, teams have locked down many of the technical weak points within infrastructure and app deployment processes, but all too often, the initial attack takes a very human form, such as a spoofed email that seems to come from a company executive, directing the recipient to transfer funds to what turns out to be an attacker’s account.

“Often, breaches don’t even require hacking,” Sanabria said. “It requires understanding of financial processes, who’s who in the company and the timing of certain transactions.”

Preventing such attacks requires that employees be equally familiar with that information, Sanabria said. That lack of awareness is driving a surge in ransomware attacks, which rely almost entirely on social engineering to hold vital company data hostage.

Collaboration and strategy vital for DevOps security

Thus, in a world of sophisticated technology, the biggest problems remain human, according to experts — and their solutions are also rooted in organizational dynamics and human collaboration, starting with a more strategic, holistic organizational approach to IT security.

Jeremy Pullen, PolodisJeremy Pullen

“Technology people don’t think of leadership skills and collaboration as primary job functions,” said Jeremy Pullen, CEO of Polodis, a digital transformation consulting firm in Atlanta. “They think the job is day-to-day technical threat remediation, but you can’t scale your organization when you have people trying to do it all themselves.”

An overreliance on individual security experts within enterprises leads to a ‘lamppost effect,’ where those individuals overcompensate for risks they’re familiar with, but undercompensate in areas they don’t understand as well, Pullen said. That kind of team structure also results in the time-honored DevOps bugaboo of siloed responsibilities, which increases security fragility in the same way it dampens application performance and infrastructure resilience.

“Developers and operations may be blind to application security issues, while security tends to focus on physical and infrastructure security, which is most clearly defined in their threat models,” Pullen said. “Then it becomes a bit of a game of Whac-a-Mole … where you’re trying to fix one thing and then another thing pops up, and it gets really noisy.”

Instead, DevSecOps teams must begin to think of themselves and their individual job functions as nodes in a network rather than layers of a stack, Pullen said, and work to understand how the entire organization fits together.

“Everyone’s unclear about what enterprise architecture is,” he said. “They stick Jenkins in the middle of a process but might not understand that they need to separate that environment into different domains and understand governance boundaries.”

Effective DevOps security requires more team practice

Strategically hardening applications and IT management processes to prevent attacks is important, but organizations must also strategically plan — and practice — their response to ongoing security incidents that can and will still happen.

“Cybersecurity so far has been focused on solitary study and being the best technical practitioner you can be, and building stand-alone applications and infrastructure to the best technical standard, which reminds me of golf,” said Nick Drage, principal consultant at Path Dependence Ltd., a cybersecurity consulting firm based in the U.K., in a presentation at DevSecCon in Seattle last month. “But in reality, cybersecurity is a fight with an opponent over territory — much more like American football.”

As long as security is practiced by isolated individuals, it will be as effective as taking the football field armed with golf clubs, Drage said. Instead, the approach should be more team-oriented, cooperative, and, especially, emphasize team practice to prepare for ‘game time.’

This is the future of governance — controlling risk on the human side of our systems.
Charles BetzAnalyst, Forrester Research

American football defenses are particularly instructive for DevOps security strategy ideas about defense in depth, Drage said in his presentation. Among other things, they demonstrate that an initial incursion into a team’s territory — yards gained — does not amount to a breach — points scored. IT teams should also apply that thinking as they try to anticipate and respond to threats — how to protect the ‘end zone,’ so to speak, and not just their half of the field.

Thinkst’s Sanabria uses a different analogy — the DevOps security team as firefighters.

“We’re not going to get good at this if we don’t practice it,” he said. “We buy all the tools, but imagine firefighters if they’d never donned the suits, never driven the truck, never used the hose and they’re not expecting the amount of force and it knocks them down. Going out to their first fire would look like a comedy.”

And yet that’s exactly what happens with many enterprise IT security teams when they must respond to incidents, Sanabria said, in part because companies don’t prioritize experiential learning over informational training.

The good news is that IT analysts expect the next wave of DevOps security to look very much like chaos engineering used in many organizations to improve system resiliency, but with a human twist. Organizations have begun to emerge such as OpenSOC, which sets up training workshops, including simulated ransomware attacks, for companies to practice security incident response. Companies can also do this internally by treating penetration tests as real attacks, otherwise known as red teaming. Free and open source tools such as Infection Monkey from Guardicore Labs also simulate attack scenarios.

Charles Betz, Forrester ResearchCharles Betz

Tech companies such as such as Google already practice their own form of human-based chaos testing, where employees are selected at random for a ‘staycation,’ directed to take a minimum of one hour to answer work emails, or to intentionally give wrong answers to questions, to test the resiliency of the rest of the organization.

“Despite the implications of the word ‘chaos,’ some companies are already presenting chaos engineering to their risk management leaders and auditors,” said Charles Betz, analyst at Forrester Research. “This is the future of governance — controlling risk on the human side of our systems.”

Go to Original Article

Kubernetes authentication project wrestles with migration problems

Kubernetes has not only matured, it’s showing a wrinkle or two.

In case there was any doubt that Kubernetes is no longer a fledgling project, it has begun to show its age with the accumulation of technical debt, as upstream developers struggle to smooth the migration process for a core set of security improvements.

Kubernetes authentication and role-based access control (RBAC) have improved significantly since the project reached version 1.0 four years ago. But one aspect of Kubernetes authentication management remains stuck in the pre-1.0 era: the management of access tokens that secure connections between Kubernetes pods and the Kubernetes API server.

Workload Identity for Google Kubernetes Engine (GKE), released last month, illuminated this issue. Workload Identity is now Google’s recommended approach to Kubernetes authentication between containerized GKE application services and GCP utilities because Workload Identity users no longer have to manage Kubernetes secrets or Google Cloud IAM service account keys themselves. Workload Identity also manages secrets rotation, and can set fine-grained secrets expiration and intended audience policies that ensure good Kubernetes security hygiene.

GKE users are champing at the bit to use Workload Identity because it could eliminate the need to manage this aspect of Kubernetes authentication with a third-party tool such as HashiCorp’s Vault, or worse, with error-prone manual techniques.

“Without this, you end up having to manage a bunch of secrets and access [controls] yourself, which creates lots of opportunities for human error, which has actually bitten us in the past,” said Josh Koenig, head of product at Pantheon Platform, a web operations platform in San Francisco that hosts more than 200,000 Drupal and WordPress sites.

Koenig recalled an incident where a misconfigured development cluster could have exposed an SSL certificate. Pantheon’s IT team caught the mistake, and tore down and reprovisioned the cluster. Pantheon also uses Vault in its production clusters to guard against such errors for more critical workloads but didn’t want that tool’s management overhead to slow down provisioning in the development environment.

“It’s a really easy mistake to make, even with the best of intentions, and the best answer to that is just not having to manage security as part of your codebase,” Koenig said. “There are ways to do that that are really expensive and heavy to implement, and then there’s Google doing it for you [with Workload Identity].”

Kubernetes authentication within clusters lags Workload Identity

Workload Identity uses updated Kubernetes authentication tokens, available in beta since Kubernetes 1.12, to more effectively and efficiently authenticate Kubernetes-based services as they interact with other Google Cloud Platform (GCP) services. Such communications are much more likely than communications within Kubernetes clusters to be targeted by attackers, and there are other ways to shore up security for intracluster communications, including using OpenID Connect tokens based on OAuth 2.0.

However, the SIG-Auth group within the Kubernetes upstream community is working to bring Kubernetes authentication tokens that pass between Kubernetes Pods up to speed with the Workload Identity standard, with automatic secrets rotation, expiration and limited-audience policies. This update would affect all Kubernetes environments, including those that run in GKE, and could allow cloud service providers to assume even more of the Kubernetes authentication management burden on behalf of users.

We think [API Server authentication tokens] improves the default security posture of Kubernetes open source.
Mike DaneseSoftware engineer, Google; chair, Kubernetes SIG-Auth

“Our legacy tokens were only meant for use against the Kubernetes API Server, and using those tokens against something like Vault or Google poses some potential escalations or security risks,” said Mike Danese, a Google software engineer and the chair of Kubernetes SIG-Auth. “We’ll push on [API Server authentication tokens] a little bit because we think it improves the default security posture of Kubernetes open source.”

There’s a big problem, however. A utility to migrate older Kubernetes authentication tokens used with the Kubernetes API server to the new system, dubbed Bound Service Account Token Volumes and available in alpha since Kubernetes 1.13, has hit a show-stopping snag. Early users have encountered a compatibility issue when they try to migrate to the new tokens, which require a new kind of storage volume for authentication data that can be difficult to configure.

Without the new tokens, Kubernetes authentication between pods and the API Server could theoretically face the same risk of human error or mismanagement as GKE services before Workload Identity was introduced, IT experts said.

“The issues that are discussed as the abilities of Workload Identity — to reduce blast radius, require the refresh of tokens, enforce a short time to live and bind them to a single pod — would be the potential impact for current Kubernetes users” if the tokens aren’t upgraded eventually, said Chris Riley, DevOps delivery director at CPrime Inc., an Agile software development consulting firm in Foster City, Calif.

Kubernetes authentication upgrade beta delayed

After Kubernetes 1.13 was rolled out in December 2018, reports indicated that a beta release for Bound Service Account Token Volumes would arrive by Kubernetes 1.15, which was released in June 2019, or Kubernetes 1.16, due in September 2019.

However, the feature did not reach beta in 1.15, and is not expected to reach beta in 1.16, Danese said. It’s also still unknown how the eventual migration to new Kubernetes authentication tokens for the API Server will affect GKE users.

“We could soften our stance on some of the improvements to break legacy clients in fewer ways — for example, we’ve taken a hard stance on file permissions, which disrupted some legacy clients, binding specifically to the Kubernetes API server,” Danese said. “We have some subtle interaction with other security features like Pod Security Policy that we could potentially pave over by making changes or allowances in Pod Security Policies that would allow old Pod Security Policies to use the new tokens instead of the legacy tokens.”

Widespread community attention to this issue seems limited, Riley said, and it could be some time before it’s fully addressed.

However, he added, it’s an example of the reasons Kubernetes users among his consulting clients increasingly turn to cloud service providers to manage their complex container orchestration environment.

“The value of services like Workload Identity is the additional integration these providers are doing into other cloud services, simplifying the adoption and increasing overall reliability of Kubernetes,” Riley said. “I expect them to do the hard work and support migration or simple adoption of future technologies.”

Go to Original Article