Tag Archives: Kubernetes

Kasten backup aims for secure Kubernetes protection

People often talk about Kubernetes “Day 1,” when you get the platform up and running. Now Kasten wants to help with “Day 2.”

Kasten’s K10 is a data management and backup platform for Kubernetes. The latest release, K10 2.0, focuses on security and simplicity.

K10 2.0 includes support for Kubernetes authentication, role-based access control, OpenID Connect, AWS Identity and Access Management roles, customer-managed keys, and integrated encryption of artifacts at rest and in flight.

“Once you put data into storage, the Day 2 operations are critical,” said Krishnan Subramanian, chief research advisor at Rishidot Research. “Day 2 is as critical as Day 1.”

Day 2 — which includes data protection, mobility, backup and restore, and disaster recovery — is becoming a pain point for Kubernetes users, Kasten CEO Niraj Tolia said.

“In 2.0, we are focused on making Kubernetes backup easy and secure,” Tolia said.

Other features the new Kasten backup software offers, which became generally available earlier in November, include a Kubernetes-native API, auto-discovery of the application environment, policy-driven operations, multi-tenancy support, and advanced logging and monitoring. The Kasten backup enables teams to operate their environments, while supporting developers’ ability to use tools of their choice, according to the vendor.

Kasten K10 dashboard screenshot
Kasten K10 provides data management and backup for Kubernetes.

Kasten backup eyes market opportunity

Kasten, which launched its original product in December 2017, generally releases an update to its customers every two weeks. A typical update that’s not as major as 2.0 typically has bug fixes, new features and increased depth in current features. Tolia said there were 55 releases between 1.0 and 2.0.

Day 2 is as critical as Day 1.
Krishnan SubramanianFounder and chief research advisor, Rishidot Research

Backup for container storage has become a hot trend in data protection. Kubernetes specifically is an open source system used to manage containers across private, public and hybrid cloud environments. Kubernetes can be used to manage microservice architectures and is deployable on most cloud providers.

“Everyone’s waking up to the fact that this is going to be the next VMware,” as in, the next infrastructure of choice, Tolia said.

Kubernetes backup products are popping up, but it looks like Kasten is a bit ahead of its time, Rishidot’s Subramanian said. He said he is seeing more enterprises using Kubernetes in production, for example, in moving legacy workloads to the platform, and that makes backup a critical element.

“Kubernetes is just starting to take off,” Subramanian said.

Kubernetes backup “has really taken off in the last two or three quarters,” Tolia said.

Subramanian said he is starting to see legacy vendors such as Dell EMC and NetApp tackling Kubernetes backup, as well as smaller vendors such as Portworx and Robin. He said Kasten had needed stronger security but caught up with K10 2.0. Down the road, he said he will look for Kasten to improve its governance and analytics.

Tolia said Kasten backup stands out because it’s “purpose-built for Kubernetes” and extends into multilayered data management.

In August, Kasten, which is based in Los Altos, Calif., closed a $14 million Series A funding round, led by Insight Partners. Tolia did not give Kasten’s customer count but said it has deployments across multiple continents.

Go to Original Article
Author:

Kubernetes tools vendors vie for developer mindshare

SAN DIEGO — The notion that Kubernetes solves many problems as a container orchestration technology belies the complexity it adds in other areas, namely for developers who need Kubernetes tools.

Developers at the KubeCon + CloudNativeCon North America 2019 event here this week noted that although native tooling for development on Kubernetes continues to improve, there’s still room for more.

“I think the tooling thus far is impressive, but there is a long way to go,” said a software engineer and Kubernetes committer who works for a major electronics manufacturer and requested anonymity.

Moreover, “Kubernetes is extremely elegant, but there are multiple concepts for developers to consider,” he said. “For instance, I think the burden of the onboarding process for new developers and even users sometimes can be too high. I think we need to build more tooling, as we flush out the different use cases that communities bring out.”

Developer-oriented approach

Enter Red Hat, which introduced an update of its Kubernetes-native CodeReady Workspaces tool at event.

Red Hat CodeReady Workspaces 2 enables developers to build applications and services on their laptops that mirror the environment they will run in production. And onboarding is but one of the target use cases for the technology, said Brad Micklea, vice president of developer tools, developer programs and advocacy at Red Hat.

The technology is especially useful in situations where security is an issue, such as bringing in new contracting teams or using offshore development teams where developers need to get up and running with the right tools quickly.

I think the tooling thus far is impressive, but there is a long way to go.
Anonymous Kubernetes committer

CodeReady Workspaces runs on the Red Hat OpenShift Kubernetes platform.

Initially, new enterprise-focused developer technologies are generally used in experimental, proof-of-concept projects, said Charles King, an analyst at Pund-IT in Hayward, Calif. Yet over time those that succeed, like Kubernetes, evolve from the proof-of-concept phase to being deployed in production environments.

“With CodeReady Workspaces 2, Red Hat has created a tool that mirrors production environments, thus enabling developers to create and build applications and services more effectively,” King said. “Overall, Red Hat’s CodeReady Workspaces 2 should make life easier for developers.”

In addition to popular features from the first version, such as an in-browser IDE, Lightweight Directory Access Protocol support, Active Directory and OpenAuth support as well as one-click developer workspaces, CodeReady Workspaces 2 adds support for Visual Studio Code extensions, a new user interface, air-gapped installs and a shareable workspace configuration known as Devfile.

“Workspaces is just generally kind of a way to package up a developer’s working workspace,” Red Hat’s Micklea said.

Overall, the Kubernetes community is primarily “ops-focused,” he said. However, tools like CodeReady Workspaces help to empower both developers and operations.

For instance, at KubeCon, Amr Abdelhalem, head of the cloud platform at Fidelity Investments, said the way he gets teams initiated with Kubernetes is to have them deliver on small projects and move on from there. CodeReady Workspaces is ideal for situations like that because it simplifies developer adoption of Kubernetes, Micklea said.

Such a tool could be important for enterprises that are banking on Kubernetes to move them into a DevOps model to achieve business transformation, said Charlotte Dunlap, an analyst with GlobalData.

“Vendors like Red Hat are enhancing Kubernetes tools and CLI [Command Line Interface] UIs to bring developers with more access and visibility into the ALM [Application Lifecycle Management] of their applications,” Dunlap said. “Red Hat CodeReady Workspaces is ultimately about providing enterprises with unified management across endpoints and environments.”

Competition for Kubernetes developer mindshare

Other companies that focus on the application development platform, such as IBM and Pivotal, have also joined the Kubernetes developer enablement game.

Earlier this week, IBM introduced a set of new open-source tools to help ease developers’ Kubernetes woes. Meanwhile, at KubeCon this week, Pivotal made its Pivotal Application Service (PAS) on Kubernetes generally available and also delivered a new release of the alpha version of its Pivotal Build Service. The PAS on Kubernetes tool enables developers to focus on coding while the platform automatically handles software deployment, networking, monitoring, and logging.

The Pivotal Build Service enables developers to build containers from source code for Kubernetes, said James Watters, senior vice president of strategy at Pivotal. The service automates container creation, management and governance at enterprise scale, he said.

The build service brings technologies such as Pivotal’s kpack and Cloud Native Buildpacks to the enterprise. Cloud Native Buildpacks address dependencies in the middleware layer, such as language-specific frameworks. Kpack is a set of resource controllers for Kubernetes. The Build Service defines the container image, its contents and where it should be kept, Watters said.

Indeed, Watters said he believes it just might be game over in the Kubernetes tools space because Pivotal owns the Spring Framework and Spring Boot, which appeal to a wide swath of Java developers, which is “one of the most popular ways enterprises build applications today,” he said.

“There is something to be said for the appeal of Java in that my team would not need to make wholesale changes to our build processes,” said a Java software developer for a financial services institution who requested anonymity because he was not cleared to speak for the organization.

Yet, in today’s polyglot programming world, programming language is less of an issue as teams have the capability to switch languages at will. For instance, Fidelity’s Abdelhalem said his teams find it easier to move beyond a focus strictly on tools and more on overall technology and strategy to determine what fits in their environment.

Go to Original Article
Author:

Kubernetes Helm Tiller is dead, and IT pros rejoice

SAN DIEGO — The death of Kubernetes Helm Tiller in version 3 was the talk of the cloud-native world here at KubeCon + CloudNativeCon North America 2019 this week, as the change promises better security and stability for a utility that underpins several other popular microservices management and GitOps tools.

Kubernetes Helm is a package manager used to deploy apps to the container orchestration platform. It’s widely used to deploy enterprise apps to containers through CI/CD pipelines, including GitOps and progressive delivery tools. It’s also a key component for installing and updating the custom resource definitions (CRDs) that underpin the Istio service mesh in upstream environments.

Helm Tiller was a core component of the software in its initial releases, which used a client-server architecture for which Tiller was the server. Helm Tiller acted as an intermediary between users and the Kubernetes API server, and handled role-based access control (RBAC) and the rendering of Helm charts for deployment to the cluster. With the first stable release of Helm version 3 on Nov. 13, however, Tiller was removed entirely, and Helm version 3 now communicates directly with the Kubernetes API Server.

Such was the antipathy for Helm Tiller among users that when maintainers proclaimed the component’s death from the KubeCon keynote stage here this week, it drew enthusiastic cheers.

“At the first Helm Summit in 2018, there was quite a lot of input from the community, especially around, ‘Can we get rid of Tiller?'” said Martin Hickey, a senior software engineer at IBM and a core maintainer of Helm, in a presentation on Helm version 3 here. “[Now there’s] no more Tiller, and the universe is safe again.”

KubeCon Helm keynote
News of Helm Tiller’s demise from the KubeCon keynote stage this week drew cheers from the audience.

Helm Tiller had security and stability issues

IT pros who used previous versions of Helm charts said the client-server setup between Helm clients and Tiller was buggy and unstable, which made it even more difficult to install already complex tools such as Istio service mesh for upstream users.

“Version 3 offers new consistency in the way it handles CRDs, which had weird dependency issues that we ran into with Istio charts,” said Aaron Christensen, principal software engineer at SPS Commerce, a communications network for supply chain and logistics businesses in Minneapolis. “It doesn’t automatically solve the problem, but if the Istio team makes use of version 3, it could really simplify deployments.”

[Now there’s] no more Tiller, and the universe is safe again.
Martin HickeySenior software engineer, IBM and a core maintainer of Helm

Helm Tiller was designed before Kubernetes had its own RBAC features, but once these were added to the core project, Tiller also became a cause for security concerns among enterprises. From a security perspective, Tiller had cluster-wide access and could potentially be used for privilege escalation attacks if not properly secured.

It was possible to lock down Helm Tiller in version 2 — heavily regulated firms such as Fidelity Investments were able to use it in production with a combination of homegrown tools and GitOps utilities from Weaveworks. But the complexity of that task and Helm Tiller stability problems meant some Kubernetes shops stayed away from Helm altogether until now, which led to other problems with rolling out apps on container clusters.

“Helm would issue false errors to our CI/CD pipelines, and say a deployment failed when it didn’t, or it would time out connecting to the Kubernetes API server, which made the deployment pipeline fail,” said Carlos Traitel, senior DevOps engineer at Primerica, a financial services firm in Duluth, Ga.

Primerica tried to substitute kube-deploy, a different open source utility for Helm, but also ran into management complexity with it. Primerica engineers plan to re-evaluate Helm version 3 as soon as possible. The new version uses a three-way merge process for updates, which compares the desired state with the actual state of the cluster along with the changes users want to apply, and could potentially eliminate many common errors during the Helm chart update process.

Despite its difficulties, Helm version 2 was a crucial element of Kubernetes management, SPS’s Christensen said.

“It worked way more [often] than it didn’t — we wouldn’t go back and use something else,” he said. “It helps keep 20-plus resources consistent across our clusters … and we were also able to implement our own automated rollbacks based on Helm.”

Go to Original Article
Author:

Kubernetes security opens a new frontier: multi-tenancy

SAN DIEGO — As enterprises expand production container deployments, a new Kubernetes security challenge has emerged: multi-tenancy.

Among the many challenges with multi-tenancy in general is that it is not easy to define, and few IT pros agree on a single definition or architectural approach. Broadly speaking, however, multi-tenancy occurs when multiple projects, teams or tenants, share a centralized IT infrastructure, but remain logically isolated from one another.

Kubernetes multi-tenancy also adds multilayered complexity to an already complex Kubernetes security picture, and demands that IT pros wire together a stack of third-party and, at times, homegrown tools on top of the core Kubernetes framework.

This is because core upstream Kubernetes security features are limited to service accounts for operations such as role-based access control — the platform expects authentication and authorization data to come from an external source. Kubernetes namespaces also don’t offer especially granular or layered isolation by default. Typically, each namespace corresponds to one tenant, whether that tenant is defined as an application, a project or a service.

“To build logical isolation, you have to add a bunch of components on top of Kubernetes,” said Karl Isenberg, tech lead manager at Cruise Automation, a self-driving car service in San Francisco, in a presentation about Kubernetes multi-tenancy here at KubeCon + CloudNativeCon North America 2019 this week. “Once you have Kubernetes, Kubernetes alone is not enough.”

Karl Isenberg, Cruise Automation
Karl Isenberg, tech lead manager at Cruise Automation, presents at KubeCon about multi-tenant Kubernetes security.

However, Isenberg and other presenters here said Kubernetes multi-tenancy can have significant advantages if done right. Cruise, for example, runs very large Kubernetes clusters, with up to 1,000 nodes, shared by thousands of employees, teams, projects and some customers. Kubernetes multi-tenancy means more highly efficient clusters and cost savings on data center hardware and cloud infrastructure.

“Lower operational costs is another [advantage] — if you’re starting up a platform operations team with five people, you may not be able to manage five [separate] clusters,” Isenberg added. “We [also] wanted to make our investments in focused areas, so that they applied to as many tenants as possible.”

Multi-tenant Kubernetes security an ad hoc practice for now

The good news for enterprises that want to achieve Kubernetes multi-tenancy securely is that there are a plethora of third-party tools they can use to do it, some of which are sold by vendors, and others open sourced by firms with Kubernetes development experience, including Cruise and Yahoo Media.

Duke Energy Corporation, for example, has a 60-node Kubernetes cluster in production that’s stretched across three on-premises data centers and shared by 100 web applications so far. The platform is comprised of several vendors’ products, from Diamanti hyper-converged infrastructure to Aqua Security Software’s container firewall, which logically isolates tenants from one another at a granular level that accounts for the ephemeral nature of container infrastructure.

“We don’t want production to talk to anyone [outside of it],” said Ritu Sharma, senior IT architect at the energy holding company in Charlotte, N.C., in a presentation at KubeSec Enterprise Summit, an event co-located with KubeCon this week. “That was the first question that came to mind — how to manage cybersecurity when containers can connect service-to-service within a cluster.”

Some Kubernetes multi-tenancy early adopters also lean on cloud service providers such as Google Kubernetes Engine (GKE) to take on parts of the Kubernetes security burden. GKE can encrypt secrets in the etcd data store, which became available in Kubernetes 1.13, but isn’t enabled by default, according to a KubeSec presentation by Mike Ruth, one of Cruise’s staff security engineers.

Google also offers Workload Identity, which matches up GCP identity and access management with Kubernetes service accounts so that users don’t have to manage Kubernetes secrets or Google Cloud IAM service account keys themselves. Kubernetes SIG-Auth looks to modernize how Kubernetes security tokens are handled by default upstream to smooth Kubernetes secrets management across all clouds, but has run into snags with the migration process.

In the meantime, Verizon’s Yahoo Media has donated a project called Athenz to open source, which handles multiple aspects of authentication and authorization in its on-premises Kubernetes environments, including automatic secrets rotation, expiration and limited-audience policies for intracluster communication similar to those offered by GKE’s Workload Identity. Cruise also created a similar open source tool called RBACSync, along with Daytona, a tool that fetches secrets from HashiCorp Vault, which Cruise uses to store secrets instead of in etcd, and injects them into running applications, and k-rail for workload policy enforcement.

Kubernetes Multi-Tenancy Working Group explores standards

While early adopters have plowed ahead with an amalgamation of third-party and homegrown tools, some users in highly regulated environments look to upstream Kubernetes projects to flesh out more standardized Kubernetes multi-tenancy options.

For example, investment banking company HSBC can use Google’s Anthos Config Management (ACM) to create hierarchical, or nested, namespaces, which make for more highly granular access control mechanisms in a multi-tenant environment, and simplifies their management by automatically propagating shared policies between them. However, the company is following the work of a Kubernetes Multi-Tenancy Working Group established in early 2018 in the hopes it will introduce free open source utilities compatible with multiple public clouds.

Sanjeev Rampal, Kubernetes Multi-Tenancy Working Group
Sanjeev Rampal, co-chair of the Kubernetes Multi-Tenancy Working Group, presents at KubeCon.

“If I want to use ACM in AWS, the Anthos license isn’t cheap,” said Scott Surovich, global container engineering lead at HSBC, in an interview after a presentation here. Anthos also requires VMware server virtualization, and hierarchical namespaces available at the Kubernetes layer could offer Kubernetes multi-tenancy on bare metal, reducing the layers of abstraction and potentially improving performance for HSBC.

Homegrown tools for multi-tenant Kubernetes security won’t fly in HSBC’s highly regulated environment, either, Surovich said.

“I need to prove I have escalation options for support,” he said. “Saying, ‘I wrote that’ isn’t acceptable.”

So far, the working group has two incubation projects that create custom resource definitions — essentially, plugins — that support hierarchical namespaces and virtual clusters that create self-service Kubernetes API Servers for each tenant. The working group has also created working definitions of the types of multi-tenancy and begun to define a set of reference architectures.

The working group is also considering certification of multi-tenant Kubernetes security and management tools, as well as benchmark testing and evaluation of such tools, said Sanjeev Rampal, a Cisco principal engineer and co-chair of the group.

Go to Original Article
Author:

Google Cloud tackles Spark on Kubernetes

An early version of a Google Cloud service that runs Apache Spark on Kubernetes is now available, but more work will be required to flesh out the container orchestration platform’s integrations with data analytics tools.

Kubernetes and containers haven’t been renowned for their use in data-intensive, stateful applications, including data analytics. But there are benefits to using Kubernetes as a resource orchestration layer under applications such as Apache Spark rather than the Hadoop YARN resource manager and job scheduling tool with which it’s typically associated. Developers and IT ops stand to gain advantages that containers bring to any application, such as portability across systems and consistency in configuration, along with automated provisioning and scaling for workloads that’s handled in the Kubernetes layer or by Helm charts, as well as container resource efficiency compared with virtual or bare metal machines.

“Analytical workloads, in particular, benefit from the ability to add rapidly scalable cloud capacity for spiky peak workloads, whereas companies might want to run routine, predictable workloads in a virtual private cloud,” said Doug Henschen, an analyst at Constellation Research in Cupertino, Calif. 

Google, which offers managed versions of Apache Spark and Apache Hadoop that run on YARN through its Cloud Dataproc service, would prefer to use its own Kubernetes platform to orchestrate resources — and to that end, released an alpha preview integration for Spark on Kubernetes within Cloud Dataproc this week. Other companies, such as Databricks (run by the creators of Apache Spark) and D2iQ (formerly Mesosphere), support Spark on Kubernetes, but Google Cloud Dataproc stands to become the first of the major cloud providers to include it in a managed service.

Customers don’t care about managing Hive or Pig, and want to use Kubernetes in hybrid clouds.
James MaloneProduct manager, Google Cloud Dataproc

Apache Spark has had a native Kubernetes scheduler since version 2.3, and Hadoop added native container support in Hadoop 3.0.3, both released in May 2018. However, Hadoop’s container support is still tied to HDFS and is too complex, in Google’s view.

“People have gotten Docker containers running on Hadoop clusters using YARN, but Hadoop 3’s container support is probably about four years too late,” said James Malone, product manager for Cloud Dataproc at Google. “It also doesn’t really solve the problems customers are trying to solve, from our perspective — customers don’t care about managing [Apache data warehouse and analytics apps] Hive or Pig, and want to use Kubernetes in hybrid clouds.”

Spark on Kubernetes only scratches the surface of big data integration

Cloud Dataproc’s Spark on Kubernetes implementation remains in a very early stage, and will require updates upstream to Spark as well as Kubernetes before it’s production-ready. Google also has its sights set on support for more Apache data analytics apps, including the Flink data stream processing framework, Druid low-latency data query system and Presto distributed SQL query engine.

“It’s still in alpha, and that’s by virtue of the fact that the work that we’ve done here has been split into multiple streams,” Malone said. One of those workstreams is to update Cloud Dataproc to run Kubernetes clusters. Another is to contribute to the upstream Spark Kubernetes operator, which remains in the experimental stage within Spark Core. Finally, Cloud Dataproc must brush up performance enhancement add-ons such as external shuffle service support, which aids in the dynamic allocation of resources.

For now, IT pros who want to run Spark on Kubernetes must assemble their own integrations among the upstream Spark Kubernetes scheduler, supported Spark from Databricks, and Kubernetes cloud services. Customers that seek hybrid cloud portability for Spark workloads must also implement a distributed storage system from vendors such as Robin Systems or Portworx. All of it can work, but without many of the niceties about fully integrated cloud platform services that would make life easier.

For example, the Spark Kubernetes executor uses Python, rather than the Scala programming language, which is a bit trickier to use.

“The Python experience of Spark in Kubernetes has always lagged the Scala experience, mostly because deploying a compiled artifact in Scala is just easier logistically than pulling in dependencies for Python jobs,” said Michael Bishop, co-founder and board member at Alpha Vertex, a New York-based fintech startup that uses machine learning deployed in a multi-cloud Kubernetes infrastructure to track market trends for financial services customers. “This is getting better and better, though.”

There also remain fundamental differences between Spark’s job scheduler and Kubernetes that must be smoothed out, Bishop said.

“There is definitely an impedance [between the two schedulers,” he said. “Spark is intimately aware of ‘where’ is for [nodes], while Kubernetes doesn’t really care beyond knowing a pod needs a particular volume mounted.”

Google will work on sanding down these rough edges, Malone pledged.

“For example, we have an external shuffle service, and we’re working hard to make it work with both YARN and Kubernetes Spark,” he said.

Go to Original Article
Author:

Kubernetes authentication project wrestles with migration problems

Kubernetes has not only matured, it’s showing a wrinkle or two.

In case there was any doubt that Kubernetes is no longer a fledgling project, it has begun to show its age with the accumulation of technical debt, as upstream developers struggle to smooth the migration process for a core set of security improvements.

Kubernetes authentication and role-based access control (RBAC) have improved significantly since the project reached version 1.0 four years ago. But one aspect of Kubernetes authentication management remains stuck in the pre-1.0 era: the management of access tokens that secure connections between Kubernetes pods and the Kubernetes API server.

Workload Identity for Google Kubernetes Engine (GKE), released last month, illuminated this issue. Workload Identity is now Google’s recommended approach to Kubernetes authentication between containerized GKE application services and GCP utilities because Workload Identity users no longer have to manage Kubernetes secrets or Google Cloud IAM service account keys themselves. Workload Identity also manages secrets rotation, and can set fine-grained secrets expiration and intended audience policies that ensure good Kubernetes security hygiene.

GKE users are champing at the bit to use Workload Identity because it could eliminate the need to manage this aspect of Kubernetes authentication with a third-party tool such as HashiCorp’s Vault, or worse, with error-prone manual techniques.

“Without this, you end up having to manage a bunch of secrets and access [controls] yourself, which creates lots of opportunities for human error, which has actually bitten us in the past,” said Josh Koenig, head of product at Pantheon Platform, a web operations platform in San Francisco that hosts more than 200,000 Drupal and WordPress sites.

Koenig recalled an incident where a misconfigured development cluster could have exposed an SSL certificate. Pantheon’s IT team caught the mistake, and tore down and reprovisioned the cluster. Pantheon also uses Vault in its production clusters to guard against such errors for more critical workloads but didn’t want that tool’s management overhead to slow down provisioning in the development environment.

“It’s a really easy mistake to make, even with the best of intentions, and the best answer to that is just not having to manage security as part of your codebase,” Koenig said. “There are ways to do that that are really expensive and heavy to implement, and then there’s Google doing it for you [with Workload Identity].”

Kubernetes authentication within clusters lags Workload Identity

Workload Identity uses updated Kubernetes authentication tokens, available in beta since Kubernetes 1.12, to more effectively and efficiently authenticate Kubernetes-based services as they interact with other Google Cloud Platform (GCP) services. Such communications are much more likely than communications within Kubernetes clusters to be targeted by attackers, and there are other ways to shore up security for intracluster communications, including using OpenID Connect tokens based on OAuth 2.0.

However, the SIG-Auth group within the Kubernetes upstream community is working to bring Kubernetes authentication tokens that pass between Kubernetes Pods up to speed with the Workload Identity standard, with automatic secrets rotation, expiration and limited-audience policies. This update would affect all Kubernetes environments, including those that run in GKE, and could allow cloud service providers to assume even more of the Kubernetes authentication management burden on behalf of users.

We think [API Server authentication tokens] improves the default security posture of Kubernetes open source.
Mike DaneseSoftware engineer, Google; chair, Kubernetes SIG-Auth

“Our legacy tokens were only meant for use against the Kubernetes API Server, and using those tokens against something like Vault or Google poses some potential escalations or security risks,” said Mike Danese, a Google software engineer and the chair of Kubernetes SIG-Auth. “We’ll push on [API Server authentication tokens] a little bit because we think it improves the default security posture of Kubernetes open source.”

There’s a big problem, however. A utility to migrate older Kubernetes authentication tokens used with the Kubernetes API server to the new system, dubbed Bound Service Account Token Volumes and available in alpha since Kubernetes 1.13, has hit a show-stopping snag. Early users have encountered a compatibility issue when they try to migrate to the new tokens, which require a new kind of storage volume for authentication data that can be difficult to configure.

Without the new tokens, Kubernetes authentication between pods and the API Server could theoretically face the same risk of human error or mismanagement as GKE services before Workload Identity was introduced, IT experts said.

“The issues that are discussed as the abilities of Workload Identity — to reduce blast radius, require the refresh of tokens, enforce a short time to live and bind them to a single pod — would be the potential impact for current Kubernetes users” if the tokens aren’t upgraded eventually, said Chris Riley, DevOps delivery director at CPrime Inc., an Agile software development consulting firm in Foster City, Calif.

Kubernetes authentication upgrade beta delayed

After Kubernetes 1.13 was rolled out in December 2018, reports indicated that a beta release for Bound Service Account Token Volumes would arrive by Kubernetes 1.15, which was released in June 2019, or Kubernetes 1.16, due in September 2019.

However, the feature did not reach beta in 1.15, and is not expected to reach beta in 1.16, Danese said. It’s also still unknown how the eventual migration to new Kubernetes authentication tokens for the API Server will affect GKE users.

“We could soften our stance on some of the improvements to break legacy clients in fewer ways — for example, we’ve taken a hard stance on file permissions, which disrupted some legacy clients, binding specifically to the Kubernetes API server,” Danese said. “We have some subtle interaction with other security features like Pod Security Policy that we could potentially pave over by making changes or allowances in Pod Security Policies that would allow old Pod Security Policies to use the new tokens instead of the legacy tokens.”

Widespread community attention to this issue seems limited, Riley said, and it could be some time before it’s fully addressed.

However, he added, it’s an example of the reasons Kubernetes users among his consulting clients increasingly turn to cloud service providers to manage their complex container orchestration environment.

“The value of services like Workload Identity is the additional integration these providers are doing into other cloud services, simplifying the adoption and increasing overall reliability of Kubernetes,” Riley said. “I expect them to do the hard work and support migration or simple adoption of future technologies.”

Go to Original Article
Author:

Free Kubernetes security tools broaden enterprise choices

Kubernetes security tools have proliferated in 2018, and their growing numbers reflect increased maturity around container security among enterprise IT shops.

The latest additions to this tool category include a feature in Google Kubernetes Engine called Binary Authorization, which can create whitelists of container images and code that are authorized to run on GKE clusters. All other attempts to launch unauthorized apps will fail, and the GKE feature will document them.

Binary Authorization is in public beta. Google will also make the feature available for on-premises deployments through updates to Kritis, an open source project focused on deployment-time policy enforcement.

Aqua Security also added to the arsenal of Kubernetes security tools at IT pros’ disposal with an open source utility, called kube-hunter, which can be used for penetration testing of Kubernetes clusters. The tool performs passive scans of Kubernetes clusters to look for common vulnerabilities, such as dashboard and management server ports that were left open. These seemingly obvious errors have taken down high-profile companies, such as Tesla, Aviva and Gemalto.

Users can also perform active penetration tests with kube-hunter. In this scenario, the tool attempts to exploit the vulnerabilities it finds as if an attacker has gained access to Kubernetes cluster servers, which may highlight additional vulnerabilities in the environment.

Fernando Montenegro, analyst, 451 ResearchFernando Montenegro

These tools join several other Kubernetes security offerings introduced in 2018 — from Docker Enterprise Edition‘s encryption and secure container registry features for the container orchestration platform to Kubernetes support in tools from Qualys and Alert Logic. The growth of Kubernetes security tools indicates the container security conversation has shifted away from ways to secure individual container images and hosts to security at the level of the application and Kubernetes cluster.

“Containers are not foolproof, but container security is good enough for most users at this point,” said Fernando Montenegro, analyst with 451 Research. “The interest in the industry shifts now to how to do security at the orchestration layer and secure broader container deployments.”

GKE throws down the gauntlet for third-party container orchestration tools

The question for users, as cloud providers add these features, is, why go for a third-party tool when the cloud provider does this kind of thing themselves?
Fernando Montenegroanalyst, 451 Research

Google’s Binary Authorization feature isn’t unique; other on-premises and hybrid cloud Kubernetes tools, such as Docker Enterprise Edition, Mesosphere DC/OS and Red Hat OpenShift, offer similar capabilities to prevent unauthorized container launches on Kubernetes clusters.

However, third-party vendors once again find themselves challenged by a free and open source alternative from Google. Just as Kubernetes supplanted other container orchestration utilities, these additional Kubernetes management features further reduce third-party tools’ competitiveness.

GKE Binary Authorization is one of the first instances of a major cloud provider adding such a feature natively in its Kubernetes service, Montenegro said.

“[A gatekeeper for Kubernetes] is not something nobody’s thought of before, but I haven’t seen much done by other cloud providers on this front yet,” Montenegro said. AWS and Microsoft Azure will almost certainly follow suit.

“The question for users, as cloud providers add these features, is, why go for a third-party tool when the cloud provider does this kind of thing themselves?” Montenegro said.

Aqua Security’s penetration testing tool is unlikely to unseat full-fledged penetration testing tools enterprises use, such as Nmap and Burp Suite, but its focus on Kubernetes vulnerabilities specifically with a free offering will attract some users, Montenegro said.

Aqua Security and its main competitor, Twistlock, also must stay ahead of Kubernetes security features as they’re incorporated into broader enterprise platforms from Google, Cisco and others, Montenegro said.

IT pros debate upstream vs. packaged Kubernetes implementations

Packaged versions of Kubernetes promise ease of use for the finicky container orchestration platform, but some enterprises will stick with a DIY approach to Kubernetes implementation.

Red Hat, Docker, Heptio, Mesosphere, Rancher, Platform9, Pivotal, Google, Microsoft, IBM and Cisco are among the many enterprise vendors seeking to cash in on the container craze with prepackaged Kubernetes implementations for private and hybrid clouds. Some of these products — such Red Hat’s OpenShift Container Platform, Docker Enterprise Edition and Rancher’s eponymous platform — offer their own distribution of the container orchestration software, and most add their own enterprise security and management features on top of upstream Kubernetes code.

However, some enterprise IT shops still prefer to download Kubernetes source code from GitHub and leave out IT vendor middlemen.

“We’re seeing a lot of companies go with [upstream] Kubernetes over Docker [Enterprise Edition] and [Red Hat] OpenShift,” said Damith Karunaratne, director of client solutions for Indellient Inc., an IT consulting firm in Oakville, Ont. “Those platforms may help with management out of the gate, but software license costs are always a consideration, and companies are confident in their technical teams’ expertise.”

The case for pure upstream Kubernetes

One such company is Rosetta Stone, which has used Docker containers in its DevOps process for years, but has yet to put a container orchestration tool into production. In August 2017, the company considered Kubernetes overkill for its applications and evaluated Docker swarm mode as a simpler approach to container orchestration.

Fast-forward a year, however, and the global education software company plans to introduce upstream Kubernetes into production due to its popularity and ubiquity as the container orchestration standard in the industry.

Concerns about Kubernetes management complexity are outdated, given how the latest versions of the tool smooth out management kinks and require less customization for enterprise security features, said Kevin Burnett, DevOps lead for Rosetta Stone in Arlington, Va.

“We’re a late adopter, but we have the benefit of more maturity in the platform,” Burnett said. “We also wanted to avoid [licensing] costs, and we already have servers. Eventually, we may embrace a cloud service like Google Kubernetes Engine more fully, but not yet.”

Burnett said his team prefers to hand-roll its own configurations of open source tools, and it doesn’t want to use features from a third-party vendor’s Kubernetes implementation that may hinder cloud portability in the future.

Other enterprise IT shops are concerned that third-party Kubernetes implementations — particularly those that rely on a vendor’s own distribution of Kubernetes, such as Red Hat’s OpenShift — will be easier to install initially, but could worsen management complexity in the long run.

“Container sprawl combined with a forked Kubernetes runtime in the hands of traditional IT ops is a management nightmare,” said a DevOps transformation leader at an insurance company who spoke on condition of anonymity, because he’s not authorized to publicly discuss the company’s product evaluation process.

His company is considering OpenShift because of an existing relationship with the vendor, but adding a new layer of orchestration and managing multiple control planes for VMs and containers would also be difficult, the DevOps leader predicted, particularly when it comes to IT ops processes such as security patching.

“Why invite that mess when you already have your hands full with a number of packaged containers that you’re going to have to develop security patching processes for?” he said.

Vendors’ Kubernetes implementations offer stability, support

Fork is a fighting word in the open source world, and most vendors say their Kubernetes implementations don’t diverge from pure Kubernetes code. And early adopters of vendors’ Kubernetes implementations said enterprise support and security features are the top priorities as they roll out container orchestration tools, rather than conformance with upstream code, per se.

Amadeus, a global travel technology company, is an early adopter of Red Hat OpenShift. As such, Dietmar Fauser, vice president of core platforms and middleware at Amadeus, said he doesn’t worry about security patching or forked Kubernetes from Red Hat. While Red Hat could theoretically choose to deviate from, or fork, upstream Kubernetes, it hasn’t done so, and Fauser said he doubts the vendor ever will.

Meanwhile, Amadeus is on the cusp of multi-cloud container portability, with instances of OpenShift on Microsoft Azure, Google and AWS public clouds in addition to its on-premises data centers. Fauser said he expects the multi-cloud deployment process will go smoothly under OpenShift.

Multi-tenancy support and a DevOps platform on top of Kubernetes were what made us want to go with third-party vendors.
Surya Suravarapuassistant vice president of product development, Change Healthcare

“Red Hat is very good at maintaining open source software distributions, patching is consistent and easy to maintain, and I trust them to maintain a portable version of Kubernetes,” Fauser said. “Some upstream Kubernetes APIs come and go, but Red Hat’s approach offers stability.”

Docker containers and Kubernetes are de facto standards that span container environments and provide portability, regardless of which vendor’s Kubernetes implementation is in place, said Surya Suravarapu, assistant vice president of product development for Change Healthcare, a healthcare information technology company in Nashville, Tenn., that spun out of McKesson in March 2017.

Suravarapu declined to specify which vendor’s container orchestration tools the company uses, but said Change Healthcare uses multiple third-party Kubernetes tools and plans to put containers into production this quarter.

“Multi-tenancy support and a DevOps platform on top of Kubernetes were what made us want to go with third-party vendors,” Suravarapu said. “The focus is on productivity improvements for our IT teams, where built-in tooling converts code to container images with the click of a button or one CLI [command-line interface] line, and compliance and security policies are available to all product teams.”

A standard way to manage containers in Kubernetes offers enough consistency between environments to improve operational efficiency, while portability between on-premises, public cloud and customer environments is a longer-term goal, Suravarapu said.

“We’re a healthcare IT company,” he added. “We can’t just go with a raw tool without 24/7 enterprise-level support.”

Still, Amadeus’s Fauser acknowledged there’s risk to trust one vendor’s Kubernetes implementation, especially when that implementation is one of the more popular market options.

“Red Hat wants to own the whole ecosystem, so there’s the danger that they could limit other companies’ access to providing plug-ins for their platform,” he said.

That hasn’t happened, but the risk exists, Fauser said.

GCP Marketplace beats AWS, Azure to Kubernetes app store

Google Cloud Platform has made the first foray into a new frontier of Kubernetes ease of use with a cloud app store that includes apps preconfigured to run smoothly on the notoriously persnickety container orchestration platform.

As Kubernetes becomes the de facto container orchestration standard for enterprise DevOps shops, helping customers tame the notoriously complex management of the platform has become de rigueur for software vendors and public cloud service providers. Google stepped further into that territory with GCP Marketplace, a Kubernetes app store with application packages that can automatically be deployed onto container clusters with one click and then billed through Google Cloud Platform as a service.

A search of the AWS Marketplace for “Kubernetes” turned up Kubernetes infrastructure packages by Rancher, Bitnami and CoreOS, but not prepackaged apps from vendors such as Nginx and Elastic ready to be deployed on Kubernetes clusters, which is what GCP Marketplace offers. Another search of the Azure Marketplace returned similar results.

Just because Google is first to market with this Kubernetes app store doesn’t mean that what the company has done is magic.

“These marketplaces are based on Kubernetes template technologies such as Helm charts, so they’re widely available to everyone,” said Gary Chen, an analyst at IDC in Framingham, Mass. “I’m sure if [AWS and Azure] don’t have it, they are already working on something like this.”

Google executives said Helm charts factored into some of the app packages it created with partners, but in some cases it created GCP Marketplace offerings by way of its work with independent software vendors.

“Our approach gives vendors flexibility to use Helm or other packaging mechanisms, given that there isn’t a clear standard today,” the company said through a spokesperson.

GCP Marketplace Kubernetes apps
A screenshot of Kubernetes apps in the revamped GCP Marketplace.

Initially, GCP Marketplace apps offer click-to-deploy support onto Kubernetes clusters that run in Google Kubernetes Engine, and GKE does telemetry monitoring and logging on those apps in addition to offering billing support.

But there’s nothing that precludes these packages to eventually run on premises or even in other public cloud providers’ infrastructures, said Jennifer Lin, product director for Google Cloud.

“It’s always within the realm of possibility, but not something we’re announcing today,” she said.

There is precedent for GCP products with third-party cloud management capabilities — the Google Stackdriver cloud monitoring tool, for example, can be used with AWS and Azure resources.

Initial app partners include the usual suspects among open source cloud-native infrastructure and middleware projects such as GitLab, Couchbase, CloudBees, Cassandra, InfluxDB, Elasticsearch, Prometheus, Nginx, RabbitMQ and Apache Spark. Commonly used web apps such as WordPress and a container security utility from Aqua Security Software will also be available in the initial release of GCP Marketplace.

Mainstream enterprise customers will look for more traditional apps such as MySQL, SQL Server and Oracle databases, as well as back-office and productivity apps. Lin said Google plans more mainstream app support, but she declined to specify which apps are on the GCP Marketplace roadmap.

Kubernetes hybrid cloud emerges from Google-Cisco partnership

A forthcoming Kubernetes hybrid cloud option that joins products from Cisco and Google promises smoother portability and security, but at this point its distinguishing features remain theoretical.

Cisco plans to release the Cisco Container Platform (CCP) in the first quarter of 2018, with support for Kubernetes container orchestration on its HyperFlex hyper-converged infrastructure product. Sometime later this year, a second version of the container platform will link up with Google Kubernetes Engine to deliver a Kubernetes hybrid cloud offering based on the Cisco-Google partnership made public in October 2017.

“Cisco can bring a consistent hybrid cloud experience to our customers,” said Thomas Scherer, chief architect at Telindus Telecom, an IT service provider in Belgium and longtime Cisco partner that plans to offer hosted container services based on CCP. Many enterprises already use Cisco’s products, which should boost CCP’s appeal, he said.

CCP 2.0 will extend the Cisco Application Centric Infrastructure software-defined network fabric into Google’s public cloud, and enable stretched Kubernetes clusters between on-premises data centers and public clouds, Cisco executives said. Stretched clusters would enable smooth container portability between multiple infrastructures, one of the most attractive promises of Kubernetes hybrid clouds for enterprise IT shops reluctant to move everything to the cloud. CCP also will support Microsoft Azure and Amazon Web Services public clouds, and eventually CCP will incorporate DevOps monitoring tools from AppDynamics, another Cisco property.

“Today, if I have a customer that is using containers, I put them on a dedicated hosting infrastructure, because I don’t have enough confidence that I can maintain customer segregation [in a container environment],” Scherer said. “I hope that Cisco will deliver in that domain.”

He also expects that the companies’ strengths in enterprise data center and public cloud infrastructure components will give the Kubernetes hybrid cloud a unified multi-cloud dashboard with container management.

“Is it going to be easy? No, and the components included in the product may change,” he said. “But my expectation is that it will happen.”

Google public cloud servers in Georgia
Version 2 of the Cisco Container Platform will connect enterprise data centers with Google’s public cloud infrastructure shown here.

Kubernetes hybrid cloud decisions require IT unity

Cisco customers have plenty of other Kubernetes hybrid cloud choices to consider, some of which are already available. Red Hat and AWS joined forces last year to integrate Red Hat’s Kubernetes-based OpenShift Container Platform with AWS services. Microsoft has its Azure public cloud and Azure Stack for on-premises environments, and late last year added Azure Container Service Engine to Azure Stack with support for Kubernetes container management templates.

What Cisco is trying to do, along with other firms, is to expand its appeal to infrastructure and operations teams with monitoring, security and analytics features not included in Kubernetes.
Stephen Elliotanalyst, IDC

However, many enterprises continue to kick the tires on container orchestration software and most do not run containers in production, which means the Cisco-Google partnership has a window to establish itself.

“Kubernetes support is table stakes at this point,” said Stephen Elliot, analyst at IDC. “Part of what Cisco is trying to do, along with other firms, is to expand its appeal to infrastructure and operations teams with monitoring, security and analytics features not included in Kubernetes.”

As Kubernetes hybrid cloud options proliferate, enterprise IT organizations must unite traditionally separate buyers in security, application development, IT management and IT operations to evaluate and select a product. Otherwise, each constituency will be swayed by its established vendor’s product and chaos could ensue, Elliot said.

“There are a lot of moving parts, and organizations are grappling with whom in their organization to turn to for leadership,” he said. “Different buyers can’t make decisions in a vacuum anymore, and there are a lot of politics involved.”

Beth Pariseau is senior news writer for TechTarget’s Cloud and DevOps Media Group. Write to her at [email protected] or follow @PariseauTT on Twitter.