Tag Archives: Kubernetes

HashiCorp Nomad vs. Kubernetes matchup intensifies with 0.11

A HashiCorp Nomad beta release this week could help it encroach on Kubernetes’ territory with advanced IT automation for legacy applications and a simpler approach to container orchestration.

Hashicorp first released the open source workload orchestrator in 2015, a year after Kubernetes arrived in the market. But since then, Kubernetes has become the industry-standard container orchestrator, while Nomad Enterprise is HashiCorp’s least-used commercial product in a portfolio that also includes Terraform infrastructure as code, Vault secrets management and Consul service discovery.

These products are also commonly used in Kubernetes environments, and HashiCorp officials typically prefer to frame Nomad as complementary to Kubernetes, rather than a competitor. In the past, HashiCorp’s documentation has pointed out that past versions of Nomad orchestrated only compute resources, scheduling workloads on separately managed underlying resources. This made for a simpler but less complete approach to workload automation, as previous versions of Nomad did not handle networking and storage for application clusters, as Kubernetes does.

However, with version 0.11, released in beta this week, HashiCorp Nomad’s storage features draw closer to those offered by Kubernetes. The new capabilities include support for shared storage volumes through the open source Container Storage Interface (CSI), a set of APIs supported by most major storage vendors. CSI is most commonly used with Kubernetes, but any CSI plugins written to work with Kubernetes will also work with HashiCorp Nomad as of version 0.11.

HashiCorp Nomad version 0.11 also introduces horizontal application autoscaling capabilities, as well as support for task dependencies in cases where application components must be deployed in a certain order on a container cluster.

“[Nomad] can still coexist with Kubernetes, especially for legacy applications when customers prefer to use Kubernetes for containers,” said Amith Nair, VP of product marketing at HashiCorp. “But the [new] features make it a more direct comparison, and we’re starting to see increased usage on the open source side, where some customers are downloading it to replace Kubernetes.”

In the last six months, open source downloads of HashiCorp Nomad have doubled each month to reach 20,000 per month, Nair said. A hosted Nomad cloud service also remains on the company’s long-term roadmap, which would likely compete with the many hosted Kubernetes services available.

HashiCorp Nomad seeks app modernization niche

Most of HashiCorp Nomad’s workload orchestration features can be used to modernize legacy applications that run on VMs. Nomad’s scheduler, when used with Consul service discovery, can optimize how applications on VMs and containers use underlying resources. With version 0.11’s CSI support, HashiCorp Nomad can perform non-disruptive rolling updates of both container-based and VM-based applications.

Such features may put HashiCorp Nomad in closer competition with IT vendors such as VMware, which offers Kubernetes container orchestration alongside VM management. HashiCorp has an uphill battle in that market as well, given VMware’s ubiquity in enterprise shops. But as with Kubernetes, HashiCorp Nomad could capture some attention from IT pros because of its simplicity, analysts said.

Nomad can infiltrate the same market as VMware’s Project Pacific and Tanzu with a low-cost alternative for users that want to manage traditional workloads and cloud-native workloads with one entity.
Roy IllsleyAnalyst, Omdia

“Nomad can infiltrate the same market as VMware’s Project Pacific and Tanzu with a low-cost alternative for users that want to manage traditional workloads and cloud-native workloads with one entity,” said Roy Illsley, analyst at Omdia, a technology market research firm in London. “The challenge is that HashiCorp hasn’t been great at marketing — tech people know it, but tech people don’t necessarily sign the checks.”

With a recent $175 million funding infusion for HashiCorp, however, that could change, and HashiCorp could play a role similar to Linkerd, a service mesh rival to Google and IBM’s Istio that has held its own in the enterprise because many consider it easier to setup and use.

HashiCorp Nomad vs. Kubernetes pros and cons

Two HashiCorp users published blog posts last year detailing their decision to deploy Nomad over Kubernetes. The on-premises IT team at hotel search site Trivago moved its IT monitoring workloads to the public cloud using Nomad in early 2019. Trivago’s IT staff already had experience with HashiCorp’s tool and found Kubernetes more complex than was necessary for its purposes.

“The additional functionality that Kubernetes had to offer was not worth the extra efforts and human resources required to keep it running,” wrote Inga Feick, a DevOps engineer at Trivago, based in Dusseldorf, Germany. “Remote cloud solutions like a managed Kubernetes cluster or [Amazon ECS] are not an option for our I/O-intense jobs either.”

Another freelance developer cited Nomad’s simplicity in a November 2019 post about porting a project to Nomad from Kubernetes.

“Kubernetes is getting all the visibility for good reasons, but it’s probably not suitable for small to medium companies,” wrote Fabrice Aneche, a software engineering consultant based in Quebec. “You don’t need to deploy Google infrastructure when you are not Google.”

Both blog posts noted significant downsides to HashiCorp Nomad vs Kubernetes at the time, however.

“Nomad is one binary, but the truth is Nomad is almost useless without Consul,” Aneche noted in his post. This adds some complexity to HashiCorp Nomad for production use, since users are required to use Consul’s template language to track changes to the Nomad environment. Version 0.11 adds more detailed insights and alerts to a Nomad remote execution UI to make service management easier. Aneche did not respond to requests for comment about the version 0.11 release this week.

Meanwhile, Trivago’s Feick noted the lack of support for autoscaling in January 2019 made HashiCorp Nomad cumbersome to manage at times.

“You need to specify the resource requirements per job,” she wrote. “Give a job too much CPU and memory and Nomad cannot allocate any, or at least not many, other jobs on the same host. Give it not enough memory and you might find it dying… It would be neat if Nomad had a way of calculating those resource needs on its own. One can dream.” Feick didn’t respond to requests for additional comment this week.

HashiCorp Nomad version 0.11 takes the first step toward full autoscaling support with horizontal application autoscaling, or the ability to provide applications with cluster resources dynamically without manual intervention, a company spokesperson said.

Subsequent releases will support horizontal cluster autoscaling that adds resources to the cluster infrastructure as necessary, along with vertical application autoscaling, which will add and remove instances of applications in response to demand. Autoscaling features will work with VM workloads but are primarily intended for use with containers.

Go to Original Article
Author:

Google Kubernetes Engine price change sparks discontent

An upcoming price change to Google Kubernetes Engine isn’t sitting well with some users of the managed container service, but analysts said Google’s move is well within reason.

As of June 6, Google will charge customers a cluster management fee of $0.10 per hour, regardless of the cluster’s size or topology. Each customer billing account will receive one free zonal cluster, and the new management fee doesn’t apply to clusters run as part of Anthos, Google’s cross-platform container orchestration service.

Along with the management fee, however, Google is also introducing a service-level agreement (SLA) for Google Kubernetes Engine (GKE). It promises 99.95% availability for regional clusters and 99.5% on zonal clusters, assuming they use a version from Google’s stable release channel, according to the price change announcement.

Google’s decision did not sit well with some users, who voiced complaints on social media.

Gary ChenGary Chen

Container service pricing could remain in flux

Others disagreed. The planned fee for Google Kubernetes Engine is reasonable, said Gary Chen, an analyst at IDC. “The fact is that Kubernetes control plane management is getting more complex and Google is constantly improving it, so there is a lot of value-add there,” he said. “Plus, as more critical workloads get deployed on containers, service levels become important and that will take some effort and investment, so it’s not unreasonable to ask for a fee for enterprise-level SLAs.”

A longer-term solution for Google could be to offer a lower-cost or free tier for those who don’t need all the features or the SLA, Chen added. “I think we’ll definitely see that in cloud container pricing in the future,” he said. “More tiers, feature add-ons, etc., to satisfy all segments of the market.”

Google previously had a management fee of $0.15 per hour for large clusters but dropped it in November 2017. The price addition coming in June will bring GKE into parity with Amazon Elastic Kubernetes Service; AWS cut the cluster management fee for EKS to $0.10 per hour in January, down from $0.20 per hour.

Kubernetes ecosystem
Managed services such as GKE fit into a continuum of technologies for managing containers

Although measured in pennies per hour, the cluster management fees amount to about $72 a month per cluster, a sum that can add up fast in larger implementations. The question Google Cloud customers must weigh now is whether the fee is worth it compared to other options.

Microsoft Azure Kubernetes Service is one, as it doesn’t currently carry any cluster management fees. But customers would have to do a close comparison of what Azure charges for the compute resources supporting managed containers, as opposed to Google, AWS and other providers.

Another alternative would be to self-manage clusters, but that would require analysis of whether doing so would be desirable in terms of staff time and training.

Above all, Google would undoubtedly like more adoption of Anthos, which became generally available in April 2019. The platform encompasses a software stack much broader than GKE and is priced accordingly. Anthos is the company’s primary focus in its bid to gain market share against Azure and AWS and represents Google Cloud CEO Thomas Kurian’s intent to get more large enterprise customers aboard.

“The cloud wars are intense and revenue matters,” said Holger Mueller, an analyst at Constellation Research in Cupertino, Calif. The cluster management pricing could be viewed as some “gentle pressure” on customers to adopt Anthos, he added.

Go to Original Article
Author:

Kasten backup aims for secure Kubernetes protection

People often talk about Kubernetes “Day 1,” when you get the platform up and running. Now Kasten wants to help with “Day 2.”

Kasten’s K10 is a data management and backup platform for Kubernetes. The latest release, K10 2.0, focuses on security and simplicity.

K10 2.0 includes support for Kubernetes authentication, role-based access control, OpenID Connect, AWS Identity and Access Management roles, customer-managed keys, and integrated encryption of artifacts at rest and in flight.

“Once you put data into storage, the Day 2 operations are critical,” said Krishnan Subramanian, chief research advisor at Rishidot Research. “Day 2 is as critical as Day 1.”

Day 2 — which includes data protection, mobility, backup and restore, and disaster recovery — is becoming a pain point for Kubernetes users, Kasten CEO Niraj Tolia said.

“In 2.0, we are focused on making Kubernetes backup easy and secure,” Tolia said.

Other features the new Kasten backup software offers, which became generally available earlier in November, include a Kubernetes-native API, auto-discovery of the application environment, policy-driven operations, multi-tenancy support, and advanced logging and monitoring. The Kasten backup enables teams to operate their environments, while supporting developers’ ability to use tools of their choice, according to the vendor.

Kasten K10 dashboard screenshot
Kasten K10 provides data management and backup for Kubernetes.

Kasten backup eyes market opportunity

Kasten, which launched its original product in December 2017, generally releases an update to its customers every two weeks. A typical update that’s not as major as 2.0 typically has bug fixes, new features and increased depth in current features. Tolia said there were 55 releases between 1.0 and 2.0.

Day 2 is as critical as Day 1.
Krishnan SubramanianFounder and chief research advisor, Rishidot Research

Backup for container storage has become a hot trend in data protection. Kubernetes specifically is an open source system used to manage containers across private, public and hybrid cloud environments. Kubernetes can be used to manage microservice architectures and is deployable on most cloud providers.

“Everyone’s waking up to the fact that this is going to be the next VMware,” as in, the next infrastructure of choice, Tolia said.

Kubernetes backup products are popping up, but it looks like Kasten is a bit ahead of its time, Rishidot’s Subramanian said. He said he is seeing more enterprises using Kubernetes in production, for example, in moving legacy workloads to the platform, and that makes backup a critical element.

“Kubernetes is just starting to take off,” Subramanian said.

Kubernetes backup “has really taken off in the last two or three quarters,” Tolia said.

Subramanian said he is starting to see legacy vendors such as Dell EMC and NetApp tackling Kubernetes backup, as well as smaller vendors such as Portworx and Robin. He said Kasten had needed stronger security but caught up with K10 2.0. Down the road, he said he will look for Kasten to improve its governance and analytics.

Tolia said Kasten backup stands out because it’s “purpose-built for Kubernetes” and extends into multilayered data management.

In August, Kasten, which is based in Los Altos, Calif., closed a $14 million Series A funding round, led by Insight Partners. Tolia did not give Kasten’s customer count but said it has deployments across multiple continents.

Go to Original Article
Author:

Kubernetes tools vendors vie for developer mindshare

SAN DIEGO — The notion that Kubernetes solves many problems as a container orchestration technology belies the complexity it adds in other areas, namely for developers who need Kubernetes tools.

Developers at the KubeCon + CloudNativeCon North America 2019 event here this week noted that although native tooling for development on Kubernetes continues to improve, there’s still room for more.

“I think the tooling thus far is impressive, but there is a long way to go,” said a software engineer and Kubernetes committer who works for a major electronics manufacturer and requested anonymity.

Moreover, “Kubernetes is extremely elegant, but there are multiple concepts for developers to consider,” he said. “For instance, I think the burden of the onboarding process for new developers and even users sometimes can be too high. I think we need to build more tooling, as we flush out the different use cases that communities bring out.”

Developer-oriented approach

Enter Red Hat, which introduced an update of its Kubernetes-native CodeReady Workspaces tool at event.

Red Hat CodeReady Workspaces 2 enables developers to build applications and services on their laptops that mirror the environment they will run in production. And onboarding is but one of the target use cases for the technology, said Brad Micklea, vice president of developer tools, developer programs and advocacy at Red Hat.

The technology is especially useful in situations where security is an issue, such as bringing in new contracting teams or using offshore development teams where developers need to get up and running with the right tools quickly.

I think the tooling thus far is impressive, but there is a long way to go.
Anonymous Kubernetes committer

CodeReady Workspaces runs on the Red Hat OpenShift Kubernetes platform.

Initially, new enterprise-focused developer technologies are generally used in experimental, proof-of-concept projects, said Charles King, an analyst at Pund-IT in Hayward, Calif. Yet over time those that succeed, like Kubernetes, evolve from the proof-of-concept phase to being deployed in production environments.

“With CodeReady Workspaces 2, Red Hat has created a tool that mirrors production environments, thus enabling developers to create and build applications and services more effectively,” King said. “Overall, Red Hat’s CodeReady Workspaces 2 should make life easier for developers.”

In addition to popular features from the first version, such as an in-browser IDE, Lightweight Directory Access Protocol support, Active Directory and OpenAuth support as well as one-click developer workspaces, CodeReady Workspaces 2 adds support for Visual Studio Code extensions, a new user interface, air-gapped installs and a shareable workspace configuration known as Devfile.

“Workspaces is just generally kind of a way to package up a developer’s working workspace,” Red Hat’s Micklea said.

Overall, the Kubernetes community is primarily “ops-focused,” he said. However, tools like CodeReady Workspaces help to empower both developers and operations.

For instance, at KubeCon, Amr Abdelhalem, head of the cloud platform at Fidelity Investments, said the way he gets teams initiated with Kubernetes is to have them deliver on small projects and move on from there. CodeReady Workspaces is ideal for situations like that because it simplifies developer adoption of Kubernetes, Micklea said.

Such a tool could be important for enterprises that are banking on Kubernetes to move them into a DevOps model to achieve business transformation, said Charlotte Dunlap, an analyst with GlobalData.

“Vendors like Red Hat are enhancing Kubernetes tools and CLI [Command Line Interface] UIs to bring developers with more access and visibility into the ALM [Application Lifecycle Management] of their applications,” Dunlap said. “Red Hat CodeReady Workspaces is ultimately about providing enterprises with unified management across endpoints and environments.”

Competition for Kubernetes developer mindshare

Other companies that focus on the application development platform, such as IBM and Pivotal, have also joined the Kubernetes developer enablement game.

Earlier this week, IBM introduced a set of new open-source tools to help ease developers’ Kubernetes woes. Meanwhile, at KubeCon this week, Pivotal made its Pivotal Application Service (PAS) on Kubernetes generally available and also delivered a new release of the alpha version of its Pivotal Build Service. The PAS on Kubernetes tool enables developers to focus on coding while the platform automatically handles software deployment, networking, monitoring, and logging.

The Pivotal Build Service enables developers to build containers from source code for Kubernetes, said James Watters, senior vice president of strategy at Pivotal. The service automates container creation, management and governance at enterprise scale, he said.

The build service brings technologies such as Pivotal’s kpack and Cloud Native Buildpacks to the enterprise. Cloud Native Buildpacks address dependencies in the middleware layer, such as language-specific frameworks. Kpack is a set of resource controllers for Kubernetes. The Build Service defines the container image, its contents and where it should be kept, Watters said.

Indeed, Watters said he believes it just might be game over in the Kubernetes tools space because Pivotal owns the Spring Framework and Spring Boot, which appeal to a wide swath of Java developers, which is “one of the most popular ways enterprises build applications today,” he said.

“There is something to be said for the appeal of Java in that my team would not need to make wholesale changes to our build processes,” said a Java software developer for a financial services institution who requested anonymity because he was not cleared to speak for the organization.

Yet, in today’s polyglot programming world, programming language is less of an issue as teams have the capability to switch languages at will. For instance, Fidelity’s Abdelhalem said his teams find it easier to move beyond a focus strictly on tools and more on overall technology and strategy to determine what fits in their environment.

Go to Original Article
Author:

Kubernetes Helm Tiller is dead, and IT pros rejoice

SAN DIEGO — The death of Kubernetes Helm Tiller in version 3 was the talk of the cloud-native world here at KubeCon + CloudNativeCon North America 2019 this week, as the change promises better security and stability for a utility that underpins several other popular microservices management and GitOps tools.

Kubernetes Helm is a package manager used to deploy apps to the container orchestration platform. It’s widely used to deploy enterprise apps to containers through CI/CD pipelines, including GitOps and progressive delivery tools. It’s also a key component for installing and updating the custom resource definitions (CRDs) that underpin the Istio service mesh in upstream environments.

Helm Tiller was a core component of the software in its initial releases, which used a client-server architecture for which Tiller was the server. Helm Tiller acted as an intermediary between users and the Kubernetes API server, and handled role-based access control (RBAC) and the rendering of Helm charts for deployment to the cluster. With the first stable release of Helm version 3 on Nov. 13, however, Tiller was removed entirely, and Helm version 3 now communicates directly with the Kubernetes API Server.

Such was the antipathy for Helm Tiller among users that when maintainers proclaimed the component’s death from the KubeCon keynote stage here this week, it drew enthusiastic cheers.

“At the first Helm Summit in 2018, there was quite a lot of input from the community, especially around, ‘Can we get rid of Tiller?'” said Martin Hickey, a senior software engineer at IBM and a core maintainer of Helm, in a presentation on Helm version 3 here. “[Now there’s] no more Tiller, and the universe is safe again.”

KubeCon Helm keynote
News of Helm Tiller’s demise from the KubeCon keynote stage this week drew cheers from the audience.

Helm Tiller had security and stability issues

IT pros who used previous versions of Helm charts said the client-server setup between Helm clients and Tiller was buggy and unstable, which made it even more difficult to install already complex tools such as Istio service mesh for upstream users.

“Version 3 offers new consistency in the way it handles CRDs, which had weird dependency issues that we ran into with Istio charts,” said Aaron Christensen, principal software engineer at SPS Commerce, a communications network for supply chain and logistics businesses in Minneapolis. “It doesn’t automatically solve the problem, but if the Istio team makes use of version 3, it could really simplify deployments.”

[Now there’s] no more Tiller, and the universe is safe again.
Martin HickeySenior software engineer, IBM and a core maintainer of Helm

Helm Tiller was designed before Kubernetes had its own RBAC features, but once these were added to the core project, Tiller also became a cause for security concerns among enterprises. From a security perspective, Tiller had cluster-wide access and could potentially be used for privilege escalation attacks if not properly secured.

It was possible to lock down Helm Tiller in version 2 — heavily regulated firms such as Fidelity Investments were able to use it in production with a combination of homegrown tools and GitOps utilities from Weaveworks. But the complexity of that task and Helm Tiller stability problems meant some Kubernetes shops stayed away from Helm altogether until now, which led to other problems with rolling out apps on container clusters.

“Helm would issue false errors to our CI/CD pipelines, and say a deployment failed when it didn’t, or it would time out connecting to the Kubernetes API server, which made the deployment pipeline fail,” said Carlos Traitel, senior DevOps engineer at Primerica, a financial services firm in Duluth, Ga.

Primerica tried to substitute kube-deploy, a different open source utility for Helm, but also ran into management complexity with it. Primerica engineers plan to re-evaluate Helm version 3 as soon as possible. The new version uses a three-way merge process for updates, which compares the desired state with the actual state of the cluster along with the changes users want to apply, and could potentially eliminate many common errors during the Helm chart update process.

Despite its difficulties, Helm version 2 was a crucial element of Kubernetes management, SPS’s Christensen said.

“It worked way more [often] than it didn’t — we wouldn’t go back and use something else,” he said. “It helps keep 20-plus resources consistent across our clusters … and we were also able to implement our own automated rollbacks based on Helm.”

Go to Original Article
Author:

Kubernetes security opens a new frontier: multi-tenancy

SAN DIEGO — As enterprises expand production container deployments, a new Kubernetes security challenge has emerged: multi-tenancy.

Among the many challenges with multi-tenancy in general is that it is not easy to define, and few IT pros agree on a single definition or architectural approach. Broadly speaking, however, multi-tenancy occurs when multiple projects, teams or tenants, share a centralized IT infrastructure, but remain logically isolated from one another.

Kubernetes multi-tenancy also adds multilayered complexity to an already complex Kubernetes security picture, and demands that IT pros wire together a stack of third-party and, at times, homegrown tools on top of the core Kubernetes framework.

This is because core upstream Kubernetes security features are limited to service accounts for operations such as role-based access control — the platform expects authentication and authorization data to come from an external source. Kubernetes namespaces also don’t offer especially granular or layered isolation by default. Typically, each namespace corresponds to one tenant, whether that tenant is defined as an application, a project or a service.

“To build logical isolation, you have to add a bunch of components on top of Kubernetes,” said Karl Isenberg, tech lead manager at Cruise Automation, a self-driving car service in San Francisco, in a presentation about Kubernetes multi-tenancy here at KubeCon + CloudNativeCon North America 2019 this week. “Once you have Kubernetes, Kubernetes alone is not enough.”

Karl Isenberg, Cruise Automation
Karl Isenberg, tech lead manager at Cruise Automation, presents at KubeCon about multi-tenant Kubernetes security.

However, Isenberg and other presenters here said Kubernetes multi-tenancy can have significant advantages if done right. Cruise, for example, runs very large Kubernetes clusters, with up to 1,000 nodes, shared by thousands of employees, teams, projects and some customers. Kubernetes multi-tenancy means more highly efficient clusters and cost savings on data center hardware and cloud infrastructure.

“Lower operational costs is another [advantage] — if you’re starting up a platform operations team with five people, you may not be able to manage five [separate] clusters,” Isenberg added. “We [also] wanted to make our investments in focused areas, so that they applied to as many tenants as possible.”

Multi-tenant Kubernetes security an ad hoc practice for now

The good news for enterprises that want to achieve Kubernetes multi-tenancy securely is that there are a plethora of third-party tools they can use to do it, some of which are sold by vendors, and others open sourced by firms with Kubernetes development experience, including Cruise and Yahoo Media.

Duke Energy Corporation, for example, has a 60-node Kubernetes cluster in production that’s stretched across three on-premises data centers and shared by 100 web applications so far. The platform is comprised of several vendors’ products, from Diamanti hyper-converged infrastructure to Aqua Security Software’s container firewall, which logically isolates tenants from one another at a granular level that accounts for the ephemeral nature of container infrastructure.

“We don’t want production to talk to anyone [outside of it],” said Ritu Sharma, senior IT architect at the energy holding company in Charlotte, N.C., in a presentation at KubeSec Enterprise Summit, an event co-located with KubeCon this week. “That was the first question that came to mind — how to manage cybersecurity when containers can connect service-to-service within a cluster.”

Some Kubernetes multi-tenancy early adopters also lean on cloud service providers such as Google Kubernetes Engine (GKE) to take on parts of the Kubernetes security burden. GKE can encrypt secrets in the etcd data store, which became available in Kubernetes 1.13, but isn’t enabled by default, according to a KubeSec presentation by Mike Ruth, one of Cruise’s staff security engineers.

Google also offers Workload Identity, which matches up GCP identity and access management with Kubernetes service accounts so that users don’t have to manage Kubernetes secrets or Google Cloud IAM service account keys themselves. Kubernetes SIG-Auth looks to modernize how Kubernetes security tokens are handled by default upstream to smooth Kubernetes secrets management across all clouds, but has run into snags with the migration process.

In the meantime, Verizon’s Yahoo Media has donated a project called Athenz to open source, which handles multiple aspects of authentication and authorization in its on-premises Kubernetes environments, including automatic secrets rotation, expiration and limited-audience policies for intracluster communication similar to those offered by GKE’s Workload Identity. Cruise also created a similar open source tool called RBACSync, along with Daytona, a tool that fetches secrets from HashiCorp Vault, which Cruise uses to store secrets instead of in etcd, and injects them into running applications, and k-rail for workload policy enforcement.

Kubernetes Multi-Tenancy Working Group explores standards

While early adopters have plowed ahead with an amalgamation of third-party and homegrown tools, some users in highly regulated environments look to upstream Kubernetes projects to flesh out more standardized Kubernetes multi-tenancy options.

For example, investment banking company HSBC can use Google’s Anthos Config Management (ACM) to create hierarchical, or nested, namespaces, which make for more highly granular access control mechanisms in a multi-tenant environment, and simplifies their management by automatically propagating shared policies between them. However, the company is following the work of a Kubernetes Multi-Tenancy Working Group established in early 2018 in the hopes it will introduce free open source utilities compatible with multiple public clouds.

Sanjeev Rampal, Kubernetes Multi-Tenancy Working Group
Sanjeev Rampal, co-chair of the Kubernetes Multi-Tenancy Working Group, presents at KubeCon.

“If I want to use ACM in AWS, the Anthos license isn’t cheap,” said Scott Surovich, global container engineering lead at HSBC, in an interview after a presentation here. Anthos also requires VMware server virtualization, and hierarchical namespaces available at the Kubernetes layer could offer Kubernetes multi-tenancy on bare metal, reducing the layers of abstraction and potentially improving performance for HSBC.

Homegrown tools for multi-tenant Kubernetes security won’t fly in HSBC’s highly regulated environment, either, Surovich said.

“I need to prove I have escalation options for support,” he said. “Saying, ‘I wrote that’ isn’t acceptable.”

So far, the working group has two incubation projects that create custom resource definitions — essentially, plugins — that support hierarchical namespaces and virtual clusters that create self-service Kubernetes API Servers for each tenant. The working group has also created working definitions of the types of multi-tenancy and begun to define a set of reference architectures.

The working group is also considering certification of multi-tenant Kubernetes security and management tools, as well as benchmark testing and evaluation of such tools, said Sanjeev Rampal, a Cisco principal engineer and co-chair of the group.

Go to Original Article
Author:

Google Cloud tackles Spark on Kubernetes

An early version of a Google Cloud service that runs Apache Spark on Kubernetes is now available, but more work will be required to flesh out the container orchestration platform’s integrations with data analytics tools.

Kubernetes and containers haven’t been renowned for their use in data-intensive, stateful applications, including data analytics. But there are benefits to using Kubernetes as a resource orchestration layer under applications such as Apache Spark rather than the Hadoop YARN resource manager and job scheduling tool with which it’s typically associated. Developers and IT ops stand to gain advantages that containers bring to any application, such as portability across systems and consistency in configuration, along with automated provisioning and scaling for workloads that’s handled in the Kubernetes layer or by Helm charts, as well as container resource efficiency compared with virtual or bare metal machines.

“Analytical workloads, in particular, benefit from the ability to add rapidly scalable cloud capacity for spiky peak workloads, whereas companies might want to run routine, predictable workloads in a virtual private cloud,” said Doug Henschen, an analyst at Constellation Research in Cupertino, Calif. 

Google, which offers managed versions of Apache Spark and Apache Hadoop that run on YARN through its Cloud Dataproc service, would prefer to use its own Kubernetes platform to orchestrate resources — and to that end, released an alpha preview integration for Spark on Kubernetes within Cloud Dataproc this week. Other companies, such as Databricks (run by the creators of Apache Spark) and D2iQ (formerly Mesosphere), support Spark on Kubernetes, but Google Cloud Dataproc stands to become the first of the major cloud providers to include it in a managed service.

Customers don’t care about managing Hive or Pig, and want to use Kubernetes in hybrid clouds.
James MaloneProduct manager, Google Cloud Dataproc

Apache Spark has had a native Kubernetes scheduler since version 2.3, and Hadoop added native container support in Hadoop 3.0.3, both released in May 2018. However, Hadoop’s container support is still tied to HDFS and is too complex, in Google’s view.

“People have gotten Docker containers running on Hadoop clusters using YARN, but Hadoop 3’s container support is probably about four years too late,” said James Malone, product manager for Cloud Dataproc at Google. “It also doesn’t really solve the problems customers are trying to solve, from our perspective — customers don’t care about managing [Apache data warehouse and analytics apps] Hive or Pig, and want to use Kubernetes in hybrid clouds.”

Spark on Kubernetes only scratches the surface of big data integration

Cloud Dataproc’s Spark on Kubernetes implementation remains in a very early stage, and will require updates upstream to Spark as well as Kubernetes before it’s production-ready. Google also has its sights set on support for more Apache data analytics apps, including the Flink data stream processing framework, Druid low-latency data query system and Presto distributed SQL query engine.

“It’s still in alpha, and that’s by virtue of the fact that the work that we’ve done here has been split into multiple streams,” Malone said. One of those workstreams is to update Cloud Dataproc to run Kubernetes clusters. Another is to contribute to the upstream Spark Kubernetes operator, which remains in the experimental stage within Spark Core. Finally, Cloud Dataproc must brush up performance enhancement add-ons such as external shuffle service support, which aids in the dynamic allocation of resources.

For now, IT pros who want to run Spark on Kubernetes must assemble their own integrations among the upstream Spark Kubernetes scheduler, supported Spark from Databricks, and Kubernetes cloud services. Customers that seek hybrid cloud portability for Spark workloads must also implement a distributed storage system from vendors such as Robin Systems or Portworx. All of it can work, but without many of the niceties about fully integrated cloud platform services that would make life easier.

For example, the Spark Kubernetes executor uses Python, rather than the Scala programming language, which is a bit trickier to use.

“The Python experience of Spark in Kubernetes has always lagged the Scala experience, mostly because deploying a compiled artifact in Scala is just easier logistically than pulling in dependencies for Python jobs,” said Michael Bishop, co-founder and board member at Alpha Vertex, a New York-based fintech startup that uses machine learning deployed in a multi-cloud Kubernetes infrastructure to track market trends for financial services customers. “This is getting better and better, though.”

There also remain fundamental differences between Spark’s job scheduler and Kubernetes that must be smoothed out, Bishop said.

“There is definitely an impedance [between the two schedulers,” he said. “Spark is intimately aware of ‘where’ is for [nodes], while Kubernetes doesn’t really care beyond knowing a pod needs a particular volume mounted.”

Google will work on sanding down these rough edges, Malone pledged.

“For example, we have an external shuffle service, and we’re working hard to make it work with both YARN and Kubernetes Spark,” he said.

Go to Original Article
Author:

Kubernetes authentication project wrestles with migration problems

Kubernetes has not only matured, it’s showing a wrinkle or two.

In case there was any doubt that Kubernetes is no longer a fledgling project, it has begun to show its age with the accumulation of technical debt, as upstream developers struggle to smooth the migration process for a core set of security improvements.

Kubernetes authentication and role-based access control (RBAC) have improved significantly since the project reached version 1.0 four years ago. But one aspect of Kubernetes authentication management remains stuck in the pre-1.0 era: the management of access tokens that secure connections between Kubernetes pods and the Kubernetes API server.

Workload Identity for Google Kubernetes Engine (GKE), released last month, illuminated this issue. Workload Identity is now Google’s recommended approach to Kubernetes authentication between containerized GKE application services and GCP utilities because Workload Identity users no longer have to manage Kubernetes secrets or Google Cloud IAM service account keys themselves. Workload Identity also manages secrets rotation, and can set fine-grained secrets expiration and intended audience policies that ensure good Kubernetes security hygiene.

GKE users are champing at the bit to use Workload Identity because it could eliminate the need to manage this aspect of Kubernetes authentication with a third-party tool such as HashiCorp’s Vault, or worse, with error-prone manual techniques.

“Without this, you end up having to manage a bunch of secrets and access [controls] yourself, which creates lots of opportunities for human error, which has actually bitten us in the past,” said Josh Koenig, head of product at Pantheon Platform, a web operations platform in San Francisco that hosts more than 200,000 Drupal and WordPress sites.

Koenig recalled an incident where a misconfigured development cluster could have exposed an SSL certificate. Pantheon’s IT team caught the mistake, and tore down and reprovisioned the cluster. Pantheon also uses Vault in its production clusters to guard against such errors for more critical workloads but didn’t want that tool’s management overhead to slow down provisioning in the development environment.

“It’s a really easy mistake to make, even with the best of intentions, and the best answer to that is just not having to manage security as part of your codebase,” Koenig said. “There are ways to do that that are really expensive and heavy to implement, and then there’s Google doing it for you [with Workload Identity].”

Kubernetes authentication within clusters lags Workload Identity

Workload Identity uses updated Kubernetes authentication tokens, available in beta since Kubernetes 1.12, to more effectively and efficiently authenticate Kubernetes-based services as they interact with other Google Cloud Platform (GCP) services. Such communications are much more likely than communications within Kubernetes clusters to be targeted by attackers, and there are other ways to shore up security for intracluster communications, including using OpenID Connect tokens based on OAuth 2.0.

However, the SIG-Auth group within the Kubernetes upstream community is working to bring Kubernetes authentication tokens that pass between Kubernetes Pods up to speed with the Workload Identity standard, with automatic secrets rotation, expiration and limited-audience policies. This update would affect all Kubernetes environments, including those that run in GKE, and could allow cloud service providers to assume even more of the Kubernetes authentication management burden on behalf of users.

We think [API Server authentication tokens] improves the default security posture of Kubernetes open source.
Mike DaneseSoftware engineer, Google; chair, Kubernetes SIG-Auth

“Our legacy tokens were only meant for use against the Kubernetes API Server, and using those tokens against something like Vault or Google poses some potential escalations or security risks,” said Mike Danese, a Google software engineer and the chair of Kubernetes SIG-Auth. “We’ll push on [API Server authentication tokens] a little bit because we think it improves the default security posture of Kubernetes open source.”

There’s a big problem, however. A utility to migrate older Kubernetes authentication tokens used with the Kubernetes API server to the new system, dubbed Bound Service Account Token Volumes and available in alpha since Kubernetes 1.13, has hit a show-stopping snag. Early users have encountered a compatibility issue when they try to migrate to the new tokens, which require a new kind of storage volume for authentication data that can be difficult to configure.

Without the new tokens, Kubernetes authentication between pods and the API Server could theoretically face the same risk of human error or mismanagement as GKE services before Workload Identity was introduced, IT experts said.

“The issues that are discussed as the abilities of Workload Identity — to reduce blast radius, require the refresh of tokens, enforce a short time to live and bind them to a single pod — would be the potential impact for current Kubernetes users” if the tokens aren’t upgraded eventually, said Chris Riley, DevOps delivery director at CPrime Inc., an Agile software development consulting firm in Foster City, Calif.

Kubernetes authentication upgrade beta delayed

After Kubernetes 1.13 was rolled out in December 2018, reports indicated that a beta release for Bound Service Account Token Volumes would arrive by Kubernetes 1.15, which was released in June 2019, or Kubernetes 1.16, due in September 2019.

However, the feature did not reach beta in 1.15, and is not expected to reach beta in 1.16, Danese said. It’s also still unknown how the eventual migration to new Kubernetes authentication tokens for the API Server will affect GKE users.

“We could soften our stance on some of the improvements to break legacy clients in fewer ways — for example, we’ve taken a hard stance on file permissions, which disrupted some legacy clients, binding specifically to the Kubernetes API server,” Danese said. “We have some subtle interaction with other security features like Pod Security Policy that we could potentially pave over by making changes or allowances in Pod Security Policies that would allow old Pod Security Policies to use the new tokens instead of the legacy tokens.”

Widespread community attention to this issue seems limited, Riley said, and it could be some time before it’s fully addressed.

However, he added, it’s an example of the reasons Kubernetes users among his consulting clients increasingly turn to cloud service providers to manage their complex container orchestration environment.

“The value of services like Workload Identity is the additional integration these providers are doing into other cloud services, simplifying the adoption and increasing overall reliability of Kubernetes,” Riley said. “I expect them to do the hard work and support migration or simple adoption of future technologies.”

Go to Original Article
Author:

Free Kubernetes security tools broaden enterprise choices

Kubernetes security tools have proliferated in 2018, and their growing numbers reflect increased maturity around container security among enterprise IT shops.

The latest additions to this tool category include a feature in Google Kubernetes Engine called Binary Authorization, which can create whitelists of container images and code that are authorized to run on GKE clusters. All other attempts to launch unauthorized apps will fail, and the GKE feature will document them.

Binary Authorization is in public beta. Google will also make the feature available for on-premises deployments through updates to Kritis, an open source project focused on deployment-time policy enforcement.

Aqua Security also added to the arsenal of Kubernetes security tools at IT pros’ disposal with an open source utility, called kube-hunter, which can be used for penetration testing of Kubernetes clusters. The tool performs passive scans of Kubernetes clusters to look for common vulnerabilities, such as dashboard and management server ports that were left open. These seemingly obvious errors have taken down high-profile companies, such as Tesla, Aviva and Gemalto.

Users can also perform active penetration tests with kube-hunter. In this scenario, the tool attempts to exploit the vulnerabilities it finds as if an attacker has gained access to Kubernetes cluster servers, which may highlight additional vulnerabilities in the environment.

Fernando Montenegro, analyst, 451 ResearchFernando Montenegro

These tools join several other Kubernetes security offerings introduced in 2018 — from Docker Enterprise Edition‘s encryption and secure container registry features for the container orchestration platform to Kubernetes support in tools from Qualys and Alert Logic. The growth of Kubernetes security tools indicates the container security conversation has shifted away from ways to secure individual container images and hosts to security at the level of the application and Kubernetes cluster.

“Containers are not foolproof, but container security is good enough for most users at this point,” said Fernando Montenegro, analyst with 451 Research. “The interest in the industry shifts now to how to do security at the orchestration layer and secure broader container deployments.”

GKE throws down the gauntlet for third-party container orchestration tools

The question for users, as cloud providers add these features, is, why go for a third-party tool when the cloud provider does this kind of thing themselves?
Fernando Montenegroanalyst, 451 Research

Google’s Binary Authorization feature isn’t unique; other on-premises and hybrid cloud Kubernetes tools, such as Docker Enterprise Edition, Mesosphere DC/OS and Red Hat OpenShift, offer similar capabilities to prevent unauthorized container launches on Kubernetes clusters.

However, third-party vendors once again find themselves challenged by a free and open source alternative from Google. Just as Kubernetes supplanted other container orchestration utilities, these additional Kubernetes management features further reduce third-party tools’ competitiveness.

GKE Binary Authorization is one of the first instances of a major cloud provider adding such a feature natively in its Kubernetes service, Montenegro said.

“[A gatekeeper for Kubernetes] is not something nobody’s thought of before, but I haven’t seen much done by other cloud providers on this front yet,” Montenegro said. AWS and Microsoft Azure will almost certainly follow suit.

“The question for users, as cloud providers add these features, is, why go for a third-party tool when the cloud provider does this kind of thing themselves?” Montenegro said.

Aqua Security’s penetration testing tool is unlikely to unseat full-fledged penetration testing tools enterprises use, such as Nmap and Burp Suite, but its focus on Kubernetes vulnerabilities specifically with a free offering will attract some users, Montenegro said.

Aqua Security and its main competitor, Twistlock, also must stay ahead of Kubernetes security features as they’re incorporated into broader enterprise platforms from Google, Cisco and others, Montenegro said.

IT pros debate upstream vs. packaged Kubernetes implementations

Packaged versions of Kubernetes promise ease of use for the finicky container orchestration platform, but some enterprises will stick with a DIY approach to Kubernetes implementation.

Red Hat, Docker, Heptio, Mesosphere, Rancher, Platform9, Pivotal, Google, Microsoft, IBM and Cisco are among the many enterprise vendors seeking to cash in on the container craze with prepackaged Kubernetes implementations for private and hybrid clouds. Some of these products — such Red Hat’s OpenShift Container Platform, Docker Enterprise Edition and Rancher’s eponymous platform — offer their own distribution of the container orchestration software, and most add their own enterprise security and management features on top of upstream Kubernetes code.

However, some enterprise IT shops still prefer to download Kubernetes source code from GitHub and leave out IT vendor middlemen.

“We’re seeing a lot of companies go with [upstream] Kubernetes over Docker [Enterprise Edition] and [Red Hat] OpenShift,” said Damith Karunaratne, director of client solutions for Indellient Inc., an IT consulting firm in Oakville, Ont. “Those platforms may help with management out of the gate, but software license costs are always a consideration, and companies are confident in their technical teams’ expertise.”

The case for pure upstream Kubernetes

One such company is Rosetta Stone, which has used Docker containers in its DevOps process for years, but has yet to put a container orchestration tool into production. In August 2017, the company considered Kubernetes overkill for its applications and evaluated Docker swarm mode as a simpler approach to container orchestration.

Fast-forward a year, however, and the global education software company plans to introduce upstream Kubernetes into production due to its popularity and ubiquity as the container orchestration standard in the industry.

Concerns about Kubernetes management complexity are outdated, given how the latest versions of the tool smooth out management kinks and require less customization for enterprise security features, said Kevin Burnett, DevOps lead for Rosetta Stone in Arlington, Va.

“We’re a late adopter, but we have the benefit of more maturity in the platform,” Burnett said. “We also wanted to avoid [licensing] costs, and we already have servers. Eventually, we may embrace a cloud service like Google Kubernetes Engine more fully, but not yet.”

Burnett said his team prefers to hand-roll its own configurations of open source tools, and it doesn’t want to use features from a third-party vendor’s Kubernetes implementation that may hinder cloud portability in the future.

Other enterprise IT shops are concerned that third-party Kubernetes implementations — particularly those that rely on a vendor’s own distribution of Kubernetes, such as Red Hat’s OpenShift — will be easier to install initially, but could worsen management complexity in the long run.

“Container sprawl combined with a forked Kubernetes runtime in the hands of traditional IT ops is a management nightmare,” said a DevOps transformation leader at an insurance company who spoke on condition of anonymity, because he’s not authorized to publicly discuss the company’s product evaluation process.

His company is considering OpenShift because of an existing relationship with the vendor, but adding a new layer of orchestration and managing multiple control planes for VMs and containers would also be difficult, the DevOps leader predicted, particularly when it comes to IT ops processes such as security patching.

“Why invite that mess when you already have your hands full with a number of packaged containers that you’re going to have to develop security patching processes for?” he said.

Vendors’ Kubernetes implementations offer stability, support

Fork is a fighting word in the open source world, and most vendors say their Kubernetes implementations don’t diverge from pure Kubernetes code. And early adopters of vendors’ Kubernetes implementations said enterprise support and security features are the top priorities as they roll out container orchestration tools, rather than conformance with upstream code, per se.

Amadeus, a global travel technology company, is an early adopter of Red Hat OpenShift. As such, Dietmar Fauser, vice president of core platforms and middleware at Amadeus, said he doesn’t worry about security patching or forked Kubernetes from Red Hat. While Red Hat could theoretically choose to deviate from, or fork, upstream Kubernetes, it hasn’t done so, and Fauser said he doubts the vendor ever will.

Meanwhile, Amadeus is on the cusp of multi-cloud container portability, with instances of OpenShift on Microsoft Azure, Google and AWS public clouds in addition to its on-premises data centers. Fauser said he expects the multi-cloud deployment process will go smoothly under OpenShift.

Multi-tenancy support and a DevOps platform on top of Kubernetes were what made us want to go with third-party vendors.
Surya Suravarapuassistant vice president of product development, Change Healthcare

“Red Hat is very good at maintaining open source software distributions, patching is consistent and easy to maintain, and I trust them to maintain a portable version of Kubernetes,” Fauser said. “Some upstream Kubernetes APIs come and go, but Red Hat’s approach offers stability.”

Docker containers and Kubernetes are de facto standards that span container environments and provide portability, regardless of which vendor’s Kubernetes implementation is in place, said Surya Suravarapu, assistant vice president of product development for Change Healthcare, a healthcare information technology company in Nashville, Tenn., that spun out of McKesson in March 2017.

Suravarapu declined to specify which vendor’s container orchestration tools the company uses, but said Change Healthcare uses multiple third-party Kubernetes tools and plans to put containers into production this quarter.

“Multi-tenancy support and a DevOps platform on top of Kubernetes were what made us want to go with third-party vendors,” Suravarapu said. “The focus is on productivity improvements for our IT teams, where built-in tooling converts code to container images with the click of a button or one CLI [command-line interface] line, and compliance and security policies are available to all product teams.”

A standard way to manage containers in Kubernetes offers enough consistency between environments to improve operational efficiency, while portability between on-premises, public cloud and customer environments is a longer-term goal, Suravarapu said.

“We’re a healthcare IT company,” he added. “We can’t just go with a raw tool without 24/7 enterprise-level support.”

Still, Amadeus’s Fauser acknowledged there’s risk to trust one vendor’s Kubernetes implementation, especially when that implementation is one of the more popular market options.

“Red Hat wants to own the whole ecosystem, so there’s the danger that they could limit other companies’ access to providing plug-ins for their platform,” he said.

That hasn’t happened, but the risk exists, Fauser said.