-->
Home
Kubernetes

A Detailed Brief About Offence and Defence on Cloud Security - Etcd Risks

Introduction

Etcd is an open source project launched by the CoreOS team in June 2013. Its goal is to build a highly available distributed key-value database for service discovery, shared configuration, and consistency assurance . It has been widely used in kubernetes, ROOK, CoreDNS, M3 and openstack and other fields.

Etcd internally uses the raft protocol as the consensus algorithm, which is implemented based on the Go language. In terms of composition, Etcd is mainly composed of four parts: HTTP Server, Store, Raft and WAL. We can better understand Etcd through the Etcd architecture diagram. The Etcd architecture diagram is shown in the following figure:


The more common versions of Etcd are the v2 version and the v3 version. The common point of the v2 and v3 versions is that they share the same set of raft protocol code. The difference is that the two are two independent applications, which are incompatible with each other. identical.

It is worth noting that the Kubernetes cluster has deprecated the Etcdv2 version in Kubernetes v1.11. In the new version of Kubernetes, Kubernetes uses Etcd v3 to store data.

Detailed on Etcd and Kubernetes

After having a preliminary understanding of Etcd, let's take a look at the application of Etcd in Kubernetes.

In a Kubernetes cluster, there are two types of components: control plane components and Node components. These two types of components contain a variety of components with different functions. These components together ensure the normal operation of the Kubernetes cluster. Kubernetes cluster component structure can refer to the following figure:


Etcd plays the role of a control plane component in Kubernetes. It is a key-value database with both consistency and high availability. It plays the role of a background database that stores all Kubernetes cluster data in a Kubernetes cluster.

There are two services in the Kubernetes system that need to use Etcd for collaboration and storage, namely Kubernetes itself and the network plug-in flannel.

During the configuration process of the Kubernetes cluster, the Etcd component needs to be installed. The yaml file of Etcd can be seen in the following figure:


As can be seen from the above configuration file, Etcd needs to configure two url addresses during the configuration process: listen-client-urls and listen-peer-urls, listening on ports 2379 and 2380 respectively. Among them, listen-peer-urls is used for etcd cluster to synchronize information and maintain connections, while listen-client-urls is used to receive HTTP requests from clients.

Ectd Common Risks

Common Etcd risks include:

  • When starting etcd, the client-cert-auth parameter is not used to turn on certificate verification;
  • Etcd 2379 port public network exposure;
  • Etcd 127.0.0.1:2379 is accessible due to SSRF vulnerability;
  • Etcd cert leaked.

Let's take a look at how the above four risk points pose a threat to Etcd. Let's first look at the risks of not using the client-cert-auth parameter to turn on certificate verification:

When starting etcd, the correct way is to use the client-cert-auth parameter to turn on certificate verification, as shown in the red box of the configuration file below:


After the certificate verification option is turned on, you can access the Etcd service without authentication through the local 127.0.0.1:2379 address, but you need to carry the cert for authentication access through other addresses.

When the client-cert-auth parameter is not used to enable certificate verification, any address to access the Etcd service does not need to perform certificate verification. At this time, the Etcd service is at risk of unauthorized access.

Next, let's analyze the risks brought by the exposure of the Etcd 2379 port to the public network :

When configuring Etcd, the correct way is to configure a reasonable IP address for the listen-client-urls parameter, Etcd will listen on the given port and interface, as shown in the red box below:


Due to incorrect configuration, configure the listen-client-urls IP to 0.0.0.0, then Etcd will listen on the given port on all interfaces, which will cause Etcd port 2379 to be exposed on the public network.
 
Next, let's analyze the accessibility risk of Etcd 127.0.0.1:2379 due to the SSRF vulnerability:

Even if Etcd is configured correctly, due to the SSRF vulnerability of the application on the server, Etcd 127.0.0.1:2379 is accessible, this interface does not require certificate verification by default, so attackers can access this interface through the SSRF vulnerability and read Etcd sensitive data in .

Finally, let's take a look at the risks posed by the Etcd cert leak :

After configuring secure communication, you need to use TLS authentication to complete the access to the Etcd service, usually using a valid certificate in the following ways:
ETCDCTL_API=3 etcdctl --endpoints etcd_ip:2379 
  --cert=/etc/kubernetes/pki/etcd/client.crt   
  --key=/etc/kubernetes/pki/etcd/client.key   
  --cacert=/etc/kubernetes/pki/etcd/ca.crt

If the certificate is stolen, the attacker can use the obtained certificate to access the Etcd service.

The Attack Scenario of Etcd

When building Kubernetes and configuring the Etcd service, if there is a misconfiguration or vulnerability risk point mentioned in the previous chapter, an attacker can use the Etcd risk point to launch an attack.

We list the common attack methods of attackers here, and guide readers to understand the risks and threats faced by the Etcd service by means of attack and defense. Before we start to introduce common attacks, let's first understand a common etcd command line tool - etcdctl.

etcdctl is a command line client that provides some concise commands. Users can interact with etcd services directly using the commands provided by etcdctl without using the HTTP API.

It can be downloaded from the following address:


Next, we analyze several attack scenarios one by one. the exposure way of public network

Etcd initial access

  • Etcd 2379 port public network exposure
In this scenario, the attacker can use port 2379 exposed on the public network to access the Etcd service. However, in this scenario, the attacker still faces two different situations, that is, whether the current Etcd service uses the client-cert-auth parameter to enable certificate verification when it is started.

If the current Etcd service does not perform certificate verification, there is an unauthorized access vulnerability. You can obtain top-levelkeys data through the following commands

/etcdctl--endpoints=https://etcd_ip:2379/ get / --prefix --keys-only

The specific operation can be seen in the following figure:


If the current Etcd service uses a certificate for verification, you need to use the obtained certificate to configure and access

Before using the etcdctl tool, you need to configure the following three environment variables:
  • ETCDCTL_CERT
  • ETCDCTL_CACERT
  • ETCDCTL_KEY
After the correct certificate is configured, the Etcd service can be accessed through the etcdctl tool.

Etcd localhost port access due to SSRF vulnerability

In this scenario, Etcd 127.0.0.1:2379 is accessible due to the SSRF vulnerability, and the Etcd 127.0.0.1:2379 address does not require certificate verification by default, that is, it can be accessed directly. Therefore, when the application on the Etcd server has an SSRF vulnerability , the attacker can send malicious instructions to the Etcd service API interface by constructing internal and external requests.

Etcd Credential Stealing

Through the initial access phase, the attacker obtains access to the Etcd service, and can use the etcdctl tool to read the data stored in Etcd. Next, we will introduce the techniques applied in this phase.

Get the ClusterRole Key

You can use the following command to read the key related to the clusterrole stored in Etcd

etcdctl --endpoints=https://etcd_ip:2379/ get / --prefix --keys-only | grep/secrets/kube-system/clusterrole.

Get the token key value

You can use the following command to read the value of the key obtained in the previous step:

etcdctl--endpoints=https://etcd_ip:2379/get/ /xxx/secrets/kube-system/clusterrole-xxx

From the returned data, pick a role with high permissions and read its token. The token data usually starts with the string "ey" and is truncated to the string. The previous part of "#kubernetes.io/service-account-token", see the selected part in the figure below:


So far, we have obtained a valid Kube Apiserver access token.

Kube Apiserver command execution

Access the Kube Apiserver through the token authentication stolen in the previous stage, and use kubectl to take over the cluster. The specific operations are as follows:

kubectl--insecure-skip-tls-verify -s https://kube_apiserver:6443/--token="[ey...]" -n kube-system get nodes

General Operations Context

The context can be generated with the following command:

Configuring context to simplify kubectl operations

Kubernetes groups access parameters by context, using convenient names. Each context has three parameters: cluster, namespace and user. The kubectl tool can specify our generated context context_config through the kubeconfig parameter:


In this way, kubectl operations can be simplified without the need to fill in TOKEN every time the kubectl command is executed.

Etcd Defence Remedies

Through the introduction of the above attack scenarios, we propose the following protection suggestions to strengthen the Etcd service:
  • When starting Etcd, use the client-cert-auth parameter to turn on certificate verification;
  • Etcd data is encrypted and stored to ensure that Etcd data cannot be used after leakage;
  • Correctly configure the listen-client-urls parameter to prevent external network exposure;
  • Try to avoid deploying web applications on the node where Etcd is located to prevent attacks on the Etcd localhost address through web application vulnerabilities.

Summary

The Etcd component plays an important role in the backend database for storing cluster data in Kubernetes, so the security of the Etcd component is also particularly important for the cluster. It can be seen from the above analysis that although the Etcd component provides a relatively secure authentication function to ensure data security, due to insufficient security awareness or incorrect configuration of the user when configuring and using the Etcd component, the cluster data will be illegally accessed or tamper. Ectd data leakage will bring serious security problems to the cluster. In the future, we will continue to pay attention to the safety of Etcd components

Reference Links










1 comment

  1. Intercom systems. which were once regarded to be simple two-way communication systems. Are now expanding in new ways that include complex capabilities that may intelligently! serve end-users and system owners. These capabilities now include automated monitoring and remote management, as well as simple, safe. secure methods of granting access to a facility.
    CCTV
    Smart Intercoms Provide
    Video Surveillance Systems