The Security Footgun in etcd

From an application security perspective databases are the most valuable parts of our systems. They store the data that gives value to our apps and companies. This data which has been entrusted to us by our users should be kept safe and away of the hands of criminals.

Every developer I talk to is very aware of this. Their MySQL, PostgreSQL and MongoDB databases are treated with caution and security is definitely a thought. It doesn’t always works but at least everyone is aware and tries their best.

But what happens with databases that don’t feel like “regular databases”? I’m talking about Memcached, Redis and of course etcd. This kind of databases are often used for a single use case and treated without much care.

The other day I was reading the CoreOS documentation and came across etcd. I have read about etcd before but in the context of CoreOS is really interesting. Etcd is an integral part of their clustering system. They use it to do orchestration and a lot of other stuff.

Etcd is really cool, It uses the raft consensus algorithm to keep all the nodes in sync, it has a very simple to use CLI tool and best of all it has an HTTP API. I really like etcd and it sure feels like we are going to use it in the near future for some of our orchestration needs.

Reading about etcd made me remember the news about MongoDB security issues that turned out to be people not enabling authentication which is not enabled by default. It really beats me why this is the happy path but hey, I don’t build databases I just use them.

Here’s an excerpt from the etcd’s documentation talking about their authentication mechanism.

“etcd before 2.1 was a completely open system; anyone with access to the API could change keys. In order to preserve backward compatibility and upgradability, this feature is off by default.” Read more

Yes. The same thing, etcd has an authentication mechanism which is disabled by default and it also has a very nice RESTful API as it’s main interface, what could go wrong right. People are smart and they will keep their etcd services from leaking to the open internet. Wrong!

I did a simple search on shodan and came up with 2,284 etcd servers on the open internet. So I clicked a few and on the third try I saw what I was hoping not to see. CREDENTIALS, a lot of CREDENTIALS. Credentials for things like cms_admin, mysql_root, postgres, etc.

In order to try to get a sense of the issue I downloaded the full shodan report and wrote a very simple script that basically called the etcd API and requested all keys. That’s basically equivalent to doing a database dump but over their very nice REST API.

GET http://<ip address>:2379/v2/keys/?recursive=true

This will return all the keys stored on the servers in JSON format. So my script basically went down the list and created a file for each IP (127-0-0-1.json) with the contents of etcd. I stopped the script at about 750 MB of data and 1,485 of the original IP list.

Then I performed a few basic searches to get a sense of what was there and found a bunch of credentials. Passwords for databases of all kinds, AWS secret keys, and API keys and secrets for a bunch of services. Also came across a few certificates, you name it. There’s a lot of stuff there.

I did not test any of the credentials but if I had to guess I would guess that at least a few of them should work and this is the scary part. Anyone with just a few minutes to spare could end up with a list of hundreds of database credentials which can be used to steal data, or perform a ransomware attacks.

In order to prevent this from happening just enable authentication and if is not strictly required please get your etcd servers off the open internet. Use a security group or set a firewall rule to avoid random people from reading and writing to your etcd server.

Finally, I did not test this but since all of these etcd servers are completely open I’m almost certain that an attacker could write to them using the same API. An attacker might use it to change the data in etcd and mess with configuration and even maybe authentication or it could be used to store exfiltrated data from other attacks.

I really hope that the etcd team would reconsider their position and make a breaking change soon to enable authentication by default. As we learned from the MongoDB experience this is a huge footgun that can be easily removed an otherwise awesome software.

Update: Twitter user @BradErzwh reminded me to mention that etcd is “used as Kubernetes’ backing store for all cluster data”. In most cases that data includes secrets encoded as base64 strings. More info