Stop Storing Secrets In Environment Variables!
By Matt Hamilton
January 28, 2022
Did that get your attention? Good. The goal of this short post is to evangelize engineering teams to end the psychosis that is storing secrets in environment variables.
First let’s review the positives about storing secrets in environment variables:
- When the machine or container exits, the environment variables and their contents are lost.
Yep, that’s it. That’s the only good thing about environment variables.
By putting cleartext secrets in environment variables, they’re out dangling in the wind. Any overly-verbose information disclosure vulnerability in a dependency could smash the sand castle.*Source: https://xkcd.com/2347/*
The Proposal: Use ephemeral filesystem mounts
What is this? The early 2000’s? A faint whisper echoes “You can’t write secrets to disk!”
This is correct, don’t write secrets to disk. Write secrets to tmpfs RAM-only volume mounts, which are filesystems that exist only in ephemeral memory. All the benefits of environment variables and all the benefits of file permissions, but you can unmount it!
All the cool kids are using Kubernetes or some other container orchestrator. The orchestrator should provide an ephemeral tmpfs volume containing application secrets to the application. Correctly, this is how the Hashicorp Vault sidecar operates – creating an ephemeral
Once the application hits “readiness” status (as determined by health endpoints or the load balancer), the secrets volume should be unmounted and made inaccessible. Regardless if you’re using a secrets management service or not, ensuring this happens is on YOU. Your secrets management service won’t do this for you.
Counterpoint: You have to trust something somewhere
AKA: chicken or the egg?
While a degree of trust is required when distributing secrets, the application is the most trustworthy at initialization prior to the execution of business logic. In a “clean slate” containerized environment this is doubly true. During initialization the functionality of the application and inherent exposure is limited. This is the best and only viable time to pass cleartext secrets. Once secrets are read and the tmpfs filesystem made unavailable, an attacker's prospect of exfiltrating these secrets is suddenly daunting. And if applications are properly encrypting their secrets in memory using tooling like memguard? Oof.
Hacker News user “fubarred” had it right in 2015:
Ultimately, secrets need to live somewhere and need to be accessed as plain text. Just make sure that the access as small window is as [sic] possible, and try to obliterate it after use, if possible.
Counterpoint: File permissions don’t provide additional protection
This is correct. This is also why it is so important to ensure the files become inaccessible after initialization.
Counterpoint: Key rotation
Sometimes you have to rotate keys. In this case, perform the rotation using a secrets management service. Ensure that the key used to bootstrap the secrets management service as well as the application secrets are made inaccessible after application initialization. Secrets accessible in cleartext on the filesystem during business logic execution is worse than storing secrets in environment variables.
Secrets management services are overkill for most static third-party API tokens used by web applications. Rotating keys for third-party APIs is almost always a manual process. The likelihood of misconfiguring the secrets management service is also not to be discounted.
Information security professionals have long recommended storing secrets in environment variables. As an industry we’ve taken the shortcut and WE WERE WRONG. It’s time we own up to the mistake and course-correct. Environment variables are never an appropriate place to store secrets – it was just a convenient and lazy recommendation to get secrets off the disk. The practice has become prolific and must change.