2024-08-06 –, Tuscany
Credentials are one of the most vulnerable components of any software system, and yet, they're notoriously difficult to change. More specifically, developers are often loath to change credentials for two reasons: they either don't know how to do it safely, or they know that to do it safely, the entire system needs to be rebooted, which causes expensive downtime. Fortunately, things need not be this way! By applying a few basic strategies, any complex codebase can be designed to handle credential rotation with no redeployments and practically zero downtime. Additionally, even just going through the exercise can teach valuable lessons about system failure points and design weaknesses, which can better inform incident response.
Credentials are one of the most vulnerable components of any software system, and yet, they're notoriously difficult to change. If you ask a developer to change a credential on a production system, typically you'll get one of three answers:
- That requires a system reboot, or we will break everything. I'll schedule it for the next maintenance window.
- We can try, but we might break everything. Is the business okay with that?
- Why do we need to do that? That's risky and you'll tank my SLAs for this month, leave me alone!
One unifying sentiment behind these responses is that credential changes are cosmic events that carry significant risk. The best way to reduce this risk is to either not change things or do so in a carefully controlled way. What this perspective fails to consider is that the risk associated with a compromised set of credentials far outweighs the risk of changing them. If this is the case, then why aren't we constantly planning and testing key rotations?
The reason for the unifying sentiment above is that from a technical perspective, credential rotations are difficult. Without a downtime window, credential changes require precise coordination between systems with many complex failure cases. The sheer variety of credential types also makes things difficult; the semantics of certificate replacement are very different than password rotation, for example. Despite this complexity, experience shows us that most credentials can be safely rotated using a set of common strategies, each of which can be slightly modified to suit specific use cases. In fact, going through the exercise of changing a credential can still be valuable even if the procedure isn't done regularly. It requires breaking down basic assumptions and running unusual tests, which can expose interesting points of failure or risk that may never have been examined otherwise.
This talk covers some basic strategies for different types of credential rotations, as well as some interesting system design changes we made at our organization based on things we learned when we started rotating credentials.
I got my MS in Computer Science from Virginia Tech in 2021 with a focus on systems and networking. I currently do security automation for Viasat Inc, a global Satellite internet service provider, with an emphasis on credential management and RBAC systems.