Overview
Secrets Management is PAM for Applications (A-to-A). When App A talks to Database B, it needs a password. Historically, developers put this password in:
- Source Code (GitHub)
- Config Files (Text)
- Environment Variables
All of these are insecure. Secrets Management Centralizes these credentials into a Vault, encrypts them, and rotates them automatically.
Architecture
Reference Pattern: The Secret Broker
Applications should never know the secret "at rest." They should request it "at runtime."
Diagram
Key Decisions
| Decision | Options | Recommendation | Notes / Gotchas |
|---|---|---|---|
| Central vs Native | Enterprise Vault (HashiCorp), Cloud Native (AWS Secrets) | Hybrid | Use AWS Secrets Manager for AWS apps. Use HashiCorp/CyberArk for On-Prem/Multi-Cloud. |
| Delivery | Sidecar, API, File Mount | Sidecar / File Mount | API requires code changes. Sidecars (k8s) inject secrets as files/env vars without code changes. |
| Rotation | Static, Dynamic | Dynamic | Dynamic Secrets generate a unique credential for every request and delete it after. Zero shared risk. |
| Dev Access | Read/Write, Write-Only | Write-Only | Developers should be able to put secrets in the vault, but not read Production secrets. |
| Detection | Git Scanning, Manual | Git Scanning | You must scan commits (pre-receive hook) to block secrets from entering repo history. |
Implementation
Phase 1: Stop the Bleeding (Git Scanning)
- Implement a tool (TruffleHog, GitGuardian) to scan repositories.
- Block commits containing high-entropy strings (API keys).
- Goal: Stop new secrets from leaking.
Phase 2: Centralize (The Vault)
- Deploy a Secrets Manager.
- Move "Config File" passwords into the Vault.
- Update apps to fetch from Vault (or use a K8s operator).
- Goal: Encryption at rest.
Phase 3: Rotate & Dynamic
- Enable auto-rotation for Databases.
- Switch to Dynamic Secrets (e.g., Vault creates a temporary SQL user for each web request).
- Goal: Ephemeral credentials.
Risks
- The "Secret Zero" Problem: To get the secret from the Vault, the App needs a token (Secret Zero). How do you protect that? (Solution: Platform Identity like AWS IAM Roles or K8s Service Accounts).
- Performance: If the Vault is slow, the App is slow.
- Lock-in: Using AWS Secrets Manager creates deep coupling to AWS.
- DevOps Velocity: If adding a secret takes a Jira ticket and 3 days, devs will hardcode it. Make it self-service.
KPIs
- Hardcoded Secret Count: Number of secrets found in code repositories scanning (Target: 0).
- Rotation Coverage: Percentage of application secrets (DB, API) that are automatically rotated.
- Secret Age: Average age of secrets (Target: < 90 days).
- Vault Adoption: Percentage of applications retrieving secrets from the central vault vs. local config.
