Securely Holding API Keys in Backend

Overview

I think many people are developing multi-tenant SaaS that holds API keys for external APIs.
How should API keys be held in the backend? I thought about it.

Design Policy

The design policy would be as follows.

1. Restrict secrets to "read-only"
   • API keys are not frequently rewritten or referenced. Since the application only needs to "read" them as needed, do not place them in plain text in databases or files, but prepare a read-only mechanism.
   • The moment you place it in plain text, the key is at high risk of being exposed. No matter how secure the server is, inadvertent mistakes and permission leaks cannot be reduced to zero. Therefore, the major premise is to "store in a place where authenticity must be shown to read and can only be touched in a limited way".
2. Thorough "separation" for each tenant
   • In a multi-tenant environment, Company A's API key must not be referenced by Company B.
   • Separate the key storage location logically and physically, such as separating the "namespace" based on the tenant ID, or using a separate secret store for each tenant.
3. Manage encryption keys (KMS) separately
   • When encrypting and saving the secret itself, manage the encryption/decryption key (KMS key) in a separate location from the application server.
   • If using cloud, utilize managed KMS such as "AWS KMS", "Azure Key Vault", "GCP Cloud KMS" to create a structure where plain text keys are not leaked no matter what happens.
4. Restrict access privileges to "least privilege"
   • When the application body (backend) goes to retrieve the secret, prepare an IAM role / service principal that only gives "read permission".
   • If preparing a separate environment from production (staging/development), issue different keys for each and ensure that the production key cannot be touched at all.
5. Enable audit logs
   • Always log who accessed which tenant's secret and when.
   • Even if there is an attacker attempting unauthorized access, you can clarify the intrusion route and responsibility location by tracing the logs.
6. Incorporate regular rotation if possible
   • The operation of "keeping using the issued API key forever" is dangerous.
   • Prepare a mechanism to automatically switch to a new key once a month or quarterly (or according to the tenant's request).
   • Rotation is not troublesome. Automate the handling of confidential information and realize operation with peace of mind.

---

2. Concrete Implementation Pattern Examples

Here are three representative approaches. Each has advantages and disadvantages, so please choose according to your company's operation scale and cloud service adoption status.

2.1. Utilization of Cloud Secret Manager (AWS Secrets Manager, etc.)

Configuration Image

┌───────────────────┐
│  Cloud KMS/KV     │   ← Encrypt and store API keys. Ex: AWS Secrets Manager,
│ (AWS Secrets      │      Azure Key Vault, GCP Secret Manager etc.
│  Manager etc.)    │
└───────────────────┘
           ↑
           │ (Encryption/Decryption Request)
           │
┌───────────────────┐
│   Backend         │   ← Grant IAM role dedicated to reading secrets
│  (API Server etc.)│
│                   │
└───────────────────┘

1. Create a secret for each tenant
   • For example, with AWS Secrets Manager, name the secret like /saas/myapp/tenant-{tenant_id}/external-api-key.
   • Azure Key Vault can also be separated in the form of "VaultName/tenant-{tenant_id}-api-key".
2. Read via SDK
   • From the backend, retrieve the encrypted API key by specifying "SecretId (name)" through AWS SDK (Java/Python/Node.js etc.) or Azure SDK.
   • By giving only "read permission" to the Instance Profile or Service Principal of the execution environment when calling the SDK, prevent unauthorized rewriting or creation.
3. Automate rotation function
   • AWS Secrets Manager has an "automatic rotation function", and by combining it with Lambda functions, you can automate from new key issuance -> replacement -> deployment completion in a few minutes.
   • If you want to send an alert to the tenant owner at each rotation, utilize SNS or email integration.
4. Merits
   • No need to set up a Vault on-premise as it is a managed service
   • Peace of mind that KMS keys and access rights are managed by the cloud vendor
   • Audit logs and rotation functions are available by default
5. Demerits/Concerns
   • Prone to service lock-in by cloud vendors
   • Costs may increase by number of tenants x number of secrets
   • Risk of being unable to retrieve API keys temporarily if a cloud-side failure hits directly

Implementation Point (AWS Example)

1. Create IAM Role
   - Name: MyApp-SecretReaderRole
   - Policy Example (JSON excerpt):
     {
       "Version": "2012-10-17",
       "Statement": [
         {
           "Effect": "Allow",
           "Action": [
             "secretsmanager:GetSecretValue"
           ],
           "Resource": [
             "arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:/saas/myapp/tenant-*/external-api-key-*"
           ]
         }
       ]
     }

2. Create Secret
   AWS CLI:
   $ aws secretsmanager create-secret \
       --name "/saas/myapp/tenant-1001/external-api-key" \
       --description "External API key for Tenant ID=1001" \
       --secret-string '{"api_key":"YOUR_API_KEY_HERE"}'