Syncing Kubernetes Secrets from Vault to Clusters

Hey there, Darian Vance here. Let me tell you, I’ve spent more late nights than I’d like to admit debugging a production outage only to find out the issue was a simple, manually updated secret that got missed during a rotation. It’s frustrating, error-prone, and a massive time sink. That’s why my team and I automated the whole process. Moving our secret management to a single source of truth—HashiCorp Vault—and syncing it programmatically to our Kubernetes clusters has saved us countless hours and prevented a lot of headaches. This guide is how we do it.

Prerequisites

Before we dive in, let’s make sure you have the basics ready. I’m assuming you’re already familiar with these tools, so I won’t go into their initial setup.

• A running HashiCorp Vault instance with a secret you want to sync.
• A Kubernetes cluster and kubectl configured to access it.
• Python 3 installed on the machine where you’ll run the sync script.
• Familiarity with Vault’s AppRole authentication method. It’s my go-to for programmatic access.

You’ll also need a few Python libraries. I’ll skip the standard virtualenv setup since you likely have your own workflow for that. Just make sure to install hvac, kubernetes, and python-dotenv into your project environment using your preferred package manager, like pip.

The Guide: From Vault to K8s

Step 1: Structure Your Project and Configuration

First things first, let’s get our credentials and configuration out of the code. It’s a security best practice. I always use a simple environment file for this. Create a file named config.env in your project directory.

# Vault Configuration
VAULT_ADDR='https://your-vault-instance.com:8200'
VAULT_ROLE_ID='your-approle-role-id'
VAULT_SECRET_ID='your-approle-secret-id'
VAULT_SECRET_PATH='kv/data/my-app/db-credentials'

# Kubernetes Configuration
K8S_NAMESPACE='my-application-namespace'
K8S_SECRET_NAME='db-credentials-secret'

Pro Tip: In my production setups, I never hardcode the VAULT_SECRET_ID. Instead, I use Vault’s API to generate a short-lived, wrapped secret ID right before the script runs. For this tutorial, we’ll keep it simple, but look into response wrapping for a more secure approach.

Step 2: Authenticate and Fetch Secrets from Vault

Now for the Python part. We’ll start by creating a function to connect to Vault and pull down the secret data. The hvac library makes this incredibly clean.

import os
import hvac
import base64
from dotenv import load_dotenv
from kubernetes import client, config

def fetch_vault_secret(addr, role_id, secret_id, path):
    """Connects to Vault, authenticates, and fetches a secret."""
    try:
        vault_client = hvac.Client(url=addr)
        
        # Authenticate using AppRole
        auth_response = vault_client.auth.approle.login(
            role_id=role_id,
            secret_id=secret_id,
        )
        
        if not vault_client.is_authenticated():
            print("Vault authentication failed.")
            return None

        print("Successfully authenticated with Vault.")
        
        # Read the secret from the specified path
        secret_response = vault_client.secrets.kv.v2.read_secret_version(path=path)
        
        # The actual data is nested under 'data' -> 'data'
        return secret_response['data']['data']
        
    except Exception as e:
        print(f"An error occurred while fetching from Vault: {e}")
        return None

What’s happening here? We initialize the Vault client, use the Role ID and Secret ID to log in, and then read the secret from the path we defined. The secret data itself is nested a couple of levels deep in the response, so we return secret_response['data']['data'] to get the key-value pairs.

Step 3: Prepare and Sync the Secret to Kubernetes

With the secret data in hand, the next step is to create or update a Kubernetes Secret. Kubernetes expects secret values to be Base64-encoded, so we need to handle that conversion. This function will check if the secret already exists; if it does, it’ll be updated (patched), and if not, it’ll be created.

def sync_k8s_secret(namespace, secret_name, secret_data):
    """Creates or updates a Kubernetes secret with the provided data."""
    try:
        # Load Kubernetes configuration from default location (e.g., ~/.kube/config)
        config.load_kube_config()
        api = client.CoreV1Api()
        
        # Base64 encode all values in the secret data
        encoded_data = {key: base64.b64encode(value.encode('utf-8')).decode('utf-8') 
                        for key, value in secret_data.items()}

        body = client.V1Secret(
            api_version="v1",
            kind="Secret",
            metadata=client.V1ObjectMeta(name=secret_name),
            data=encoded_data,
            type="Opaque"
        )

        try:
            # Check if the secret already exists
            api.read_namespaced_secret(name=secret_name, namespace=namespace)
            print(f"Secret '{secret_name}' already exists. Patching...")
            api.patch_namespaced_secret(name=secret_name, namespace=namespace, body=body)
            print(f"Successfully patched secret '{secret_name}' in namespace '{namespace}'.")
        except client.ApiException as e:
            if e.status == 404:
                # Secret does not exist, so create it
                print(f"Secret '{secret_name}' not found. Creating...")
                api.create_namespaced_secret(namespace=namespace, body=body)
                print(f"Successfully created secret '{secret_name}' in namespace '{namespace}'.")
            else:
                # Re-raise other API errors
                raise e

    except Exception as e:
        print(f"An error occurred during Kubernetes sync: {e}")
        return

Pro Tip: Making this script idempotent (safe to run multiple times with the same outcome) is key. The “check-then-patch” or “create-if-not-found” logic ensures you don’t cause errors by trying to create a resource that already exists. It just brings the state of the cluster in line with the state in Vault.

Step 4: The Main Execution Block

Finally, let’s tie it all together in a main execution block that loads our configuration and calls the functions in order.

if __name__ == "__main__":
    load_dotenv('config.env')

    # Load config from environment variables
    vault_addr = os.getenv('VAULT_ADDR')
    vault_role_id = os.getenv('VAULT_ROLE_ID')
    vault_secret_id = os.getenv('VAULT_SECRET_ID')
    vault_secret_path = os.getenv('VAULT_SECRET_PATH')
    
    k8s_namespace = os.getenv('K8S_NAMESPACE')
    k8s_secret_name = os.getenv('K8S_SECRET_NAME')

    print("--- Starting Vault to Kubernetes Secret Sync ---")
    
    # 1. Fetch secrets from Vault
    retrieved_data = fetch_vault_secret(
        vault_addr, vault_role_id, vault_secret_id, vault_secret_path
    )
    
    # 2. Sync secrets to Kubernetes if fetch was successful
    if retrieved_data:
        sync_k8s_secret(k8s_namespace, k8s_secret_name, retrieved_data)
        print("--- Sync process completed successfully. ---")
    else:
        print("--- Sync process failed. No data retrieved from Vault. ---")

Step 5: Automation

You can run this script manually, but the real power comes from automation. A simple cron job is perfect for this. For example, to run it every Monday at 2 AM, you’d set up a cron job like this. Remember to use the correct path to your Python interpreter and script.

0 2 * * 1 python3 script.py

Common Pitfalls (Where I Usually Mess Up)

• RBAC Permissions: The machine or pod running this script needs a role with permissions to get, create, and patch secrets in the target namespace. I’ve wasted hours debugging a script only to find it was a simple permissions issue.
• Vault Pathing: The KVv2 secrets engine in Vault nests data under /data/. I often forget this and use kv/my-app/db-credentials instead of the correct kv/data/my-app/db-credentials for the API call. Double-check your paths!
• Base64 Encoding: Kubernetes requires the values in the secret’s data field to be Base64-encoded, not the keys. My script handles this, but if you’re writing your own, it’s an easy detail to miss.

Conclusion

And there you have it. It might seem like a bit of setup, but once this script is running, you have a robust, automated pipeline for managing your Kubernetes secrets. It establishes Vault as your single source of truth, eliminates manual errors, and lets you and your team focus on building features instead of chasing down configuration drift. Give it a try—it’s a real game-changer for operational sanity.

All the best,
Darian Vance

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions