Secrets

External Secrets Operator

To create Kubernetes secrets resource from Vault - we can use ESO for that purpose. Install:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace \
  --set installCRDs=true

These secret resources will be needed to reload pods automatically.

SecretStore configuration and ExternalSecret template for each service are in app/templates/eso/ folder in devops repo.

Vault

Dev Mode

First install vault (from hashicorp helm repo by default, but if it's unavailable in some regions best option is installing from github charts). Example from github:

 git clone https://github.com/hashicorp/vault-helm.git ~/vault-helm
 cd ~/vault-helm

Install on dev mode:

 helm install vault . --set "server.dev.enabled=true"

Wait until vault pods be ready. After, exec into the vault-0 pod:

kubectl exec -it vault-0 -- /bin/sh

Configure k8s auth:

vault auth enable kubernetes
vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT"

Enable secrets engine:

vault secrets enable -path=secret kv-v2

Create policy for External Secrets Operator:

vault policy write external-secrets - <<EOH
path "secret/data/*" {
  capabilities = ["read", "list"]
}
path "secret/metadata/*" {
  capabilities = ["list"]
}
EOH

Create roles (same with policies, need to be done for each service), example for auth:

vault write auth/kubernetes/role/external-secrets \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets \
  policies=external-secrets \
  ttl=1h

Save secrets in storage:

vault kv put secret/auth-service \
    SECRET="secret"

Or use Vault UI to add secrets. Access it by patching vault-ui to LoadBalancer if there is no Ingress configuration exists.

Prod Mode

Same installation steps as dev mode, but without dev flag and with vault prod values

helm install vault hashicorp/vault -f ~/devops/vault/values.yaml

or if there is no access from some regions:

cr ~/vault-helm/
helm install vault . -f ~/devops/vault/values.yaml

Wait pod to be ready, then initialize:

kubectl exec vault-0 -- vault operator init \
    -key-shares=1 \
    -key-threshold=1 \
    -format=json > vault-keys.json

Unseal:

UNSEAL_KEY=$(jq -r ".unseal_keys_b64[]" vault-keys.json)
kubectl exec vault-0 -- vault operator unseal $UNSEAL_KEY

Login to Vault:

ROOT_TOKEN=$(jq -r ".root_token" vault-keys.json)
kubectl exec vault-0 -- vault login $ROOT_TOKEN

If Vault was configured before and no longer access to key/token - please contact to teamlead.

Further fill proccess and access of new production mode is exactly the same as dev mode earlier.

Reloader

Install stakater/Reloader:

helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install reloader stakater/reloader

This controller needed to observe secrets update (created by External Secrets Operator from Vault) and trigger pod recreation to use those new environment variables.

Complete scheme:

1. Vault Secrets set by team 
2. ESO syncs with k8s secrets at defined interval
3. Reloader observes changes on secret resources and restarts pods
Outcome: Auto-refreshed applications with no manual intervention