HashiDays One conference. Three cities. Find a city near you
Vault
Archive

Configure Vault as a certificate manager in Kubernetes with Helm

  • 15min
  • |
  • Vault

Kubernetes configured to use Vault as a certificate manager enables your services to establish their identity and communicate securely over the network with other services or clients internal or external to the cluster.

Jetstack's cert-manager enables Vault's PKI secrets engine to dynamically generate X.509 certificates within Kubernetes through an Issuer interface.

In this tutorial, you set up Vault with the Vault Helm chart, configure the PKI secrets engine and Kubernetes authentication. Then install Jetstack's cert-manager, configure it to use Vault, and request a certificate.

Prerequisites

Install supporting tools

This tutorial was last tested 11 October 2023 on a macOS 13.5.2 using this configuration. 

$ docker versionClient: Cloud integration: v1.0.35-desktop+001 Version:           24.0.5## ...
$ helm versionversion.BuildInfo{Version:"v3.12.3", GitCommit:"3a31588ad33fe3b89af5a2a54ee1d25bfe6eaa5e", GitTreeState:"clean", GoVersion:"go1.20.7"}
$ kubectl version --shortClient Version: v1.28.2Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3Server Version: v1.27.4
$ minikube versionminikube version: v1.31.2commit: fd7ecd9c4599bef9f04c0986c4a0187f98a4396e

These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.

Clone GitHub repository

  1. Retrieve the web application and additional configuration by cloning the hashicorp-education/learn-vault-external-kubernetes repository from GitHub.

    $ git clone https://github.com/hashicorp-education/learn-vault-external-kubernetes.git

  2. The content specific to this tutorial can be found within a sub-directory.

  3. Go into the learn-vault-external-kubernetes directory.

    $ cd learn-vault-external-kubernetes

Working directory

This tutorial assumes that the remainder of commands are executed within this directory.

Start Minikube

Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters. These clusters are run locally inside Virtual Machines (VM).

  1. Start a Kubernetes cluster.

    $ minikube start😄  minikube v1.8.2 on Darwin 10.15.4✨  Automatically selected the hyperkit driver🔥  Creating hyperkit VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...🐳  Preparing Kubernetes v1.17.3 on Docker 19.03.6 ...🚀  Launching Kubernetes ...🌟  Enabling addons: default-storageclass, storage-provisioner⌛  Waiting for cluster to come online ...🏄  Done! kubectl is now configured to use "minikube"

  2. Verify the status of the Minikube cluster.

    $ minikube statushost: Runningkubelet: Runningapiserver: Runningkubeconfig: Configured

    Additional waiting

    Even if the previous step completed successfully, you may have to wait for Minikube to be available. If you see an error, try again after a few minutes.

The host, kubelet, and apiserver report that they are running. The kubectl, a command line interface (CLI) for running commands against Kubernetes cluster, is also configured to communicate with this recently started cluster.

Install the Vault Helm chart

  1. Add the HashiCorp Helm repository.

    $ helm repo add hashicorp https://helm.releases.hashicorp.com"hashicorp" has been added to your repositories

  2. Update all the repositories to ensure helm is aware of the latest versions.

    $ helm repo updateHang tight while we grab the latest from your chart repositories......Successfully got an update from the "hashicorp" chart repositoryUpdate Complete. ⎈Happy Helming!⎈

  3. Install the latest version of the Vault server running in standalone mode with the Vault Agent Injector service disabled.

    $ helm install vault hashicorp/vault --set "injector.enabled=false"NAME: vault## ...

    The Vault server runs in standalone mode on a single pod. By default the Helm chart starts a Vault Agent Injector pod but that is disabled injector.enabled=false.

  4. Get all the pods within the default namespace.

    $ kubectl get podsNAME      READY   STATUS    RESTARTS   AGEvault-0   0/1     Running   0          87s

    The vault-0 pod is deployed. The Vault server running in the pod's container reports that it is running but it is not ready (0/1). To ready the pod requires that the Vault server is initialized and unsealed.

  5. Get all the services within the default namespace.

    $ kubectl get serviceNAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGEkubernetes       ClusterIP   10.96.0.1       <none>        443/TCP             9m24svault            ClusterIP   10.106.149.97   <none>        8200/TCP,8201/TCP   5m58svault-internal   ClusterIP   None            <none>        8200/TCP,8201/TCP   5m58s

    The Vault Helm chart creates a Service that directs requests to the Vault pod. This enables us to address the Vault server within the cluster with the address http://vault.default:8200.

Initialize and unseal Vault

Vault run in standalone mode starts uninitialized and in the sealed state.
Prior to initialization the storage backend is not prepared to receive data.

  1. Initialize Vault with one key share and one key threshold.

    $ kubectl exec vault-0 -- vault operator init -key-shares=1 -key-threshold=1 \      -format=json > init-keys.json

  2. The operator init command generates a root key that it disassembles into key shares -key-shares=1 and then sets the number of key shares required to unseal Vault -key-threshold=1. These key shares are written to the output as unseal keys in JSON format -format=json. Here the output is redirected to a local file named init-keys.json

    View the unseal key found in init-keys.json.

    $ cat init-keys.json | jq -r ".unseal_keys_b64[]"hmeMLoRiX/trBTx/xPZHjCcZ7c4H8OCt2Njkrv2yXZY=

    Insecure operation

    Do not run an unsealed Vault in production with a single key share and a single key threshold. This approach is only used here to simplify the unsealing process for this demonstration.

  3. Create a variable named VAULT_UNSEAL_KEY to capture the Vault unseal key.

    $ VAULT_UNSEAL_KEY=$(cat init-keys.json | jq -r ".unseal_keys_b64[]")

    After initialization, Vault is configured to know where and how to access the storage, but does not know how to decrypt any of it. Unsealing is the process of constructing the root key necessary to read the decryption key to decrypt the data, allowing access to the Vault.

  4. Unseal Vault running on the vault-0 pod with the $VAULT_UNSEAL_KEY.

    $ kubectl exec vault-0 -- vault operator unseal $VAULT_UNSEAL_KEYKey             Value---             -----Seal Type       shamirInitialized     trueSealed          falseTotal Shares    1Threshold       1Version         1.4.0Cluster Name    vault-cluster-3ecefcf2Cluster ID      0998f747-af15-f994-cded-295836b718d6HA Enabled      false

    The operator unseal command reports that Vault is initialized and unsealed.

    Insecure operation

    Providing the unseal key with the command writes the key to your shell's history. This approach is only used here to simplify the unsealing process for this demonstration.

  5. Get all the pods within the default namespace.

    $ kubectl get podsNAME                                    READY   STATUS    RESTARTS   AGEvault-0                                 1/1     Running   0          5m49s

    The vault-0 pod reports that it is ready 1/1. Vault is ready for you to login with the root token generated during the initialization.

  6. View the root token found in init-keys.json.

    $ cat init-keys.json | jq -r ".root_token"s.XzExf8TjRVYKm85xMATa6Q7U

  7. Create a variable named VAULT_ROOT_TOKEN to capture the root token.

    $ VAULT_ROOT_TOKEN=$(cat init-keys.json | jq -r ".root_token")

  8. Login to Vault running on the vault-0 pod with the $VAULT_ROOT_TOKEN.

    $ kubectl exec vault-0 -- vault login $VAULT_ROOT_TOKENSuccess! You are now authenticated. The token information displayed belowis already stored in the token helper. You do NOT need to run "vault login"again. Future Vault requests will automatically use this token.Key                  Value---                  -----token                s.P3Koh6BZikQPDxPSNwDzmKJ5token_accessor       kHFYypyS2EcYpMyrsyXUQmNatoken_duration       ∞token_renewable      falsetoken_policies       ["root"]identity_policies    []policies             ["root"]

    The Vault server is ready to be configured as a certificate store.

Configure PKI secrets engine

  1. Start an interactive shell session on the vault-0 pod.

    $ kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh/ $

    Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.

  2. Enable the PKI secrets engine at its default path.

    $ vault secrets enable pkiSuccess! Enabled the pki secrets engine at: pki/

    By default the KPI secrets engine sets the time-to-live (TTL) to 30 days. A certificate can have its lease extended to ensure certificate rotation on a yearly basis (8760h).

  3. Configure the max lease time-to-live (TTL) to 8760h.

    $ vault secrets tune -max-lease-ttl=8760h pkiSuccess! Tuned the secrets engine at: pki/

    Vault can accept an existing key pair, or it can generate its own self-signed root. In general, we recommend maintaining your root CA outside of Vault and providing Vault a signed intermediate CA.

  4. Generate a self-signed certificate valid for 8760h.

    $ vault write pki/root/generate/internal \    common_name=example.com \    ttl=8760h

    Example output: 

    Key              Value---              -----certificate      -----BEGIN CERTIFICATE-----## ...-----END CERTIFICATE-----expiration       1619120269issuing_ca       -----BEGIN CERTIFICATE-----## ...-----END CERTIFICATE-----serial_number    65:37:b5:b3:91:6c:7b:d8:33:22:03:28:b1:58:ff:be:8a:72:a4:c0

  5. Configure the PKI secrets engine certificate issuing and certificate revocation list (CRL) endpoints to use the Vault service in the default namespace.

    $ vault write pki/config/urls \    issuing_certificates="http://vault.default:8200/v1/pki/ca" \    crl_distribution_points="http://vault.default:8200/v1/pki/crl"

    Output: 

    Key                        Value---                        -----crl_distribution_points    [http://vault.default:8200/v1/pki/crl]enable_templating          falseissuing_certificates       [http://vault.default:8200/v1/pki/ca]ocsp_servers               []

  6. Configure a role named example-dot-com that enables the creation of certificates example.com domain with any subdomains.

    $ vault write pki/roles/example-dot-com \    allowed_domains=example.com \    allow_subdomains=true \    max_ttl=72h

    Output: 

    Key                                   Value---                                   -----allow_any_name                        falseallow_bare_domains                    falseallow_glob_domains                    falseallow_ip_sans                         trueallow_localhost                       trueallow_subdomains                      trueallow_token_displayname               falseallow_wildcard_certificates           trueallowed_domains                       [example.com]allowed_domains_template              falseallowed_other_sans                    []allowed_serial_numbers                []allowed_uri_sans                      []allowed_uri_sans_template             falseallowed_user_ids                      []basic_constraints_valid_for_non_ca    falseclient_flag                           truecn_validations                        [email hostname]code_signing_flag                     falsecountry                               []email_protection_flag                 falseenforce_hostnames                     trueext_key_usage                         []ext_key_usage_oids                    []generate_lease                        falseissuer_ref                            defaultkey_bits                              2048key_type                              rsakey_usage                             [DigitalSignature KeyAgreement KeyEncipherment]locality                              []max_ttl                               72hno_store                              falsenot_after                             n/anot_before_duration                   30sorganization                          []ou                                    []policy_identifiers                    []postal_code                           []province                              []require_cn                            trueserver_flag                           truesignature_bits                        256street_address                        []ttl                                   0suse_csr_common_name                   trueuse_csr_sans                          trueuse_pss                               false

    The role, example-dot-com, is a logical name that maps to a policy used to generate credentials. This generates a number of endpoints that are used by the Kubernetes service account to issue and sign these certificates. A policy must be created that enables these paths.

  7. Create a policy named pki that enables read access to the PKI secrets engine paths.

    $ vault policy write pki - <<EOFpath "pki*"                        { capabilities = ["read", "list"] }path "pki/sign/example-dot-com"    { capabilities = ["create", "update"] }path "pki/issue/example-dot-com"   { capabilities = ["create"] }EOF

    Output: 

    Success! Uploaded policy: pki

    These paths enable the token to view all the roles created for this PKI secrets engine and access the sign and issues operations for the example-dot-com role.

  8. Lastly, exit the vault-0 pod.

    $ exit

Configure Kubernetes authentication

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.

  1. Start an interactive shell session on the vault-0 pod.

    $ kubectl exec --stdin=true --tty=true vault-0 -- /bin/sh/ $

    Your system prompt is replaced with a new prompt / $. Commands issued at this prompt are executed on the vault-0 container.

  2. Enable the Kubernetes authentication method.

    $ vault auth enable kubernetesSuccess! Enabled kubernetes auth method at: kubernetes/

  3. Configure the Kubernetes authentication method to use location of the Kubernetes API.

    For the best compatibility with recent Kubernetes versions, ensure you are using Vault v1.9.3 or greater.

$ vault write auth/kubernetes/config \    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

Output: 

Success! Data written to: auth/kubernetes/config

The environment variable KUBERNETES_PORT_443_TCP_ADDR references the internal network address of the Kubernetes host.

You can validate the issuer name of your Kubernetes cluster using [this method](/vault/docs/auth/kubernetes#discovering-the-service-account-issuer).

  1. Create a Kubernetes authentication role named issuer that binds the pki policy with a Kubernetes service account named issuer.

    $ vault write auth/kubernetes/role/issuer \    bound_service_account_names=issuer \    bound_service_account_namespaces=default \    policies=pki \    ttl=20m

    Output: 

    Success! Data written to: auth/kubernetes/role/issuer

    The role connects the Kubernetes service account, issuer, in the default namespace with the pki Vault policy. The tokens returned after authentication are valid for 20 minutes. This Kubernetes service account name, issuer, is created in the Deploy Issuer and Certificate section.

  2. Exit the vault-0 pod.

    $ exit

Deploy Cert Manager

Jetstack's cert-manager is a Kubernetes add-on that automates the management and issuance of TLS certificates from various issuing sources. Vault can be configured as one of those sources. The cert-manager requires the creation of a set of Kubernetes resources that provide the interface to the certificate creation.

  1. Install Jetstack's cert-manager's version 1.12.3 resources.

    $ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v1.12.3/cert-manager.crds.yamlcustomresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io createdcustomresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io createdcustomresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io createdcustomresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io createdcustomresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io createdcustomresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created

  2. Create a namespace named cert-manager to host the cert-manager.

    $ kubectl create namespace cert-managernamespace/cert-manager created

  3. Jetstack's cert-manager Helm chart is available in a repository that they maintain. Helm can request and install Helm charts from these custom repositories.

    Add the jetstack chart repository.

    $ helm repo add jetstack https://charts.jetstack.io"jetstack" has been added to your repositories

  4. Helm maintains a cached list of charts for every repository that it maintains. This list needs to be updated periodically so that Helm knows about all available charts and their releases. A repository recently added needs to be updated before any chart is requested.

    Update the local list of Helm charts.

    $ helm repo updateHang tight while we grab the latest from your chart repositories......Successfully got an update from the "jetstack" chart repositoryUpdate Complete. ⎈ Happy Helming!⎈

  5. The results show that the jetstack chart repository has retrieved an update.

    Install the cert-manager chart version 0.11 in the cert-manager namespace.

    $ helm install cert-manager \    --namespace cert-manager \    --version v1.12.3 \  jetstack/cert-manager

    Output: 

    NAME: cert-manager## ...

    The cert-manager chart deploys a number of pods within the cert-manager namespace.

  6. Get all the pods within the cert-manager namespace.

    $ kubectl get pods --namespace cert-managerNAME                                       READY   STATUS    RESTARTS   AGEcert-manager-66958f45fc-pdf64              1/1     Running   0          27scert-manager-cainjector-755bbf9c6b-gpgtg   1/1     Running   0          27scert-manager-webhook-76954fcbcd-w4lll      1/1     Running   0          27s

  7. Wait until the pods prefixed with cert-manager are running and ready (1/1).

These pods now require configuration to interface with Vault.

Configure an issuer and generate a certificate

The cert-manager enables you to define Issuers that interface with the Vault certificate generating endpoints. These Issuers are invoked when a Certificate is created.

When you configured Vault's Kubernetes authentication a Kubernetes service account, named issuer, was granted the policy, named pki, to the certificate generation endpoints.

  1. Create a service account named issuer within the default namespace.

    $ kubectl create serviceaccount issuerserviceaccount/issuer created

Kubernetes 1.24+

The service account generated a secret that is required by the Issuer automatically in Kubernetes 1.23. In Kubernetes 1.24+, you need to create the secret explicitly.

  1. Create a secret definition.

    issuer-secret.yaml

    $ cat >> issuer-secret.yaml <<EOFapiVersion: v1kind: Secretmetadata:  name: issuer-token-lmzpj  annotations:    kubernetes.io/service-account.name: issuertype: kubernetes.io/service-account-tokenEOF

  2. Create an issuer secret. 

    $ kubectl apply -f issuer-secret.yamlsecret/issuer-token-lmzpj created

  3. Get all the secrets in the default namespace.

    $ kubectl get secretsNAME                          TYPE                                  DATA   AGEissuer-token-lmzpj            kubernetes.io/service-account-token   3      10ssh.helm.release.v1.vault.v1   helm.sh/release.v1                    1      22h

    The issuer secret is displayed here as the secret prefixed with issuer-token.

  4. Create a variable named ISSUER_SECRET_REF to capture the secret name.

    $ ISSUER_SECRET_REF=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("issuer-token-")).name')

  5. Define an Issuer, named vault-issuer, that sets Vault as a certificate issuer.

    $ cat > vault-issuer.yaml <<EOFapiVersion: cert-manager.io/v1kind: Issuermetadata:  name: vault-issuer  namespace: defaultspec:  vault:    server: http://vault.default:8200    path: pki/sign/example-dot-com    auth:      kubernetes:        mountPath: /v1/auth/kubernetes        role: issuer        secretRef:          name: $ISSUER_SECRET_REF          key: tokenEOF

  6. Create the vault-issuer Issuer.

    $ kubectl apply --filename vault-issuer.yamlissuer.cert-manager.io/vault-issuer created

    The specification defines the signing endpoint and the authentication endpoint and credentials.

  7. Define a certificate named example-com.

    $ cat > example-com-cert.yaml <<EOFapiVersion: cert-manager.io/v1kind: Certificatemetadata:  name: example-com  namespace: defaultspec:  secretName: example-com-tls  issuerRef:    name: vault-issuer  commonName: www.example.com  dnsNames:  - www.example.comEOF

    The Certificate, named example-com, requests from Vault the certificate through the Issuer, named vault-issuer. The common name and DNS names are names within the allowed domains for the configured Vault endpoint.

  8. Create the example-com certificate.

    $ kubectl apply --filename example-com-cert.yamlcertificate.cert-manager.io/example-com created

  9. View the details of the example-com certificate.

    $ kubectl describe certificate.cert-manager example-comName:         example-comNamespace:    default...Events:  Type    Reason     Age   From                                       Message  ----    ------     ----  ----                                       -------  Normal  Issuing    10s   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist  Normal  Generated  10s   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "example-com-bpfwz"  Normal  Requested  9s    cert-manager-certificates-request-manager  Created new CertificateRequest resource "example-com-wfclm"  Normal  Issuing    9s    cert-manager-certificates-issuing          The certificate has been successfully issued

    The certificate reports that it has been issued successfully.

Next steps

In this tutorial, you installed Vault configured the PKI secrets engine and Kubernetes authentication. Then installed Jetstack's cert-manager, configured it to use Vault, and requested a certificate.

Besides creation, these certificates can be revoked and removed. Learn more about Jetstack's cert-manager used in this tutorial and explore Vault's PKI secrets engine as a certificate authority in the Build Your Own Certificate Authority.

References

Previous

Securing logs in Confluent HashiCorp Vault

 Next

Vault AWS Lambda extension