Retrieve secrets with Vault AWS Lambda extension

Challenge

AWS Lambda is a powerful tool available from Amazon Web Services. It offers Serverless functions that cost nothing when not used, and then fractions of a penny when invoked.

A common requirement for a Lambda function is to retrieve information from a secure location, and use it to access a database, file store or other secure location.

If you are using Vault for secrets management, Lambda requires access to Vault for authentication and to retrieve secrets.

Solution

Including the Vault Lambda Extension in the execution environment enables functions to retrieve secrets from Vault.

With the Vault Lambda Extension with your Lambda runtime, and setting the proper environment variables, the function will be able to retrieve secrets from Vault before the function executes. The function now has the credentials needed to implement your business logic.

Background

Lambda creates instances of serverless functions as needed, which the structure includes a few things like a runtime, environment variables and the executable/interpreter. Lambda has standard runtimes for many languages available, but if that runtime does not include a particular library or dependency it can be added a few ways. One of particular interest is the Lambda layer, which is an addition of a library, module or dependency to the standard Lambda runtime for your language. When a Lambda function is invoked, these are all loaded into the runtime process and then executed.

Extensions are additions to the runtime process and can added as a Lambda layer to your runtime environment. If the proper environment variables are set, the Vault AWS Lambda Extension will run authenticate and securely retrieve secrets before your Lambda function invokes.

Scenario introduction

First, using a Terraform config you will create the function with supporting IAM Roles and Policies, and review the infrastructure that is created.

You will then set up HCP Vault Dedicated to allow access from your AWS Account to Vault Dedicated via an IAM Role. After the initial Vault Dedicated setup, you will create a Vault Role and Policy that allow the execution role the Lambda function uses to access one KV Secret (kv/data/test/lambda) in your instance of Vault Dedicated.

Next, you will create a secret in Vault, and then set up AWS Authentication to enable the function you created earlier to connect to your Vault Dedicated instance.

Finally, you will observe it all working together.

Prerequisites

This tutorial requires an AWS account, Terraform, Vault CLI and the example Terraform configuration to create a demonstration environment.

• AWS account - you need to be familiar with the AWS Console UI and where to find the Lambda. For this tutorial, you should create an AWS IAM User that has sufficient permissions to create resources in your account, and generate an Access Key and Secret Access Key for this lab. Please refer to IAM Users for more details.
• Terraform CLI
• AWS CLI
• Git
• HCP account with a service principal with contributor role already created.
  • Some experience with the HCP interface, particularly getting the values for VAULT_TOKEN and VAULT_ADDR.

Clone example repository

1. Clone the Terraform configuration and Lambda function code from the following repository:

   $ git clone https://github.com/hashicorp-education/learn-vault-intro-lambda-extension.git

2. Change into the repository directory.

   $ cd learn-vault-intro-lambda-extension

   The repository contains Terraform configuration to create the following resources:

   • Vault Dedicated cluster
   • Lambda Function
   • IAM Lambda execution role

Set up for the Terraform configuration

1. Log into the HCP Portal.

1. In HCP, under Access control (IAM) choose Service Principals.

2. From the Service Principals page, choose the link that your service principle's name uses and go into the details page.

3. in the Create service principal key page, choose + Generate key.

4. Copy the Client ID then, in a terminal, set the HCP_CLIENT_ID environment variable to the copied value.

   $ export HCP_CLIENT_ID=<COPIED_CLIENT_ID>

5. Switch back to the HCP Portal and copy the Client Secret then, in a terminal, set the HCP_CLIENT_SECRET environment variable to the copied value.

   $ export HCP_CLIENT_SECRET=<COPIED_CLIENT_SECRET>

   Terraform is now able to authenticate with HCP.

6. Set AWS_ACCESS_KEY_ID to store your AWS Access Key.

   $ export AWS_ACCESS_KEY_ID=<<AWS Access Key here>>

7. Set AWS_SECRET_ACCESS_KEY to store your AWS Secret Access Key.

   $ export AWS_SECRET_ACCESS_KEY=<<AWS Secret Access Key here>>

8. Now set the target AWS region.

   $ export TF_VAR_aws_region=us-west-2

Create the infrastructure

1. Run terraform init to initialize the Terraform configuration.

   $ terraform init ...snip... Terraform has been successfully initialized! ...snip...

2. Package the Lambda function through the build script.

   $ ./build.sh $PWD/learn-vault-intro-lambda-extension/demo-function $PWD/learn-vault-intro-lambda-extension Removing old builds... Making new zip...    adding: handler.py (deflated 31%) $PWD/learn-vault-intro-lambda-extension

3. Apply the changes.

   $ terraform apply...Do you want to perform these actions?Terraform will perform the actions described above.Only 'yes' will be accepted to approve.Enter a value: yes

   The terminal output displays the plan that it found and the resources it creates.

   Enter yes to confirm and resume.

   Note

   Keep in mind that answering yes at this time creates actual resources with associated costs.

4. Verify that when the terraform apply command completes, smd you see the following.

    ...snip... Apply complete! Resources: 8 added, 0 changed, 0 destroyed.

Tour your creation

Take a quick tour to learn what was provisioned by Terraform in the AWS and HCP UI.

1. Launch your AWS Account Console and open the Lambda console. Select the vault-lambda-extension-demo-function function.

2. Scroll down to the Code Source tab, and examine the Python code.

   import jsonimport osdef lambda_handler(event, context):   # the location of secret   secret_file = os.environ['VAULT_SECRET_FILE_DB']   f = open(secret_file, "r")   # put the json object into a python dictionary   dct = json.loads(f.read())   # print off your API Key   print("API Key: ", dct["data"]["data"]["api-key"])

   This code is straightforward, getting the location of the secret that is returned from Vault from an environment variable, reading it into a Python map and then printing it to standard output. Understanding the environment variables are critical to understanding the Vault lambda extension.

3. Click on the Configuration tab and click on the environment variables section on the left.

   

   The environment variables should be similar to as follows.

   Variable Name	Value
   VAULT_ADDR	https://XXX-XXXX-XXXX-NNNNNNNN.NNNNNNNN.NN.hashicorp.cloud:8200
   VAULT_AUTH_PROVIDER	aws
   VAULT_AUTH_ROLE	vault-role-for-aws-lambdarole
   VAULT_NAMESPACE	admin
   VAULT_SECRET_FILE_DB	/tmp/vault_secret.json
   VAULT_SECRET_PATH_DB	kv/data/test/ec2

   The Vault Lambda Extension is configured via these environment variables. So VAULT_AUTH_PROVIDER indicates it should use AWS Authentication, VAULT_NAMESPACE indicates the namespace that will be used, and so forth. Please see the previous link for details.

4. In order to use the Vault CLI locally, you need to go to the HCP UI, find your cluster to get the Vault token, and address.

5. Choose your Vault instance and click on Access Vault, and choose Command-line (CLI).

6. From there some variables need to set in your terminal.

   

   $ export VAULT_ADDR="[YOUR_CLUSTER_ADDER_HERE]"; export VAULT_NAMESPACE="admin"

7. You still need to get the token. Go back to the main screen for your Vault cluster and find for the New admin token and choose Generate token. Generating it places it in your paste buffer. Copy the below and then set the admin token with:

   $ export VAULT_TOKEN=[YOUR_TOKEN_HERE]

   Note

   This token expires in 6 hours.

   For more details. please refer to the configure development hosts, paying attention to the HCP sections for details on setting environment variables.

Create a secret in Vault

Create a standard KV secret through CLI.

1. Check Vault status to make sure the CLI is properly set up.

   $ vault statusKey                      Value---                      -----Recovery Seal Type       shamirInitialized              trueSealed                   falseTotal Recovery Shares    1Threshold                1Version                  1.12.0+entBuild Date               2022-10-10T19:00:46ZStorage Type             raftCluster Name             vault-cluster-2127cf0fCluster ID               28bcf54f-94ec-2eed-1e54-2d67b58bbaafHA Enabled               trueHA Cluster               https://XXX.XX.XX.XXX:8201HA Mode                  activeActive Since             2022-10-26T16:57:12.105415484ZRaft Committed Index     413Raft Applied Index       413Last WAL                 173

2. Enable KV Secrets Engine v2.

   $ vault secrets enable -version=2 kvSuccess! Enabled the kv secrets engine at: kv/

3. Create a secret named api-key.

   $ vault kv put kv/test/lambda api-key="ABCDEFG9876"=== Secret Path ===kv/data/test/lambda======= Metadata =======Key                Value---                -----created_time       2022-10-27T16:30:34.405574019Zcustom_metadata    <nil>deletion_time      n/adestroyed          falseversion            1

Enable AWS auth method in Vault

In order for the Lambda function to be able to contact the Vault Dedicated instance you need AWS authentication set up.

1. List the authentication is already enabled.

   $ vault auth listPath      Type        Accessor                  Description----      ----        --------                  -----------token/    ns_token    auth_ns_token_dd101c2c    token based credentials

2. Now create a policy called vault-policy-for-aws-lambda-role, which allows read access the secret created above.

   $ vault policy write vault-policy-for-aws-lambda-role - << EOF# Grant 'read' permission to paths prefixed by 'kv/data/test/lambda'path "kv/data/test/lambda" {    capabilities = [ "read" ]  }EOF

3. List the available policies.

   $ vault policy listdefaulthcp-rootvault-policy-for-aws-lambda-role

4. For your lambda function to be able to connect to your instance of Vault Dedicated, go ahead and enable AWS Authentication.

   $ vault auth enable awsSuccess! Enabled aws auth method at: aws/

5. List the available auth methods.

   $ vault auth listPath      Type        Accessor                  Description----      ----        --------                  -----------aws/      aws         auth_aws_b3ac2184         n/atoken/    ns_token    auth_ns_token_dd101c2c    token based credentials

6. Configure AWS Authentication with the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY already set up.

   $ vault write auth/aws/config/client secret_key=$AWS_SECRET_ACCESS_KEY access_key=$AWS_ACCESS_KEY_IDSuccess! Data written to: auth/aws/config/client

7. Create an AWS auth method role in Vault that gives access to the IAM role vault-lambda-extension-demo-lambda-role and attach the policy vault-policy-for-aws-lambda-role to it.

   Note

   Update the <YOUR_ACCOUNT_ID> with your AWS Account ID in the following command before running it.

   $ vault write auth/aws/role/vault-role-for-aws-lambdarole auth_type=iam bound_iam_principal_arn=arn:aws:iam::<YOUR_ACCOUNT_ID>:role/vault-lambda-extension-demo-lambda-role policies=vault-policy-for-aws-lambda-roleSuccess! Data written to: auth/aws/role/vault-role-for-aws-lambdarole

8. List the roles.

   $ vault list /auth/aws/roleKeys----vault-role-for-aws-lambdarole

9. Examine the details of vault-role-for-aws-lambdarole.

   $ vault read auth/aws/role/vault-role-for-aws-lambdaroleKey                               Value---                               -----allow_instance_migration          falseauth_type                         iambound_account_id                  []bound_ami_id                      []bound_ec2_instance_id             <nil>bound_iam_instance_profile_arn    []bound_iam_principal_arn           [arn:aws:iam::144830530561:role/vault-lambda-extension-demo-lambda-role]bound_iam_principal_id            [AROA3SCJYFFCPXMX5QMOP]bound_iam_role_arn                []bound_region                      []bound_subnet_id                   []bound_vpc_id                      []disallow_reauthentication         falseinferred_aws_region               n/ainferred_entity_type              n/apolicies                          [vault-policy-for-aws-lambda-role]resolve_aws_unique_ids            truerole_id                           b821fe4b-ff5c-22ac-9594-7f95858a64fcrole_tag                          n/atoken_bound_cidrs                 []token_explicit_max_ttl            0stoken_max_ttl                     0stoken_no_default_policy           falsetoken_num_uses                    0token_period                      0stoken_policies                    [vault-policy-for-aws-lambda-role]token_ttl                         0stoken_type                        default

Test the function

1. Go back to the AWS Console, find vault-lambda-extension-demo-function and choose the Test tab and press Test button.

2. Examine the output. In addition to a summary of execution, find for text similar to the below.

   2022-10-28T14:45:04.456Z [INFO]  vault-lambda-extension: Initialising2022-10-28T14:45:04.667Z [INFO]  vault-lambda-extension: Initialised2022-10-28T14:45:04.667Z [INFO]  vault-lambda-extension: Starting HTTP proxy server220140c1-4d3c-4656-b9ba-a6df57a1f1d82022-10-28T14:45:04.668Z [INFO]  vault-lambda-extension: Waiting for event...EXTENSION   Name: vault-lambda-extension    State: Ready    Events: [INVOKE,SHUTDOWN]START RequestId: 7588b408-1ae6-4866-822b-cb19947d50b5 Version: $LATEST2022-10-28T14:45:04.774Z [INFO]  vault-lambda-extension: Received event2022-10-28T14:45:04.774Z [INFO]  vault-lambda-extension: Waiting for event...API Key:  ABCDEFG9876END RequestId: 7588b408-1ae6-4866-822b-cb19947d50b5REPORT RequestId: 7588b408-1ae6-4866-822b-cb19947d50b5  Duration: 1.54 ms   Billed Duration: 2 ms   Memory Size: 128 MB Max Memory Used: 65 MB  Init Duration: 437.45 ms

   From the line with API Key: ABCDEFG9876 you know that the extension retrieved the API_KEY value from Vault.

Clean up your infrastructure

1. Clean up your infrastructure provisioned by Terraform.

   $ terraform destroy

   The terraform destroy deletes the HCP Cluster, Lambda function and all the support structures - IAM Role, Policy, etc.

2. Delete the terraform state.

   $ rm *tfstate*

3. In the AWS IAM Console, find the AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID you created for this tutorial and make them inactive and delete them both.

4. Unset all the environment variables used in this tutorial.

   $ unset AWS_SECRET_ACCESS_KEY; unset AWS_ACCESS_KEY_ID; unset HCP_CLIENT_ID; unset HCP_CLIENT_SECRET

5. Check the environment variables were unset:

   $ env | grep 'AWS\|HCP'

   If the variables were successfully unset they will not appear in the result of this command. An execution and successful unset will provide no output.

Next steps

In this tutorial you built and deployed a Lambda function using the Vault AWS Lambda Extension using Terraform. If you are interested in more detail on using Terraform to manage Vault Dedicated refer to Manage Codified Vault on HCP Vault Dedicated with Terraform That simple function retrieved a simple KV secret from Vault, and printed it to the CloudWatch logs. The extension is configured by the Lambda environment variables, and automatically authenticates and retrieves a secret before the function is invoked.

You can learn more about the Vault AWS Lambda Extension by reading the blog post Use AWS Lambda Extensions to Securely Retrieve Secrets From HashiCorp Vault and by reviewing the Vault AWS Lambda Extension code repository. Of particular interest are the environment variables used to configure the extension.

Next, you enabled AWS Authentication for Vault Dedicated, and you can learn more through the Set up AWS Auth Method for HCP Vault Dedicated tutorial and the documentation AWS Authentication.

If you are interested in a lab with a more complex example, you can try the Lambda Vault Extension tutorial. That tutorial has a more detailed scenario using a secret to then access a MYSQL Database. In addition there is a section on caching the secret retrieved from Vault.

References

• Vault AWS Lambda Extension
• AWS Lambda Extensions in container images.
• GitHub for Vault Lambda Extension
• Learn Vault Lambda Extension
• Set up AWS Auth Method for HCP Vault Dedicated
• Deploy HCP Vault Dedicated with Terraform
• AWS Authentication
• Introducing AWS Lambda Extensions