Self-Service CI/CD Secrets with Vault

6 minutes read
CICD • Vault • Security

In this post, we demonstrate how developers can manage their own secrets, which can then be used in CI/CD pipelines.

As mentioned, we are using Vault to store secrets, with the aim to have secrets self-serviceable, a singular owner for each secret, and separate secrets per project. It’s also a good excuse to show how we can use Vault’s policy templates to reduce the number of individual policies that have to be created, thus limiting the permissions that a provisioner needs for setting up new CI jobs.

Objectives

  • Secrets are encrypted at rest and in transit, through Vault.
  • Automatic retrieval of secrets per project during CI job execution.
  • Secrets are isolated per project, not accessible across projects.
  • CI jobs can read and delete secrets, but not update them.
  • Enable secrets accountability: every long-lived secret has a single owner (a person). For compliance reasons, there are no shared or group secrets.
  • Users can create/read/update/delete their own secrets, but not others.
  • Secrets access and modification leaves an audit trail.

Principle of Operation

Authentication

We use the AppRole authentication mechanism for machines (e.g., CI jobs), and federate authentication for humans to e.g., Vault authentication backends like Okta or AD. AppRoles are essentially accounts created and managed in Vault. Compared to Vault token roles, they are tied into the identity system, which is crucial for using policy templates (see below).

Access Control

One way to achieve separation of concerns is by using overlapping path schemas for the various actors in a CI systems:

  • CI jobs can read any secrets in their part of the K/V store namespace, e.g.:
secret/cicd/${PROJECT_ID}/*
  • Users can read and write secrets to any project namespace, as long as it is below their own sub-namespace within:
secret/cicd/+/u/${USER}/*

Here, ${PROJECT_ID} represents the role-id of the AppRole for a particular project, and ${USER} is a user’s ID in a configured authentication backend. Wildcards + and * represent single and multiple path components, respectively. Since this proposal is geared towards automation purposes, we choose to use UUIDs as project IDs, that can be searched for with a low chance of false positives. We can configure nicer names, at the expense of having to deal with name clashes, issues when projects get renamed, etc.

Users can also obtain the role-id for a given CI/CD project AppRole. We choose to allow users to read the role-ids for all projects. A role-id would be committed to the project’s code repository, and is thus considered public knowledge.

Example

A user obtains the role-id for a particular project:

$ vault read -field=role_id auth/approle/role/project1/role-id
2bde89db-4235-d1d5-04f7-92acb0b438ec

The user can then create new secrets for project1 under their own namespace (given a particular authentication backend), e.g.:

$ vault kv put \
    secret/cicd/2bde89db-4235-d1d5-04f7-92acb0b438ec/u/${USER}/test1 \
    value=secret

Users can neither manipulate nor read other secrets except their own. CI jobs can then refer to the secrets in their code:

$ vault kv get -field=value \
    secret/cicd/2bde89db-4235-d1d5-04f7-92acb0b438ec/u/${USER}/test1
secret

CI jobs cannot change any secrets, and only read their own. We can optionally allow CI jobs to delete their own secrets, allowing self-service garbage collection.

Secrets Rotation

If a user retires, a new secrets owner needs to step up. Once the new secrets are written to Vault’s secrets backend, the project’s code can be changed to point to the new secrets paths. Subsequently, the old secrets can be garbage collected. If the old secrets are not reused, this ensures seamless secrets rotation and self-service transfer of ownership.

An AppRole’s secret-id can be rotated regularly without disruption. Resulting tokens are short-lived (duration of a CI job, can be use-limited and wrapped as well). All secrets for a given CI jobs can be made inaccessible at once by creating a new role-id, and writing it to the respective AppRole of a project.

Extensions

The scheme described here can be extended straight-forwardly to cater for protected/unprotected branches.

In principle, it is possible to extend the same scheme to any backend that allows to layer the secrets path namespace in similar ways, e.g., for ephemeral tokens. Nevertheless, specific policies may still need to be generated per project, due to limitations in the expressivity of policy templates.

Limitations

Policy templates are a first-order system. That is, we can use template parameters for individual entities like users, approles. However, the templating language is not second-order, i.e., we cannot directly express constraints on arbitrary sets of entities (like groups of users). Only concrete groups can be named in policy templates. Hashicorp’s Sentinel integration offers more expressivity.

Setup

One-time Vault Setup

We are using a Vault AppRole per project for authenticating automated jobs, and Okta for humans to authenticate.

$ vault auth enable approle 
Success! Enabled approle auth method at: approle/

$ APPROLE_ID=$(vault auth list -format=json | jq -r '."approle/".accessor' | tee /dev/stderr)
auth_approle_b8c4b16d

$ AUTH_ID=$(vault auth list -format=json | jq -r '."okta/".accessor' | tee /dev/stderr)   
auth_okta_ee155f41

The CI system can create new approles, and read secret IDs. The secret IDs will be made available to CI jobs. We configure policies so that the CI system itself cannot read role-ids, and hence cannot mint approle access tokens without also obtaining a role-id for elsewhere. In practice, this is a relatively low barrier, given that role-ids are effectively public knowledge.

# we allow specific policies only
$ cat > ci-system.hcl <<EOF
path "auth/approle/role/+" {
   capabilities = [ "create", "update" ]
   allowed_parameters = {
     "policies" = [ "ci-project" ]
   }
}

path "auth/approle/role/+/secret-id" {
   capabilities = [ "update" ]
}
EOF

$ cat > ci-project.hcl <<EOF
path "auth/approle/role/+/role-id" {
   capabilities = [ "read" ]
}

path "secret/data/cicd/{{identity.entity.aliases.${APPROLE_ID}.name}}/*" {
    capabilities = [ "read", "delete", "list" ]
}

path "secret/metadata/cicd/{{identity.entity.aliases.${APPROLE_ID}.name}}/*" {
  capabilities = [ "list" ]
}
EOF

$ cat > ci-user.hcl <<EOF
path "secret/data/cicd/+/u/{{identity.entity.aliases.${AUTH_ID}.name}}/*" {
    capabilities = [ "create", "update", "read", "delete", "list" ]
}

path "secret/metadata/cicd/+/u/{{identity.entity.aliases.${AUTH_ID}.name}}/*" {
  capabilities = [ "list" ]
}
EOF

# write policies
$ vault policy write ci-project ci-project.hcl
$ vault policy write ci-system ci-system.hcl
$ vault policy write ci-user ci-user.hcl

# enable policy for CI users
$ vault write auth/okta/groups/ci-users policies=ci-user

# enable CI system to create restricted AppRoles and read secret IDs
# This is usually attached to a deployment token, e.g., via IAM roles,
# K8s service accounts, etc.
$ vault write auth/aws/role/ci-runner auth_type=ec2 \
    bound_iam_role_arn=arn:aws:iam::xxx:role/yyy \
    policies=ci-system

AppRole per Project

At the creation of each CI project, we need to create a new AppRole which matches the name of the project, ${PROJECT_ID}. CI systems provide various ways to automate this, e.g., webhooks, or a service or plugin that can check just-in-time before a pipeline is kicked off:

$ vault write -force auth/approle/role/${PROJECT_ID} policies=ci-project

CI/CD Workflows

CI/CD Secret-ID Injection

From the CI system, CI Jobs receive a fresh secret-id for each job execution:

$ vault write -force auth/approle/role/${PROJECT_ID}/secret-id
f03c2dd4-d2d6-fc9d-5679-6e7bf82fcf67

The secret-id can be time and use-limited, see AppRole documentation. Due to this, there is not much additional value in response-wrapping it, although that can be done in addition.

CI/CD Job Login

When a CI job runs, it receives the secret-id from the CI system, and the role-id from the project (usually, from the source code repository, or CI job spec). The job (e.g., through consul-template or vault-agent) can use both together to obtain a token, which can then in turn be used to obtain secrets for that particular AppRole (and hence, CI project):

$ vault write -force auth/approle/login \
    role_id=2bde89db-4235-d1d5-04f7-92acb0b438ec \
    secret_id=f03c2dd4-d2d6-fc9d-5679-6e7bf82fcf67

The resulting Vault token can also be response-wrapped.

comments powered by Disqus