At Banzai Cloud, we’re building a feature rich platform as a service on Kubernetes, called Pipeline. With Pipeline, we provision large, multi-tenant Kubernetes clusters on all major cloud providers, such as AWS, GCP, Azure and BYOC, and deploy all kinds of predefined or ad-hoc workloads to these clusters. When we needed a way for our users to login and interact with protected endpoints and, at the same time, provide dynamic secrets management support, while simultaneously providing native Kubernetes support for all our applications, we turned to Vault.
Security series:
Authentication and authorization of Pipeline users with OAuth2 and Vault Dynamic credentials with Vault using Kubernetes Service Accounts Dynamic SSH with Vault and Pipeline Secure Kubernetes Deployments with Vault and Pipeline Policy enforcement on K8s with Pipeline The Vault swiss-army knife The Banzai Cloud Vault Operator Vault unseal flow with KMS Kubernetes secret management with Pipeline Container vulnerability scans with Pipeline Kubernetes API proxy with Pipeline
In this blog post we’ll explore one of the many workflows we support: authentication via GitHub using OAuth2 tokens, organization mapping to a Vault team, and seamless SSH in cloud instances belonging to that organization. The cloud instances/VMs become trusted by connecting to Vault via AppRole and signing host keys. This way we neither have to exchange keys with our end users, nor concern ourselves with manually managing or revoking them, all of which is automated by Pipeline and secured by Vault.
All of the following operations are automated by Pipeline, thus, for clusters/control plains launched by Pipeline, these steps may not necessarily need to be taken.
Dynamic SSH credentials 🔗︎
Once we provision a Kubernetes cluster in the cloud or on-prem, our customers typically need to access the hosts, so we need a very dynamic method of distribution access. Static SSH keys that access these clusters (especially so that they can’t be dynamically revoked) are not an option for us and our customers. For this purpose, we’ve decided to use Vault’s SSH Secret backend, which does dynamic Client Key Signing and Host Key Signing for accessing our remote VMs.
Requirements:
- A working Vault server accessible inside your infrastructure (automated by Pipeline)
- A GitHub organization (can be GitHub Enterprise as well, but this demo uses the public version)
- A VM (the examples below are based on Ubuntu)
- An SSH keypair on your machine
Client Key Signing 🔗︎
With this method a user signs (requests a certificate for) his/her SSH public key with a CA key which is stored in Vault. The target machine has to be configured to trust users signed with this key. This is automated by Pipeline as well.
Administrative steps 🔗︎
First and foremost you have to log in as an administrator into Vault:
# Point your Vault CLI to the Vault instance
export VAULT_ADDR=https://vault.yourdomain.com
vault login
Let’s configure a policy for the users who can SSH into the developer VMs. Configure the dev policy like this:
vault write sys/policy/dev policy='
path "ssh-client-signer/sign/dev" {
capabilities = ["update"]
}'
Configure the GitHub Auth method for the users. Everyone who logs in through the GitHub Auth method and is part of the dev team in the specified GitHub organization will be mapped to the dev policy in Vault:
vault auth enable github
vault write auth/github/config organization=myorganizationongithub
vault write auth/github/map/teams/dev value=dev
Enable the ssh secret engine for client key signing on the ssh-client-signer path (can be something else, but make it consistent):
vault secrets enable -path=ssh-client-signer ssh
vault write ssh-client-signer/config/ca generate_signing_key=true
# This policy makes the CA signings valid for 24h, and is acceptable for daily workflow
vault write ssh-client-signer/roles/dev -<<EOH
{
"allow_user_certificates": true,
"allowed_users": "*",
"default_extensions": [
{
"permit-pty": ""
}
],
"key_type": "ca",
"default_user": "ubuntu",
"ttl": "24h"
}
EOH
Download the public key from Vault onto the machine you would like to SSH into:
curl https://${VAULT_ADDR}/v1/ssh-client-signer/public_key > /etc/ssh/trusted-user-ca-keys.pem
Change /etc/ssh/sshd_config and add the following line so it will accept users with signed keys:
TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
Restart sshd to realize changes:
service ssh restart
User steps 🔗︎
Using the following steps, a user mapped to the dev policy can request Vault to sign his/her SSH public key with the trusted CA key:
# Point your Vault CLI to your Vault instance
export VAULT_ADDR=https://vault.yourdomain.com
# Enter your GitHub access token here with read:org capabilities, or create one at https://github.com/settings/tokens
export GITHUB_TOKEN=...
vault login -method github $GITHUB_TOKEN
vault write -field=signed_key ssh-client-signer/sign/dev public_key=@$HOME/.ssh/id_rsa.pub > ~/.ssh/id_rsa-cert.pub
Now you can log in to the VM:
ssh ubuntu@xy.xyz.xyz.xyz
Host Key Signing 🔗︎
How many times have you seen this message?
The authenticity of host 'xy.xyz.xyz.xyz (xy.xyz.xyz.xyz)' can't be established.
ECDSA key fingerprint is SHA256:wu5UbyRRuKw+JwCotA2QzQa7dsgs627pzwIYpXhprqA.
Are you sure you want to continue connecting (yes/no)?
That means you’re unable to confirm that the machine you’re SSHing into is a machine you can trust. With an additional configuration mechanism, and with the help of Vault, you can make sure those VMs are trusted from the get go. This reduces the probability of a user accidentally SSHing into an unmanaged or malicious machine. This is a method wherein the VMs have to authenticate themselves and sign their own keys with Vault. The VMs will use Vault’s AppRole Auth Method which is an auth method oriented to automated workflows (machines and services), and is less useful for human operators.
Administrative steps 🔗︎
Enable the ssh secret engine for host key signing on the ssh-host-signer path (can be something else, but make it consistent):
vault secrets enable -path=ssh-host-signer ssh
vault write ssh-host-signer/config/ca generate_signing_key=true
vault secrets tune -max-lease-ttl=87600h ssh-host-signer
Configure the hostsigning policy and hostrole role:
vault write ssh-host-signer/roles/hostrole \
key_type=ca \
ttl=87600h \
allow_host_certificates=true \
allowed_domains="*" \
allow_subdomains=true
vault write sys/policy/hostsigning policy='
path "ssh-host-signer/sign/hostrole" {
capabilities = ["update"]
}'
Let’s update the dev policy with two additional paths. The two additional AppRole based paths are needed so developers can request AppRole leased temporary tokens, and pass them to a newly started VM via User Data, for example:
vault write sys/policy/dev policy='
path "ssh-client-signer/sign/dev" {
capabilities = ["update"]
}
path "auth/approle/role/hostrole/role-id" {
capabilities = ["read"]
}
path "auth/approle/role/hostrole/secret-id" {
capabilities = ["update"]
}'
Setup AppRole auth method so that hosts will be able to lease Vault tokens for host key (self) signing:
vault write auth/approle/role/hostrole policies=hostsigning \
secret_id_ttl=10m \
token_num_uses=10 \
token_ttl=20m \
token_max_ttl=30m \
secret_id_num_uses=40
Sign the key on the host (log in with AppRole auth method first):
# Get a token with Vault AppRole
VAULT_TOKEN=$(vault write -field token auth/approle/login role_id=${VAULT_ROLE_ID} secret_id=${VAULT_SECRET_ID})
# Log in with that token
vault login ${VAULT_TOKEN}
# Sign the key of this host
vault write -field=signed_key ssh-host-signer/sign/hostrole \
cert_type=host \
public_key=@/etc/ssh/ssh_host_rsa_key.pub > /etc/ssh/ssh_host_rsa_key-cert.pub
# Set permissions on the certificate to be 0640:
chmod 0640 /etc/ssh/ssh_host_rsa_key-cert.pub
Change /etc/ssh/sshd_config and add the host certificate path to it:
# For host keys
HostKey /etc/ssh/ssh_host_rsa_key
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
Restart SSH to realize the changes:
service ssh restart
User steps 🔗︎
When starting a machine:
# Point your Vault CLI to your Vault instance
export VAULT_ADDR=https://vault.yourdomain.com
# Enter your GitHub access token here with read:org capabilities, or create one at https://github.com/settings/tokens
export GITHUB_TOKEN=...
vault login -method github $GITHUB_TOKEN
VAULT_ROLE_ID=$(vault read -field role_id auth/approle/role/hostrole/role-id)
VAULT_SECRET_ID=$(vault write -f -field secret_id auth/approle/role/hostrole/secret-id)
# Pass VAULT_ROLE_ID and VAULT_SECRET_ID to the newly started VM.
When logging into that machine:
# Setup your local ssh configuration to allow hosts signed by this CA (coming from Vault)
CERT_AUTHORITY=$(vault read -field=public_key ssh-host-signer/config/ca)
echo "@cert-authority * $CERT_AUTHORITY" >> ~/.ssh/known_hosts
# If you are using client-side signed keys, sign your key temporarily
vault write -field=signed_key ssh-client-signer/sign/dev public_key=@$HOME/.ssh/id_rsa.pub > ~/.ssh/id_rsa-cert.pub
# SSH into target machines as usual
ssh ubuntu@xy.xyz.xyz.xyz
Learn by the code 🔗︎
We already start our Pipeline Control Plane instances with this method, which you can check out, here, in our GitHub repository.
As we’ve just demonstrated, Vault has a superb engine for Dynamic SSH credentials and Host Key signing, and, with simple configurations and some code, most security features required by enterprises can be quickly and effectively implemented. The whole workflow can be automated with the help of Vault’s dynamic nature and the help of AppRole. For simplicity’s sake we’ll end this post here. However, we’ll keep posting about how we seal and unseal Vault and continue to explore other advanced scenarios in our forthcoming posts. Meanwhile, please check out the open source code on our GitHub and make sure to go through the excellent tutorials and use cases made available by Vault.
Learn more about Bank-Vaults:
- Secret injection webhook improvements
- Backing up Vault with Velero
- Vault replication across multiple datacenters
- Vault secret injection webhook and Istio
- Mutate any kind of k8s resources
- HSM support
- Injecting dynamic configuration with templates
- OIDC issuer discovery for Kubernetes service accounts
- Show all posts related to Bank-Vaults
