Cloud agnostic bucket management

Banzai Cloud’s Pipeline provides a platform which allows enterprises to develop, deploy and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures—multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, etc.—are a tier zero feature of the Pipeline platform, which we strive to automate and enable for all enterprises.

There are a lot of different storage solutions out there, but Object storage has emerged as the de facto leader in today’s fast-moving world of technology; it’s cost effective at scale, ensures data is available across multiple regions, and scales to petabytes in a highly available and highly durable manner. Today’s modern applications that are deployed to K8s produce and consume large datasets, which are stored in the object storages offered by cloud providers.

Pipeline simplifies the flow and management of object stores for those applications deployed through it. It provides a cloud agnostic unified API to create, list, delete, and check object storage on all major cloud providers, like Amazon, Google, Azure. Moreover, this API uses Bank-Vaults to store/retrieve secrets necessary for object storage-related operations.

There may be prerequisites to creating object stores on cloud providers that must be fulfilled before object stores can be created. Pipeline automates the steps required to create object storage on any cloud provider. Users need only specify the necessary input parameters and they can let Pipeline to take care of everything else.

As mentioned above, when Pipeline executes an object storage operation it talks to Bank-Vaults to retrieve credentials needed to engage the cloud provider’s API.

Object storage management flow in Pipeline looks like this:

1. A user obtains a bearer token that is required by Pipeline’s REST API. This token can be obtained via login, first at http://{pipeline-url}/auth/github/login, then http://{pipeline-url}/api/v1/token by using a web browser.

2. A user registers their cloud provider credentials in Pipeline, which are stored as secrets securely in Bank-Vaults. Note, these credentials must have object storage creation privileges on the cloud provider.

3. A user invokes the creation, deletion, checking, or listing of object stores in the API on Pipeline.

Also note, the create and delete API calls run asynchronously, as these might to be lengthy operations.

When listing object stores, a field called managed indicates whether a store is being managed via Pipeline or not. The delete operation is permitted only on those stores where "managed": true.

Amazon S3 buckets 🔗︎

To create an S3 bucket 🔗︎

curl -X POST \
  'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
	"secretName": "my-amazon-secret",
	"name": "my-s3-bucket1113",
	"properties": {
		"amazon": {
			"location": "eu-west-1"
		}
	}
}'

The above API call creates an S3 bucket on Amazon with the name my-s3-bucket1113 in eu-west-1, using those Amazon credentials referenced by the Vault secret identified as my-amazon-secret.

Verify if the Amazon S3 bucket was created 🔗︎

curl -X HEAD -I \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-s3-bucket1113?cloudType=amazon&location=eu-west-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-amazon-secret'

 HTTP/1.1 200 OK

Amazon console 🔗︎

To list Amazon S3 buckets 🔗︎

curl -X GET \
    'http://{pipeline-url}/api/v1/orgs/1/buckets?cloudType=amazon&location=eu-west-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-amazon-secret'

[
...
  {
    "name": "my-s3-bucket1113",
    "managed": true,
    "location": "eu-west-1"
  },
...
]

The above API call lists all Amazon S3 buckets that the credentials referenced by my-amazon-secret have listing access to. In addition to the name of the S3 bucket, it also returns the region the S3 bucket was originally created in.

To delete the Amazon S3 bucket 🔗︎

curl -X DELETE \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-s3-bucket1113?cloudType=amazon&location=eu-west-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-amazon-secret'

Google Cloud Storage buckets 🔗︎

To create a bucket in Google Cloud Storage 🔗︎

curl -X POST \
  'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
	"secretName": "my-google-secret",
	"name": "my-gs-bucket1113",
	"properties": {
		"google": {
			"location": "eu"
		}
	}
}'

This creates a multi-regional bucket with the name my-gs-bucket1113 in eu location using the Google Cloud credentials referenced by my-google-secret. To create a regional bucket, pass a region name (e.g. europe-west1) as location.

Verify if the bucket was created 🔗︎

curl -X HEAD -I \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-gs-bucket1113?cloudType=google' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-google-secret'

 HTTP/1.1 200 OK

Google console 🔗︎

To list Google Storage buckets 🔗︎

curl -X GET \
    'http://{pipeline-url}/api/v1/orgs/1/buckets?cloudType=google' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-google-secret'

[
...
  {
    "name": "my-gs-bucket1113",
    "managed": true
  },
...
]

The above API call lists all the buckets on Google Cloud Storage that the credential referenced by my-google-secret has listing access to.

To delete the Google bucket 🔗︎

curl -X DELETE \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-gs-bucket1113?cloudType=google' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-google-secret'

Azure Blob Storage 🔗︎

To create an Azure Blob Storage 🔗︎

To create Azure Blob Storage a storage account and a resource group are required:

curl -X POST \
  'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
	"secretName": "my-azure-secret",
	"name": "my-azure-blobstore1113",
	"properties": {
		"azure": {
			"location": "westeurope",
			"storageAccount": "mysa1113",
			"resourceGroup": "myrg1113"
		}
	}
}'

Pipeline verifies that the specified my-sa1113 storage account and my-rg1113 resource group exist, and creates them if they don’t. Only afterward does it create a my-azure-blobstore1113 Azure Blob Storage. Also, it retrieves the access key of the storage account and stores it as a Vault secret so that this accesss key is only available later, when other operations require the account access key.

Verify if the bucket was created 🔗︎

curl -X HEAD -I \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-azure-blobstore1113?cloudType=azure&resourceGroup=myrg1113&storageAccount=mysa1113' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-azure-secret'

 HTTP/1.1 200 OK

Azure portal 🔗︎

To list blob storages on Azure 🔗︎

url -X GET \
    'http://{pipeline-url}/api/v1/orgs/1/buckets?cloudType=azure' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-azure-secret'

[
  ...
  {
    "name": "my-azure-blobstore1113",
    "managed": true,
    "aks": {
      "resourceGroup": "myrg1113",
      "storageAccount": "mysa1113"
    }
  },
  ...
]

The above API call lists all blob stores on Azure that the credential referenced by my-azure-secret has listing access to. In addition to the name of a blob store, it also lists the resource group and storage account it belongs to.

To delete blob storage 🔗︎

In order to delete a blob store, we need the access key of the storage account that holds it. Pipeline then uses this access key, which was stored in Vault at the creation of the blob store.

curl -X DELETE \
    'http://{pipeline-url}/api/v1/orgs/1/buckets/my-azure-blobstore1113?cloudType=azure&resourceGroup=myrg1113&storageAccount=mysa1113' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-azure-secret'

Oracle Cloud Storage 🔗︎

To create a bucket in Oracle Cloud Object storage 🔗︎

curl -X POST \
  'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
	"secretName": "my-oracle-secret",
	"name": "my-oracle-bucket1113",
	"properties": {
		"oracle": {
			"location": "eu-frankfurt-1"
		}
	}
}'

The API call above creates a bucket in Oracle Cloud Storage with name my-oracle-bucket1113 in eu-frankfurt-1, using the Oracle cloud credential referenced by the Vault secret identified by my-oracle-secret.

Verify that the bucket was created 🔗︎

curl -X HEAD -I \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-oracle-bucket1113?cloudType=oracle&location=eu-frankfurt-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-oracle-secret'

HTTP/1.1 200 OK

Oracle Cloud Console 🔗︎

To list buckets on Oracle Cloud 🔗︎

curl -X GET \
  'http://{pipeline-url}/api/v1/orgs/1/buckets?cloudType=oracle' \
  -H "Authorization: Bearer $TOKEN" \
  -H "secretName: my-oracle-secret"

  [
  ...
  {
    "name": "my-oracle-bucket1113",
    "managed": true,
    "location": "eu-frankfurt-1"
  },
  ...
]

The above API call lists all buckets on Oracle Cloud that the credential referenced by my-oracle-secret has listing access to.

To delete the bucket 🔗︎

curl -X HEAD -I \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-oracle-bucket1113?cloudType=oracle&location=eu-frankfurt-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-oracle-secret'

Alibaba Object Storage 🔗︎

To create a bucket in Alibaba Cloud Object storage 🔗︎

curl -X POST \
  'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets' \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
	"secretName": "my-alibaba-secret",
	"name": "my-ali-bucket1113",
	"properties": {
		"alibaba": {
			"location": "oss-eu-central-1"
		}
	}
}'

The API call above creates a bucket in Alibaba Storage with name my-ali-bucket1113 in oss-eu-central-1 using the Alibaba cloud credential referenced by the Vault secret identified by my-alibaba-secret.

Verify if the bucket was created 🔗︎

curl -X HEAD -I \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-ali-bucket1113?cloudType=alibaba&location=oss-eu-central-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-alibaba-secret'

HTTP/1.1 200 OK

Alibaba Cloud Console 🔗︎

To list buckets on Alibaba Cloud 🔗︎

curl -X GET \
    'http://{pipeline-url}/api/v1/orgs/1/buckets?cloudType=alibaba&location=oss-eu-central-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-alibaba-secret'

[
  ...
  {
    "name": "my-ali-bucket1113",
    "managed": true
  },
  ...
]

The above API call lists all buckets on Alibaba Cloud that the credential referenced by my-alibaba-secret has listing access to.

To delete the bucket 🔗︎

curl -X DELETE \
    'http://{pipeline-url}/api/v1/orgs/{orgId}/buckets/my-ali-bucket1113?cloudType=alibaba&location=oss-eu-central-1' \
    -H "Authorization: Bearer $TOKEN" \
    -H 'secretName: my-alibaba-secret'

The Pipeline object storage management API definition 🔗︎

The OpenAPI 3.0.0 definition for the Pipeline’s object storage management API can be found here.