Oracle Kubernetes Engine (OKE) - the easy way

Continuing our commitment to support all major cloud providers, today we are adding support for Oracle’s Kubernetes-managed cloud service, OKE – Oracle Kubernetes Engine in Pipeline. We are building a feature rich enterprise-grade application platform on top of Kubernetes - called Pipeline - to deliver a better DevOps experience by automating the lifecycle management of applications. This experience is now available to OKE users as well; here, we will guide you through the first steps of using OKE and summarize the benefits of using Pipeline.

tl;dr: - Pipeline and OKE in a nutshell 🔗︎

• Pipeline now supports OKE - launch clusters, deploy, scale, monitor and manage applications through a UI or a CLI or directly from the REST API or from Go code
• Pipeline automates the management of infrastructural OKE prerequisites such as networks, subnets, load balancers, internet gateways and a few ingress/engress rules
• Cloudinfo adds support for Oracle Cloud instance types
• Bank-Vaults brings Vault-based enterprise grade security to Oracle/OKE clusters
• This PR extends Vault with the Oracle Object Storage backend
• All OKE deployments recieve out of the box secure and centralized logging and monitoring
• Easy deployment of applications to OKE using Pipeline’s API
• Seamless and automated DNS management, provided to all services deployed to OKE
• The unique ability among cloud providers to deploy to bare metal

There is a diverse list of open source projects managed by Oracle, we at Banzai Cloud use, support and have contributed to
Running MySQL on Kubernetes using an operator
Fn and Hollowtrees
Fn - a container native serverless platform

Getting started 🔗︎

There are a few simple steps you need to follow, if you don’t have an Oracle Cloud account already.

Registration

First, you need an Oracle account. The good news is that Oracle grants new users a $300 startup bonus so don’t hesitate, register now. You can register your account by following this TryIt link.

We suggest that you use the eu-frankfurt-1 region, as it has more than enough capacity. Also our performance tests indicate that it is the fastest region. (Kubernetes’ cluster creation time is similar to EKS’, or often faster)

If your home region is not eu-frankfurt-1, you can subscribe to it on this page under the Regions section in order to make it available.

Login

Login to the management console on the following url: https://console.eu-frankfurt-1.oraclecloud.com/. The OKE service doesn’t support federated user login yet, so please use the username/password instead of the SSO (Single Sign-On) login.

Create Compartment

To keep projects seperate, we suggest that you create a new compartment. This way you can avoid conflicting with already existing resources. Navigate to: Identity > Compartments > Create Compartment.

Note: You will need the compartment’s OCID in the future, so please save it.

Policy

The next step is to create a new OKE policy in the root compartment of your tenancy. Follow the links to: Identity > Policies > Create Policy

Note that the policy statement should be: allow service OKE to manage all-resources in tenancy

API Token

Finally, to communicate with the Oracle API we need an API Token.

To generate the API key without a passphrase, use the following command:

openssl genrsa -out ~/.oci/oci_api_key.pem 2048
  

Change the file permissions:

chmod go-rwx ~/.oci/oci_api_key.pem
  

Generate the public key:

openssl rsa -pubout -in ~/.oci/oci_api_key.pem -out ~/.oci/oci_api_key_public.pem
  

Generate the fingerprint with the following command:

openssl rsa -pubout -outform DER -in ~/.oci/oci_api_key.pem | openssl md5 -c
  

The fingerprint helps to distinguish between different API keys. It should look similar to this:

"21:34:56:88:90:ab:cd:ef:12:34:56:78:90:ab:cd:ff"
  

Finally upload the API key at: Identity > Users > User Details > API keys > Add Public Key

Note: You will need your User’s OCID, so please save it.

Create an Oracle Kubernetes cluster - OKE 🔗︎

If you want to start a new Kubernetes Cluster on Oracle Cloud you can follow the steps in this documentation:

• Create 1 network
• Create 3 subnets
• Allocate 2 loadbalancers
• Add 1 internet gateway
• Fill some ingress/egress rules

As you can see this manual describes a long and error prone process, and you would have to repeat large portions of it when launching a new cluster, so we have automated all of these steps. All the resources and prerequisites above are created, updated and deleted by Pipeline, so you won’t have to do anything manually.

Create an OKE secret 🔗︎

The prerequisites for cluster creation and the addition of provider specific secrets are a GitHub account and an OAuth App that has to be setup upfront, as OAuth2 is being used for authentication. Note that we support LDAP/AD based authn/authz to clusters launched by Pipeline for all enterprise users. For our advanced security model, please follow this tag.

Before you start, you will need to create secrets via Pipeline. Note that we put a great deal of emphasis on security, and we store all credentials or sensitive information in Vault using Bank-Vaults. These secrets are never available as environment variables, placed in files, etc - they are securely stored and retrieved from Vault using plugins and interchanges between applications or APIs behind the scenes. All the heavy lifting of dealing with Vault is done by our open source project, Bank-Vaults.

Pipeline stores these secrets in Vault and you can use them by referring them as secret_id.

The relationship between the credential variables and secret keys:

Credential variable name	Secret variable
User’s OCID	user_ocid
API Fingerprint	api_key_fingerprint
API Key	api_key
Compartment OCID	compartment_ocid
Tenant OCID	tenancy_ocid

curl -g --request POST \
  --url 'http://{{url}}/api/v1/orgs/{{orgId}}/secrets' \
  --header 'Authorization: Bearer {{token}}' \
  --header 'Content-Type: application/json' \
  -d '{
	"name": "my-oci-secret",
	"type": "oracle",
	"version": 1,
	"values": {
		"user_ocid": "{{user_ocid}}",
		"api_key_fingerprint": "{{api_key_fingerprint}}",
		"api_key": "{{api_key}}",
		"region": "eu-frankfurt-1",
		"compartment_ocid": "{{compartment_ocid}}",
		"tenancy_ocid": "{{tenancy_ocid}}"
	}
}'

Create an OKE cluster

The following example is a cURL request to create an OKE cluster with 2 nodes:

curl -g --request POST \
  --url 'http://{{url}}/api/v1/orgs/{{orgId}}/clusters' \
  --header 'Authorization: Bearer {{token}}' \
  --header 'Content-Type: application/json' \
  -d '{
    "name":"okecluster-demo-{{$randomInt}}",
    "location": "eu-frankfurt-1",
    "cloud": "oracle",
    "secretId": "{{secret_id}}",
    "properties": {
        "oracle": {
            "version":"v1.10.3",
            "nodePools": {
                "pool1": {
                	"version":"v1.10.3",
                	"count": 2,
                	"image": "Oracle-Linux-7.4",
                	"shape": "VM.Standard1.1",
                	"labels": {
                		"type": "general"
                	}
                }
            }
        }
    }}'

Successful cluster creation can be verified on the OKE cluster status page:

$ kubectl cluster-info
Kubernetes master is running at https://c4tsy2g.eu-frankfurt-1.clusters.oci.oraclecloud.com:6443
KubeDNS is running at https://c4tsy2g.eu-frankfurt-1.clusters.oci.oraclecloud.com:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

$ kubectl get nodes
NAME           STATUS    ROLES     AGE       VERSION
130.61.29.39   Ready     node      21m       v1.10.3
130.61.35.26   Ready     node      23m       v1.10.3

Future plans 🔗︎

We are pushing some advanced features through UAT and, once our users are happy with the results, we will add these features to the master branch.

Autoscaling on custom metrics 🔗︎

All our clusters support metrics based autoscaling. Now that we have added support for centralized monitoring, all OKE and deployment metrics are stored in Prometheus. To bring the same level of support as on all other cloud providers - horizontal, vertical and cluster autoscaling - we are integrating OKE with our autoscaling Kubernetes operators.

Cluster layout / infrastructure recommendations 🔗︎

After Cloudinfo integration, we can query the available instance types per region and their prices through a REST API. This was our first step towards adding Telescopes support. Telescopes is a cluster instance type, and full cluster layout recommender engine. Based on predefined resource requirements such as CPU, memory, GPU or network performance, it recommends a diverse set of cost optimized Kubernetes node pools.

UI integration 🔗︎

We are finishing the first draft of the UI, with support for OKE - as a service - so stay tuned.

If you’d like to learn more about Banzai Cloud check out our other posts in this blog, the Pipeline, Hollowtrees and Bank-Vaults projects.