Kubernetes port hunting

As part of the Debug 101 series, we’re back hunting a small but annoying bug. This kind of bug is not really a bug, but a side effect of several tools working together.

Here comes trouble 🔗︎

I deploy a development version of Pipeline on a Kubernetes cluster running on top of AWS infrastructure. For this deployment I use the following Helm chart command.

$: helm install --name pipeline banzaicloud-stable/pipeline-cp \
--set=drone.server.env.DRONE_ORGS=banzaicloud \
--set=global.auth.clientid=00000000000 \
--set=global.auth.clientsecret=00000000000000000000

Once the deployment is finished, the Pipeline API pod crashes constantly among the other healthy pods.

Okay, no problem. Let’s check the logs:

[GIN-debug] Listening and serving HTTP on :tcp://10.103.190.181:9090
[GIN-debug] [ERROR] listen tcp: address :tcp://10.103.190.181:9090: too many colons in address

So we recieved an error upon opening a listener port. Moreover, the error says, too many colons in address. Let’s investigate the Go code a little bit. We’ll use Gin as our web framework. A simple Web listener looks like this:

router.Run(":" + viper.GetString("pipeline.port"))

At first, I suspect that an empty attribute in the configuration file is messing with the code. However, if the port is missing, an empty colon would work with any random port. So we are getting a port from somewhere.

137
138
139
140
141
142
143
144
145
146
147
148
149
150
151

func resolveAddress(addr []string) string {
	switch len(addr) {
	case 0:
		if port := os.Getenv("PORT"); port != "" {
			debugPrint("Environment variable PORT=\"%s\"", port)
			return ":" + port
		}
		debugPrint("Environment variable PORT is undefined. Using port :8080 by default")
		return ":8080"
	case 1:
		return addr[0]
	default:
		panic("too much parameters")
	}
}

I check Gin to find out what’s happening at runtime while resolving the address. It’s simple; it uses a PORT environment variable if nothing passes, or else it uses the default port which is 8080. I investigate Golang’s dial lib but find nothing suspicious about its colons.

So we’re getting this from an ENV variable. I check to make sure there is no PORT variable set on the container, but find that there are several.

Viper 🔗︎

To continue, we’ll have to take a small detour to Viper’s Go lib.

Viper is a complete configuration solution for Go applications, including 12-Factor apps. It’s designed to work within an application, and can handle all kinds of configuration requirements and formats.

It’s one of our preferred Golang tools. One of its best features is to automatically merge command line flags, config file values, and environment variables. When using this feature, you can even specify several parsing rules.

An example configuration file looks like this with viper:

[pipeline]
port = 9090

The default behavior is such that hierarchy is indicated with dots. so the above configuration file can be used to retrieve viper.GetString("pipeline.port") as a string.

To enable environment variables, we use an extremely simple methodology, like this:

viper.SetEnvPrefix("pipeline")
viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
viper.AutomaticEnv()

A little bit of an explanation for the code above: we set up a prefix that’s used in each ENV variable. This is a unique prefix that corresponds to all configurations i.e: PIPELINE_DB_HOST.

Note: viper calls ToUpper method to all ENV variable names.

Because dots are not allowed in an environment variable’s name, we set up a replacer that changes dots to underscores. Last, but not least, we enable the automatic parsing of ENV variables. This means that the example above can be overwritten with the following expression:

export PIPELINE_PIPELINE_PORT=9000

Wrapping it up 🔗︎

And, unsurprisingly, there’s an exact environment variable set.

If you’re familiar with Helm and Kubernetes, you may already know what went wrong.

There’s a rarely used feature in Kubernetes wherein service parameters are exported to Pods as environment variables.

A list of all the services that were running when a Container is created is available to that Container in the form of environment variables. Those environment variables match the syntax of Docker links. (docs https://kubernetes.io/docs/concepts/containers/container-environment-variables/)

If you read the initial Helm command carefully, you’ll notice that I set the release-name to pipeline. In Helm charts, the convention is to prefix all resources with the release name. Here’s a snippet from the helper function of Pipeline’s chart, wherein we create a service as ReleaseName-ChartName.

{{- define "fullname" -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}

Finally, we’ve found the root cause: that viper’s EnvPrefix name was the same as the ReleaseName, and Kubernetes automatically generated overlapping environment variables.

There are several ways to avoid this:

• use different release name
• use different ENV prefix
• use different variable naming
• …

That’s it. It took awhile to debug this, which shows how such a small thing can ruin your day and crash your deployment.