Banzai Cloud is now part of Cisco

Banzai Cloud Logo Close
Home Products Benefits Blog Company Contact

The content of this page hasn't been updated for years and might refer to discontinued products and projects.

As part of the Debug 101 series, we’re back hunting a small but annoying bug. This kind of bug is not really a bug, but a side effect of several tools working together.

Here comes trouble 🔗︎

I deploy a development version of Pipeline on a Kubernetes cluster running on top of AWS infrastructure. For this deployment I use the following Helm chart command.

$: helm install --name pipeline banzaicloud-stable/pipeline-cp \
--set=drone.server.env.DRONE_ORGS=banzaicloud \
--set=global.auth.clientid=00000000000 \

Once the deployment is finished, the Pipeline API pod crashes constantly among the other healthy pods.

Okay, no problem. Let’s check the logs:

[GIN-debug] Listening and serving HTTP on :tcp://
[GIN-debug] [ERROR] listen tcp: address :tcp:// too many colons in address

So we recieved an error upon opening a listener port. Moreover, the error says, too many colons in address. Let’s investigate the Go code a little bit. We’ll use Gin as our web framework. A simple Web listener looks like this:

router.Run(":" + viper.GetString("pipeline.port"))

At first, I suspect that an empty attribute in the configuration file is messing with the code. However, if the port is missing, an empty colon would work with any random port. So we are getting a port from somewhere.

func resolveAddress(addr []string) string {
	switch len(addr) {
	case 0:
		if port := os.Getenv("PORT"); port != "" {
			debugPrint("Environment variable PORT=\"%s\"", port)
			return ":" + port
		debugPrint("Environment variable PORT is undefined. Using port :8080 by default")
		return ":8080"
	case 1:
		return addr[0]
		panic("too much parameters")

I check Gin to find out what’s happening at runtime while resolving the address. It’s simple; it uses a PORT environment variable if nothing passes, or else it uses the default port which is 8080. I investigate Golang’s dial lib but find nothing suspicious about its colons.

So we’re getting this from an ENV variable. I check to make sure there is no PORT variable set on the container, but find that there are several.

Viper 🔗︎

To continue, we’ll have to take a small detour to Viper’s Go lib.

Viper is a complete configuration solution for Go applications, including 12-Factor apps. It’s designed to work within an application, and can handle all kinds of configuration requirements and formats.

It’s one of our preferred Golang tools. One of its best features is to automatically merge command line flags, config file values, and environment variables. When using this feature, you can even specify several parsing rules.

An example configuration file looks like this with viper:

port = 9090

The default behavior is such that hierarchy is indicated with dots. so the above configuration file can be used to retrieve viper.GetString("pipeline.port") as a string.

To enable environment variables, we use an extremely simple methodology, like this:

viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))

A little bit of an explanation for the code above: we set up a prefix that’s used in each ENV variable. This is a unique prefix that corresponds to all configurations i.e: PIPELINE_DB_HOST.

Note: viper calls ToUpper method to all ENV variable names.

Because dots are not allowed in an environment variable’s name, we set up a replacer that changes dots to underscores. Last, but not least, we enable the automatic parsing of ENV variables. This means that the example above can be overwritten with the following expression:


Wrapping it up 🔗︎

And, unsurprisingly, there’s an exact environment variable set.

If you’re familiar with Helm and Kubernetes, you may already know what went wrong.

There’s a rarely used feature in Kubernetes wherein service parameters are exported to Pods as environment variables.

A list of all the services that were running when a Container is created is available to that Container in the form of environment variables. Those environment variables match the syntax of Docker links. (docs

If you read the initial Helm command carefully, you’ll notice that I set the release-name to pipeline. In Helm charts, the convention is to prefix all resources with the release name. Here’s a snippet from the helper function of Pipeline’s chart, wherein we create a service as ReleaseName-ChartName.

{{- define "fullname" -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}

Finally, we’ve found the root cause: that viper’s EnvPrefix name was the same as the ReleaseName, and Kubernetes automatically generated overlapping environment variables.

There are several ways to avoid this:

  • use different release name
  • use different ENV prefix
  • use different variable naming

That’s it. It took awhile to debug this, which shows how such a small thing can ruin your day and crash your deployment.