This guide describes how to collect application and container logs in Kubernetes using the Logging operator, and how to send them to CloudWatch.

The following figure gives you an overview about how the system works. The Logging operator collects the logs from the application, selects which logs to forward to the output, and sends the selected log messages to the output. For more details about the Logging operator, see the Logging operator overview.

Deploy the Logging operator and a demo Application 🔗︎

Install the Logging operator and a demo application using Helm.

Deploy the Logging operator with Helm 🔗︎

To install the Logging operator using Helm, complete these steps.

Note: For the Helm-based installation you need Helm v3.2.1 or later.

  1. Add the chart repository of the Logging operator using the following commands:

    helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
    helm repo update
    
  2. Install the Logging operator.

    helm upgrade --install --wait --create-namespace --namespace logging logging-operator banzaicloud-stable/logging-operator
    
  3. Install the demo application and its logging definition.

    helm upgrade --install --create-namespace --namespace logging logging-demo banzaicloud-stable/logging-demo \
     --set "cloudwatch.enabled=True" \
     --set "cloudwatch.aws.secret_key=" \
     --set "cloudwatch.aws.access_key=" \
     --set "cloudwatch.aws.region=" \
     --set "cloudwatch.aws.log_group_name=" \
     --set "cloudwatch.aws.log_stream_name="
    
  4. Validate your deployment.

  5. Create logging Namespace

    kubectl create ns logging
    
  6. Create AWS secret

    If you have your $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY set you can use the following snippet.

        kubectl -n logging create secret generic logging-cloudwatch --from-literal "awsAccessKeyId=$AWS_ACCESS_KEY_ID" --from-literal "awsSecretAccessKey=$AWS_SECRET_ACCESS_KEY"
    

    Or set up the secret manually.

        kubectl -n logging apply -f - <<"EOF"
        apiVersion: v1
        kind: Secret
        metadata:
          name: logging-cloudwatch
        type: Opaque
        data:
          awsAccessKeyId: <base64encoded>
          awsSecretAccessKey: <base64encoded>
        EOF
    
  7. Create the logging resource.

    kubectl -n logging apply -f - <<"EOF"
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Logging
    metadata:
      name: default-logging-simple
    spec:
      fluentd: {}
      fluentbit: {}
      controlNamespace: logging
    EOF
    

    Note: You can use the ClusterOutput and ClusterFlow resources only in the controlNamespace.

  8. Create an CloudWatch output definition.

    kubectl -n logging apply -f - <<"EOF"
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Output
    metadata:
     name: cloudwatch-output
     namespace: logging
    spec:
     cloudwatch:
       aws_key_id:
         valueFrom:
           secretKeyRef:
             name: logging-cloudwatch
             key: awsAccessKeyId
       aws_sec_key:
         valueFrom:
           secretKeyRef:
             name: logging-cloudwatch
             key: awsSecretAccessKey
       log_group_name: operator-log-group
       log_stream_name: operator-log-stream
       region: us-east-1
       auto_create_stream: true
       buffer:
         timekey: 30s
         timekey_wait: 30s
         timekey_use_utc: true
    EOF
    

    Note: In production environment, use a longer timekey interval to avoid generating too many objects.

  9. Create a flow resource.

    kubectl -n logging apply -f - <<"EOF"
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Flow
    metadata:
      name: cloudwatch-flow
    spec:
      filters:
        - tag_normaliser: {}
        - parser:
            remove_key_name_field: true
            reserve_data: true
            parse:
              type: nginx
      match:
        - select:
            labels:
              app.kubernetes.io/name: log-generator
      localOutputRefs:
        - cloudwatch-output
    EOF
    
  10. Install the demo application.

    kubectl -n logging apply -f - <<"EOF"
    apiVersion: apps/v1
    kind: Deployment
    metadata:
     name: log-generator
    spec:
     selector:
       matchLabels:
         app.kubernetes.io/name: log-generator
     replicas: 1
     template:
       metadata:
         labels:
           app.kubernetes.io/name: log-generator
       spec:
         containers:
         - name: nginx
           image: banzaicloud/log-generator:0.3.2
    EOF
    
  11. Validate your deployment.

Validate the deployment 🔗︎

If you don’t get the expected result you can find help in the troubleshooting section.