The following tips and commands can help you to troubleshoot your Logging operator installation.
First things to do 🔗︎
-
Check that the necessary CRDs are installed. Issue the following command:
kubectl get crdThe output should include the following CRDs:clusterflows.logging.banzaicloud.io 2019-12-05T15:11:48Z clusteroutputs.logging.banzaicloud.io 2019-12-05T15:11:48Z flows.logging.banzaicloud.io 2019-12-05T15:11:48Z loggings.logging.banzaicloud.io 2019-12-05T15:11:48Z outputs.logging.banzaicloud.io 2019-12-05T15:11:48Z -
Verify that the Logging operator pod is running. Issue the following command:
kubectl get pods |grep logging-operatorThe output should include the a running pod, for example:NAME READY STATUS RESTARTS AGE logging-demo-log-generator-6448d45cd9-z7zk8 1/1 Running 0 24m -
Check the status of your resources. Beginning with Logging Operator 3.8, all custom resources have a
Statusand aProblemsfield. In a healthy system, the Problems field of the resources is empty, for example:kubectl get clusteroutput -ASample output:
NAMESPACE NAME ACTIVE PROBLEMS default nullout trueThe
ACTIVEcolumn indicates that theClusterOutputhas successfully passed theconfigcheckand presented it in the current fluentd configuration. When no errors are reported thePROBLEMScolumn is empty.Take a look at another example, in which we have an incorrect
ClusterFlow.kubectl get clusterflow -o wideSample output:
NAME ACTIVE PROBLEMS all-log true nullout false 1You can see that the nullout
Clusterflowis inactive and there is 1 problem with the configuration. To display the problem, check thestatusfield of the object, for example:kubectl get clusterflow nullout -o=jsonpath='{.status}' | jqSample output:
{ "active": false, "problems": [ "dangling global output reference: nullout2" ], "problemsCount": 1 }
After that, check the following sections for further tips.
- Troubleshooting Fluent Bit »
- Troubleshooting Fluentd »
- Troubleshooting syslog-ng »
- Running on KinD »
Getting Support 🔗︎
If you encounter any problems that the documentation does not address, file an issue or talk to us on the Banzai Cloud Slack channel #logging-operator.
Commercial support is also available for the Logging operator.
Before asking for help, prepare the following information to make troubleshooting faster:
- Logging operator version
- kubernetes version
- helm/chart version (if you installed the Logging operator with helm)
- Logging operator logs
- fluentd configuration
- fluentd logs
- fluentbit configuration
- fluentbit logs
Do not forget to remove any sensitive information (for example, passwords and private keys) before sharing.