Fluent Operator

In Kubernetes clusters, applications and system services generate a large number of logs. To facilitate analysis and storage of these logs, they can be collected in systems such as Elasticsearch, Loki, Kafka, Prometheus, and others.

Fluent Bit and Fluentd are tools for log collection and processing. However, configuring them in Kubernetes can be complex:

• Manually creating resources (DaemonSet, StatefulSet) is required.
• Configuration files need to be set up.
• Container runtime specifics must be considered.
• Secrets must be managed for authentication.
• Updates and compatibility need to be monitored.

Fluent Operator simplifies this process by:

• Automating the deployment of Fluent Bit and Fluentd, creating the necessary resources.
• Using Custom Resource Definitions (CRD) to configure inputs, filters, outputs, and other parameters via YAML manifests.
• Managing updates and scaling of Fluent Bit/Fluentd in response to cluster changes.
• Simplifying integration with Elasticsearch, Loki, Kafka, Prometheus, and others.

Instead of manually configuring Fluent Bit and Fluentd, you install Fluent Operator and describe the settings. The operator brings the cluster to the desired state and automatically updates the agents when the configuration changes.

Main Components

• Fluent Bit: A lightweight log collection agent.
• Fluentd: A powerful agent that can be used as a DaemonSet or StatefulSet for advanced filtering and buffering.
• Fluent Operator: Manages Fluent Bit and Fluentd through CRDs.

Installation

You can install Fluent Operator via the Hostman control panel:

1. Go to the Kubernetes section and click on the cluster.
2. Navigate to the Addons tab and select Fluent Operator.
3. Select Advanced Installation and adjust the configuration for your needs.
4. Click Install.

Installation Parameters

Operator

The operator section:

• containerRuntime: Defines the container runtime used. In our managed Kubernetes, containerd is used. This value does not need to be changed.
• logPath: Log file path. For containerd, it is /var/log.
• disableComponentControllers: A list of Fluent Operator components to disable. Possible values: fluent-bit and fluentd. By default, both components are installed.

Fluent Bit

The input section:

• tail:
  • enable: true — Enables log collection from files located at /var/log/containers/*.log.
  • path: "/var/log/containers/*.log" — Defines the path to container log files.
  • refreshIntervalSeconds: 10 — The frequency (in seconds) for refreshing the log file list.
  • memBufLimit: 100MB — Memory limit for buffering.
  • storageType: memory — Storage type for the buffer. An alternative is filesystem, but a volume must be prepared in advance.
  • readFromHead: false — Starts reading logs from the end.
  • skipLongLines: true — Skips excessively long lines.
• systemd:
  • enable: true — Enables log collection from journald.
  • includeKubelet: true — Includes kubelet logs from journald.
  • storageType: memory — Sets the storage type for journald log buffers.

The output section:

Lists the main output options. To enable each one, set enable: true and specify the parameters:

• Elasticsearch

  es:
    enable: false
    host: "<Elasticsearch url>"
    port: 9200
    logstashPrefix: ks-logstash-log
    bufferSize: 20MB
    traceError: true
    httpUser: "<username>"
    httpPassword: "<password>"

Configuration for sending logs to Elasticsearch. Specify the service URL, port, and authentication credentials.

• Kafka

  kafka:
    enable: false
    brokers: "<kafka broker list>"
    topics: ks-log
    bufferSize: 20MB
    retryLimit: "10"

Configuration for sending logs to Kafka. Specify a comma-separated list of brokers and the topic for logs.

• Loki

  loki:
    enable: false
    retryLimit: "no_limits"
    host: 127.0.0.1
    port: 3100
    httpUser: myuser
    httpPassword: mypass
    tenantId: "<tenant>"

Allows sending logs to Loki. Specify the host, port, credentials, and tenant ID.

• stdout

  stdout:
    enable: true

Useful for testing, as it outputs logs to standard output (stdout).

The filter section:

• multiline:
  • enable: false — By default, multiline is disabled.
  • You can configure parsers (go, python, java, etc.) or add custom ones.
• kubernetes:
  • enable: true — Enables the Kubernetes filter to add pod metadata, such as namespace, pod, labels, and annotations.
  • labels: false, annotations: false — By default, these fields are not added.
• containerd:
  • enable: true — Enables the filter for parsing Containerd logs.
• systemd:
  • enable: true — Enables the filter for parsing journald logs.

Fluentd

• enable — If you need to use Fluentd (instead of or alongside Fluent Bit), set it to true.
• mode:
  • collector — Deploys as a StatefulSet.
  • agent — Deploys as a DaemonSet (like Fluent Bit), meaning one agent per node.
• forward:
  • port: 24224 — The default Fluentd input for forwarding.
• output — Similar to Fluent Bit, it includes es, kafka, opensearch, etc.

Removing Fluent Operator

Uninstalling the add-on via the management panel is not supported, but it can be done manually. Follow these steps:

1. Delete the fluent-bit object:

kubectl delete -n fluent-operator fluentbit/fluent-bit

2. Uninstall the fluent-operator Helm release:

helm uninstall -n fluent-operator fluent-operator

3. Delete all CRDs related to fluent-operator:

kubectl delete crd $(kubectl get crd | grep fluent | awk '{print $1}')

4. Check that all related resources and namespaces are removed:

kubectl get namespaces

After completing these steps, Fluent Operator will be completely removed from your cluster.