Outputs and ClusterOutputs

For the full details on configuring Outputs and ClusterOutputs, see the Banzai Cloud Logging operator documentation.

• Configuration
• YAML Examples
  • Cluster Output to ElasticSearch
  • Output to Splunk
  • Output to Syslog
  • Unsupported Outputs

Configuration

• Outputs
• ClusterOutputs

Outputs

The Output resource defines where your Flows can send the log messages. Outputs are the final stage for a logging Flow.

The Output is a namespaced resource, which means only a Flow within the same namespace can access it.

You can use secrets in these definitions, but they must also be in the same namespace.

Outputs can be configured by filling out forms in the Rancher UI.

For the details of Output custom resource, see OutputSpec.

The Rancher UI provides forms for configuring the following Output types:

• Amazon ElasticSearch
• Azure Storage
• Cloudwatch
• Datadog
• Elasticsearch
• File
• Fluentd
• GCS
• Kafka
• Kinesis Stream
• LogDNA
• LogZ
• Loki
• New Relic
• Splunk
• SumoLogic
• Syslog

The Rancher UI provides forms for configuring the Output type, target, and access credentials if applicable.

For example configuration for each logging plugin supported by the logging operator, see the logging operator documentation.

ClusterOutputs

ClusterOutput defines an Output without namespace restrictions. It is only effective when deployed in the same namespace as the logging operator.

ClusterOutputs can be configured by filling out forms in the Rancher UI.

For the details of the ClusterOutput custom resource, see ClusterOutput.

YAML Examples

Once logging is installed, you can use these examples to help craft your own logging pipeline.

• Cluster Output to ElasticSearch
• Output to Splunk
• Output to Syslog
• Unsupported Outputs

Cluster Output to ElasticSearch

Let's say you wanted to send all logs in your cluster to an elasticsearch cluster. First, we create a cluster Output.

apiVersion: logging.banzaicloud.io/v1beta1kind: ClusterOutputmetadata:    name: "example-es"    namespace: "cattle-logging-system"spec:    elasticsearch:      host: elasticsearch.example.com      port: 9200      scheme: http

We have created this ClusterOutput, without elasticsearch configuration, in the same namespace as our operator: cattle-logging-system.. Any time we create a ClusterFlow or ClusterOutput, we have to put it in the cattle-logging-system namespace.

Now that we have configured where we want the logs to go, let's configure all logs to go to that ClusterOutput.

apiVersion: logging.banzaicloud.io/v1beta1kind: ClusterFlowmetadata:    name: "all-logs"    namespace: "cattle-logging-system"spec:  globalOutputRefs:    - "example-es"

We should now see our configured index with logs in it.

Output to Splunk

What if we have an application team who only wants logs from a specific namespaces sent to a splunk server? For this case, we can use namespaced Outputs and Flows.

Before we start, let's set up that team's application: coolapp.

apiVersion: v1kind: Namespacemetadata:  name: devteam---apiVersion: apps/v1kind: Deploymentmetadata:  name: coolapp  namespace: devteam  labels:    app: coolappspec:  replicas: 2  selector:    matchLabels:      app: coolapp  template:    metadata:      labels:        app: coolapp    spec:      containers:        - name: generator          image: paynejacob/loggenerator:latest

With coolapp running, we will follow a similar path as when we created a ClusterOutput. However, unlike ClusterOutputs, we create our Output in our application's namespace.

apiVersion: logging.banzaicloud.io/v1beta1kind: Outputmetadata:    name: "devteam-splunk"    namespace: "devteam"spec:    SplunkHec:        host: splunk.example.com        port: 8088        protocol: http

Once again, let's feed our Output some logs:

apiVersion: logging.banzaicloud.io/v1beta1kind: Flowmetadata:    name: "devteam-logs"    namespace: "devteam"spec:  localOutputRefs:    - "devteam-splunk"

Output to Syslog

Let's say you wanted to send all logs in your cluster to an syslog server. First, we create a ClusterOutput:

apiVersion: logging.banzaicloud.io/v1beta1    kind: ClusterOutput    metadata:      name: "example-syslog"      namespace: "cattle-logging-system"    spec:      syslog:        buffer:          timekey: 30s          timekey_use_utc: true          timekey_wait: 10s          flush_interval: 5s        format:          type: json          app_name_field: test        host: syslog.example.com        insecure: true        port: 514        transport: tcp

Now that we have configured where we want the logs to go, let's configure all logs to go to that Output.

apiVersion: logging.banzaicloud.io/v1beta1    kind: ClusterFlow    metadata:      name: "all-logs"      namespace: cattle-logging-system    spec:      globalOutputRefs:        - "example-syslog"

Unsupported Outputs

For the final example, we create an Output to write logs to a destination that is not supported out of the box:

> Note on syslog syslog is a supported Output. However, this example still provides an overview on using unsupported plugins.

apiVersion: v1kind: Secretmetadata:  name: syslog-config  namespace: cattle-logging-systemtype: OpaquestringData:  fluent-bit.conf: |    [INPUT]        Name              forward        Port              24224
    [OUTPUT]        Name              syslog        InstanceName      syslog-output        Match             *        Addr              syslog.example.com        Port              514        Cluster           ranchers
---apiVersion: apps/v1kind: Deploymentmetadata:  name: fluentbit-syslog-forwarder  namespace: cattle-logging-system  labels:    output: syslogspec:  selector:    matchLabels:      output: syslog  template:    metadata:      labels:        output: syslog    spec:      containers:      - name: fluentbit        image: paynejacob/fluent-bit-out-syslog:latest        ports:          - containerPort: 24224        volumeMounts:          - mountPath: "/fluent-bit/etc/"            name: configuration      volumes:      - name: configuration        secret:          secretName: syslog-config---apiVersion: v1kind: Servicemetadata:  name: syslog-forwarder  namespace: cattle-logging-systemspec:  selector:    output: syslog  ports:    - protocol: TCP      port: 24224      targetPort: 24224---apiVersion: logging.banzaicloud.io/v1beta1kind: ClusterFlowmetadata:  name: all-logs  namespace: cattle-logging-systemspec:  globalOutputRefs:    - syslog---apiVersion: logging.banzaicloud.io/v1beta1kind: ClusterOutputmetadata:  name: syslog  namespace: cattle-logging-systemspec:  forward:    servers:      - host: "syslog-forwarder.cattle-logging-system"    require_ack_response: false    ignore_network_errors_at_startup: false

Let's break down what is happening here. First, we create a deployment of a container that has the additional syslog plugin and accepts logs forwarded from another fluentd. Next we create an Output configured as a forwarder to our deployment. The deployment fluentd will then forward all logs to the configured syslog destination.