FluentBit Configuration

Configure FluentBit to collect workflow events from Quarkus Flow applications.

Container Runtime Detection

FluentBit must use the correct parser based on your Kubernetes cluster’s container runtime. Using the wrong parser causes FluentBit to fail silently - it will tail log files but won’t parse any events.

Check Your Cluster’s Runtime

# For KIND clusters
docker exec <cluster-name>-control-plane crictl version

# For real Kubernetes clusters
kubectl get nodes -o wide
# Look at CONTAINER-RUNTIME column

# Or quick test
kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.containerRuntimeVersion}'

Output examples:

containerd://1.7.2 → Use CRI parser
cri-o://1.25.0 → Use CRI parser
docker://20.10.21 → Use Docker parser

Parser Configuration

CRI Runtime (Most Common)

Used by: KIND, GKE, EKS (recent versions), AKS, most modern Kubernetes

Log format:

2026-04-23T23:07:15.123456789Z stdout F {"eventType":"io.serverlessworkflow.workflow.started.v1",...}
└────────timestamp──────────┘  └stream┘ └logtag┘ └──────────────message──────────────────────────┘

FluentBit configuration:

fluent-bit.conf:

[INPUT]
    Name              tail
    Path              /var/log/containers/*_workflows_*.log
    Parser            cri     # Use CRI parser
    Tag               kube.*

parsers.conf:

[PARSER]
    Name   cri
    Format regex
    Regex  ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ
    Time_Keep On

Docker Runtime (Legacy)

Used by: Older Kubernetes clusters, Docker Desktop Kubernetes

Log format:

{"log":"{\"eventType\":\"io.serverlessworkflow.workflow.started.v1\",...}\n","stream":"stdout","time":"2026-04-23T23:07:15.123456789Z"}

FluentBit configuration:

fluent-bit.conf:

[INPUT]
    Name              tail
    Path              /var/log/containers/*_workflows_*.log
    Parser            docker  # Use Docker parser
    Tag               kube.*

parsers.conf:

[PARSER]
    Name   docker
    Format json
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ
    Time_Keep On
    Decode_Field_As escaped log

Namespace Configuration

FluentBit tails logs from specific namespaces using the pattern:

/var/log/containers/*_${WORKFLOW_NAMESPACE}_*.log

Example log file:

/var/log/containers/workflow-test-app-7d8f9c6b5-abc123_workflows_workflow-app-xyz789.log
                     └──────pod-name──────────┘  └─namespace┘ └──container-name+id──┘

Single Namespace

Set in DaemonSet:

env:
- name: WORKFLOW_NAMESPACE
  value: "workflows"  # Change to match your deployment namespace

Multiple Namespaces

Option 1: Multiple Path entries

[INPUT]
    Name    tail
    Path    /var/log/containers/*_workflows_*.log
    Path    /var/log/containers/*_production_*.log
    Path    /var/log/containers/*_staging_*.log
    Parser  cri
    Tag     kube.*

Option 2: Wildcard + Filter (for dynamic namespaces)

[INPUT]
    Name    tail
    Path    /var/log/containers/*.log
    Parser  cri
    Tag     kube.*

[FILTER]
    Name    kubernetes
    Match   kube.*

[FILTER]
    Name    grep
    Match   kube.*
    Regex   kubernetes.namespace_name ^(workflows|production|staging)$

Use Option 1 if you know the namespaces (better performance). Use Option 2 for dynamic namespaces.

Nested JSON Parsing

After the CRI/Docker parser extracts the log field, you need a second parser for the JSON event:

[FILTER]
    Name              parser
    Match             kube.*
    Key_Name          log        # Parse the 'log' field
    Parser            json       # Use JSON parser
    Reserve_Data      On
    Preserve_Key      Off

Output Configuration by Storage Backend

FluentBit output configuration depends on your storage backend choice.

PostgreSQL Output (MODE 1)

FluentBit uses Lua script (flatten-event.lua) to route events and PostgreSQL output plugin.

PostgreSQL output configuration:

[OUTPUT]
    Name         pgsql
    Match        workflow.*
    Host         ${POSTGRESQL_HOST}
    Port         ${POSTGRESQL_PORT}
    User         ${POSTGRESQL_USER}
    Password     ${POSTGRESQL_PASSWORD}
    Database     ${POSTGRESQL_DATABASE}
    Table        workflow_events_raw
    Timestamp_Key time

Lua script routing:

[FILTER]
    Name    lua
    Match   kube.*
    script  flatten-event.lua
    call    flatten_event

What the Lua script does:

Reads the eventType field from each JSON event
Routes workflow events (e.g., io.serverlessworkflow.workflow.started.v1) to tag workflow.*
Routes task events (e.g., io.serverlessworkflow.task.started.v1) to tag task.*
Flattens nested Kubernetes metadata for easier querying

See: scripts/fluentbit/postgresql/ for complete PostgreSQL mode configuration.

Elasticsearch Output (MODE 2)

FluentBit uses rewrite_tag filters for routing and Elasticsearch output plugin.

Elasticsearch output configuration:

[OUTPUT]
    Name            es
    Match           workflow.instance
    Host            ${ELASTICSEARCH_HOST}
    Port            ${ELASTICSEARCH_PORT}
    Index           workflow-instance-events-raw
    Type            _doc
    Logstash_Format On
    Logstag_Prefix workflow-instance-events-raw
    Logstash_DateFormat %Y.%m.%d
    Retry_Limit     5
    Suppress_Type_Name On
    tls             ${ELASTICSEARCH_TLS}
    tls.verify      ${ELASTICSEARCH_TLS_VERIFY}

[OUTPUT]
    Name            es
    Match           workflow.task
    Host            ${ELASTICSEARCH_HOST}
    Port            ${ELASTICSEARCH_PORT}
    Index           task-execution-events-raw
    Type            _doc
    Logstash_Format On
    Logstash_Prefix task-execution-events-raw
    Logstash_DateFormat %Y.%m.%d
    Retry_Limit     5
    Suppress_Type_Name On
    tls             ${ELASTICSEARCH_TLS}
    tls.verify      ${ELASTICSEARCH_TLS_VERIFY}

Event routing (rewrite_tag filters):

[FILTER]
    Name          rewrite_tag
    Match         flow.events
    Rule          $eventType ^io\.serverlessworkflow\.workflow\. workflow.instance false
    Emitter_Mem_Buf_Limit 10M

[FILTER]
    Name          rewrite_tag
    Match         flow.events
    Rule          $eventType ^io\.serverlessworkflow\.task\. workflow.task false
    Emitter_Mem_Buf_Limit 10M

Index naming:

Workflow events → workflow-instance-events-raw-YYYY.MM.DD
Task events → task-execution-events-raw-YYYY.MM.DD
Date-based rollover enables ILM management (automatic deletion after retention period)

Elasticsearch Transform normalization:

After FluentBit writes raw events to Elasticsearch, Transforms aggregate them:

workflow-instance-events-raw-* → workflow-instances (normalized index)
task-execution-events-raw-* → task-executions (normalized index)

See: scripts/fluentbit/elasticsearch/ for complete Elasticsearch mode configuration.

Debugging FluentBit Output

Check which output is configured:

# PostgreSQL mode
kubectl get configmap -n logging workflows-fluent-bit-config -o yaml | grep -A 10 "\[OUTPUT\]"

# Elasticsearch mode
kubectl get configmap -n logging workflows-fluent-bit-config -o yaml | grep -A 10 "\[OUTPUT\]"

PostgreSQL mode errors:

# Check FluentBit logs for PostgreSQL connection issues
kubectl logs -n logging -l app=workflows-fluent-bit | grep -i pgsql

# Common errors:
# - "connection refused" → PostgreSQL service not reachable
# - "authentication failed" → Incorrect credentials
# - "relation does not exist" → Table not created

Elasticsearch mode errors:

# Check FluentBit logs for Elasticsearch connection issues
kubectl logs -n logging -l app=workflows-fluent-bit | grep -i elasticsearch

# Common errors:
# - "connection refused" → Elasticsearch service not reachable
# - "unauthorized" → Incorrect credentials
# - "index creation failed" → Index template misconfiguration

Lua script debugging (PostgreSQL mode only):

# Check FluentBit logs for Lua errors
kubectl logs -n logging -l app=workflows-fluent-bit | grep -i lua

# Common errors:
# - "attempt to index nil value" → Missing field in event JSON
# - "syntax error" → Invalid Lua syntax in script

After modifying FluentBit configuration, regenerate the ConfigMap using generate-configmap.sh and restart FluentBit pods.

Common Issues

Events Not Reaching Storage

Symptoms:

FluentBit logs show no errors
FluentBit is tailing log files (inotify_fs_add messages)
No events appear in storage
No processing activity in FluentBit logs

Diagnosis:

# Check FluentBit logs for parser errors
kubectl logs -n logging -l app=workflows-fluent-bit | grep -i "parser\|error"

# Expected error if using wrong parser:
# [error] [input:tail:tail.0] parser 'docker' is not registered

Solution:

Detect container runtime (see above)
Update fluent-bit.conf INPUT section with correct parser
Ensure matching parser exists in parsers.conf
Regenerate ConfigMap and restart FluentBit pods

Parser Not Registered

Error:

[error] [input:tail:tail.0] parser 'cri' is not registered

Solution:

Add the parser definition to parsers.conf:

[PARSER]
    Name   cri
    Format regex
    Regex  ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ
    Time_Keep On

Verification

Verify Parser is Working

# Restart FluentBit
kubectl delete pod -n logging -l app=workflows-fluent-bit

# Trigger a workflow
curl -X POST http://localhost:8082/test-workflows/simple-set \
  -H "Content-Type: application/json" \
  -d '{}'

# Check FluentBit is processing events
kubectl logs -n logging -l app=workflows-fluent-bit --tail=100 | grep -E "stdout|eventType"

Verify PostgreSQL Storage (MODE 1)

# Check raw events table
kubectl exec -n postgresql postgresql-0 -- \
  env PGPASSWORD=dataindex123 psql -U dataindex -d dataindex -c \
  "SELECT COUNT(*) FROM workflow_events_raw;"

# Check normalized table
kubectl exec -n postgresql postgresql-0 -- \
  env PGPASSWORD=dataindex123 psql -U dataindex -d dataindex -c \
  "SELECT id, name, status FROM workflow_instances LIMIT 5;"

Expected: Count should increase after triggering workflows, and normalized table should contain workflow instances.

Verify Elasticsearch Storage (MODE 2)

# Port-forward to Elasticsearch
kubectl port-forward -n elasticsearch svc/elasticsearch 9200:9200 &

# Check raw events index
curl -s http://localhost:9200/workflow-instance-events-raw-*/_count | jq .

# Check normalized index (after transform runs, ~1s delay)
curl -s http://localhost:9200/workflow-instances/_search?size=5 | jq '.hits.hits[]._source'

# Check transform status
curl -s http://localhost:9200/_transform/workflow-instance-transform | jq '.transforms[0].stats'

Expected: Raw events index count should increase immediately, normalized index populated after ~1 second delay.

KIND-Specific Notes

KIND always uses containerd (CRI runtime), even though KIND itself runs in Docker.

For KIND clusters, always use the CRI parser.

Production Checklist

Identified container runtime (CRI vs Docker)
Configured correct parser in fluent-bit.conf INPUT section
Added matching parser definition to parsers.conf
Regenerated FluentBit ConfigMap
Tested with real workflow execution
Verified events reaching storage
Documented runtime in deployment notes

FluentBit Configuration

Container Runtime Detection

Check Your Cluster’s Runtime

Parser Configuration

CRI Runtime (Most Common)

Docker Runtime (Legacy)

Namespace Configuration

Single Namespace

Multiple Namespaces

Nested JSON Parsing

Output Configuration by Storage Backend

PostgreSQL Output (MODE 1)

Elasticsearch Output (MODE 2)

Debugging FluentBit Output

Common Issues

Events Not Reaching Storage

Parser Not Registered

Verification

Verify Parser is Working

Verify PostgreSQL Storage (MODE 1)

Verify Elasticsearch Storage (MODE 2)

KIND-Specific Notes

Production Checklist

References