FluentBit Configuration (MODE 1 - PostgreSQL)

MODE 1 (PostgreSQL) ONLY

This configuration guide applies to MODE 1 (PostgreSQL) deployments only.

MODE 2 (Elasticsearch) now uses Vector instead of FluentBit. See Vector Configuration for MODE 2.

Configure FluentBit to collect workflow events from Quarkus Flow applications and send them to PostgreSQL.

Container Runtime Detection

FluentBit must use the correct parser based on your Kubernetes cluster’s container runtime. Using the wrong parser causes FluentBit to fail silently - it will tail log files but won’t parse any events.

Check Your Cluster’s Runtime

# For KIND clusters
docker exec <cluster-name>-control-plane crictl version

# For real Kubernetes clusters
kubectl get nodes -o wide
# Look at CONTAINER-RUNTIME column

# Or quick test
kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.containerRuntimeVersion}'

Output examples:

containerd://1.7.2 → Use CRI parser
cri-o://1.25.0 → Use CRI parser
docker://20.10.21 → Use Docker parser

Parser Configuration

CRI Runtime (Most Common)

Used by: KIND, GKE, EKS (recent versions), AKS, most modern Kubernetes

Log format:

2026-04-23T23:07:15.123456789Z stdout F {"eventType":"io.serverlessworkflow.workflow.started.v1",...}
└────────timestamp──────────┘  └stream┘ └logtag┘ └──────────────message──────────────────────────┘

FluentBit configuration:

fluent-bit.conf:

[INPUT]
    Name              tail
    Path              /var/log/containers/*_workflows_*.log
    Parser            cri     # Use CRI parser
    Tag               kube.*

parsers.conf:

[PARSER]
    Name   cri
    Format regex
    Regex  ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ
    Time_Keep On

Docker Runtime (Legacy)

Used by: Older Kubernetes clusters, Docker Desktop Kubernetes

Log format:

{"log":"{\"eventType\":\"io.serverlessworkflow.workflow.started.v1\",...}\n","stream":"stdout","time":"2026-04-23T23:07:15.123456789Z"}

FluentBit configuration:

fluent-bit.conf:

[INPUT]
    Name              tail
    Path              /var/log/containers/*_workflows_*.log
    Parser            docker  # Use Docker parser
    Tag               kube.*

parsers.conf:

[PARSER]
    Name   docker
    Format json
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ
    Time_Keep On
    Decode_Field_As escaped log

Namespace Configuration

FluentBit tails logs from specific namespaces using the pattern:

/var/log/containers/*_${WORKFLOW_NAMESPACE}_*.log

Example log file:

/var/log/containers/workflow-test-app-7d8f9c6b5-abc123_workflows_workflow-app-xyz789.log
                     └──────pod-name──────────┘  └─namespace┘ └──container-name+id──┘

Single Namespace

Set in DaemonSet:

env:
- name: WORKFLOW_NAMESPACE
  value: "workflows"  # Change to match your deployment namespace

Multiple Namespaces

Option 1: Multiple Path entries

[INPUT]
    Name    tail
    Path    /var/log/containers/*_workflows_*.log
    Path    /var/log/containers/*_production_*.log
    Path    /var/log/containers/*_staging_*.log
    Parser  cri
    Tag     kube.*

Option 2: Wildcard + Filter (for dynamic namespaces)

[INPUT]
    Name    tail
    Path    /var/log/containers/*.log
    Parser  cri
    Tag     kube.*

[FILTER]
    Name    kubernetes
    Match   kube.*

[FILTER]
    Name    grep
    Match   kube.*
    Regex   kubernetes.namespace_name ^(workflows|production|staging)$

Use Option 1 if you know the namespaces (better performance). Use Option 2 for dynamic namespaces.

Nested JSON Parsing

After the CRI/Docker parser extracts the log field, you need a second parser for the JSON event:

[FILTER]
    Name              parser
    Match             kube.*
    Key_Name          log        # Parse the 'log' field
    Parser            json       # Use JSON parser
    Reserve_Data      On
    Preserve_Key      Off

Output Configuration

FluentBit output configuration for PostgreSQL MODE 1.

PostgreSQL Output

FluentBit uses Lua script (flatten-event.lua) to route events and PostgreSQL output plugin.

PostgreSQL output configuration:

[OUTPUT]
    Name         pgsql
    Match        workflow.*
    Host         ${POSTGRESQL_HOST}
    Port         ${POSTGRESQL_PORT}
    User         ${POSTGRESQL_USER}
    Password     ${POSTGRESQL_PASSWORD}
    Database     ${POSTGRESQL_DATABASE}
    Table        workflow_events_raw
    Timestamp_Key time

Lua script routing:

[FILTER]
    Name    lua
    Match   kube.*
    script  flatten-event.lua
    call    flatten_event

What the Lua script does:

Reads the eventType field from each JSON event
Routes workflow events (e.g., io.serverlessworkflow.workflow.started.v1) to tag workflow.*
Routes task events (e.g., io.serverlessworkflow.task.started.v1) to tag task.*
Flattens nested Kubernetes metadata for easier querying

See: data-index/scripts/fluentbit/postgresql/ for complete PostgreSQL mode configuration.

MODE 2 (Elasticsearch) uses Vector

Elasticsearch deployments (MODE 2) have migrated to Vector for log collection.

See Vector Configuration for MODE 2 configuration details.

Debugging FluentBit Output

Check output configuration:

# PostgreSQL mode (MODE 1)
kubectl get configmap -n logging workflows-fluent-bit-config -o yaml | grep -A 10 "\[OUTPUT\]"

PostgreSQL connection errors:

# Check FluentBit logs for PostgreSQL connection issues
kubectl logs -n logging -l app=workflows-fluent-bit | grep -i pgsql

# Common errors:
# - "connection refused" → PostgreSQL service not reachable
# - "authentication failed" → Incorrect credentials
# - "relation does not exist" → Table not created

Lua script debugging:

# Check FluentBit logs for Lua errors
kubectl logs -n logging -l app=workflows-fluent-bit | grep -i lua

# Common errors:
# - "attempt to index nil value" → Missing field in event JSON
# - "syntax error" → Invalid Lua syntax in script

After modifying FluentBit configuration, regenerate the ConfigMap using generate-configmap.sh and restart FluentBit pods.

Common Issues

Events Not Reaching Storage

Symptoms:

FluentBit logs show no errors
FluentBit is tailing log files (inotify_fs_add messages)
No events appear in storage
No processing activity in FluentBit logs

Diagnosis:

# Check FluentBit logs for parser errors
kubectl logs -n logging -l app=workflows-fluent-bit | grep -i "parser\|error"

# Expected error if using wrong parser:
# [error] [input:tail:tail.0] parser 'docker' is not registered

Solution:

Detect container runtime (see above)
Update fluent-bit.conf INPUT section with correct parser
Ensure matching parser exists in parsers.conf
Regenerate ConfigMap and restart FluentBit pods

Parser Not Registered

Error:

[error] [input:tail:tail.0] parser 'cri' is not registered

Solution:

Add the parser definition to parsers.conf:

[PARSER]
    Name   cri
    Format regex
    Regex  ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ
    Time_Keep On

Verification

Verify Parser is Working

# Restart FluentBit
kubectl delete pod -n logging -l app=workflows-fluent-bit

# Trigger a workflow
curl -X POST http://localhost:8082/test-workflows/simple-set \
  -H "Content-Type: application/json" \
  -d '{}'

# Check FluentBit is processing events
kubectl logs -n logging -l app=workflows-fluent-bit --tail=100 | grep -E "stdout|eventType"

Verify PostgreSQL Storage

# Check raw events table
kubectl exec -n postgresql postgresql-0 -- \
  env PGPASSWORD=dataindex123 psql -U dataindex -d dataindex -c \
  "SELECT COUNT(*) FROM workflow_events_raw;"

# Check normalized table
kubectl exec -n postgresql postgresql-0 -- \
  env PGPASSWORD=dataindex123 psql -U dataindex -d dataindex -c \
  "SELECT id, name, status FROM workflow_instances LIMIT 5;"

Expected: Count should increase after triggering workflows, and normalized table should contain workflow instances.

KIND-Specific Notes

KIND always uses containerd (CRI runtime), even though KIND itself runs in Docker.

For KIND clusters, always use the CRI parser.

Production Checklist

Identified container runtime (CRI vs Docker)
Configured correct parser in fluent-bit.conf INPUT section
Added matching parser definition to parsers.conf
Regenerated FluentBit ConfigMap
Tested with real workflow execution
Verified events reaching storage
Documented runtime in deployment notes

FluentBit Configuration (MODE 1 - PostgreSQL)

Container Runtime Detection

Check Your Cluster’s Runtime

Parser Configuration

CRI Runtime (Most Common)

Docker Runtime (Legacy)

Namespace Configuration

Single Namespace

Multiple Namespaces

Nested JSON Parsing

Output Configuration

PostgreSQL Output

Debugging FluentBit Output

Common Issues

Events Not Reaching Storage

Parser Not Registered

Verification

Verify Parser is Working

Verify PostgreSQL Storage

KIND-Specific Notes

Production Checklist

References