Vector Configuration

Data Index uses Vector as the log collector for both MODE 1 (PostgreSQL) and MODE 2 (Elasticsearch). Vector tails Quarkus Flow container logs, keeps only structured workflow/task events, and writes them to the mode’s raw storage. Normalization then happens in the storage layer (PostgreSQL triggers or Elasticsearch Transforms).

Sources of truth

Location Purpose

data-index/collectors/vector/mode1-postgresql/vector.yaml

MODE 1 config — postgres sinks. Consumed by logic-operator (Go module) and validated by the collectors Maven module.

data-index/collectors/vector/mode2-elasticsearch/vector.yaml

MODE 2 config — elasticsearch sinks.

data-index/helm/data-index/configs/vector/vector-mode{1,2}-*.yaml

Symlinks to the collector configs (rendered into a ConfigMap via .Files.Get, which Helm resolves transparently). Editing a collector config is automatically reflected here - nothing to copy or sync. make check-vector-configs guards that the symlinks exist and still point at the right source.

data-index/collectors/examples/mode{1,2}-*/daemonset.yaml

Reference DaemonSet manifests (RBAC, volume mounts, env). Not embedded by the operator.

Vector version is pinned in data-index/collectors/pom.xml (<vector.version>) and data-index/helm/data-index/values.yaml (vector.image.tag); Renovate keeps them in sync.

Pipeline

Quarkus Flow pod → stdout → /var/log/containers/*.log
        ↓  kubernetes_logs source (namespace-filtered)
   parse_json          → drop non-JSON lines, stash event under .flow_event
   filter_workflow_events → keep eventType starting with "io.serverlessworkflow."
   route_by_type       → workflow_events | task_events
        ↓
   MODE 1: build_{workflow,task}_row → postgres sinks → workflow_events_raw / task_events_raw
   MODE 2: (enrich @timestamp)       → elasticsearch sinks → workflow-events-* / task-events-*

MODE 1 (PostgreSQL)

The postgres sink requires Vector >= 0.46.0. Because it cannot template the table name per event, MODE 1 uses one sink per raw table.

transforms:
  build_workflow_row:
    type: remap
    inputs: [route_by_type.workflow_events]
    source: |
      . = { "tag": .flow_event.eventType, "time": now(), "data": .flow_event }

sinks:
  postgres_workflow:
    type: postgres
    inputs: [build_workflow_row]
    endpoint: "postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}?options=-c%20search_path%3D${POSTGRES_SCHEMA}"
    table: workflow_events_raw
    batch: { max_events: 100, timeout_secs: 1 }
  postgres_task:
    type: postgres
    inputs: [build_task_row]
    endpoint: "postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}?options=-c%20search_path%3D${POSTGRES_SCHEMA}"
    table: task_events_raw
    batch: { max_events: 100, timeout_secs: 1 }

Schema Configuration

The endpoint URL includes ?options=-c%20search_path%3D${POSTGRES_SCHEMA}, which sets PostgreSQL’s search_path connection parameter. This ensures Vector resolves table names in the specified schema (default: public).

The PostgreSQL schema must exist before Vector connects. When using a custom schema:

  1. Create the schema: CREATE SCHEMA IF NOT EXISTS <schema-name>;

  2. Ensure Flyway migrations run against the same schema (via JDBC currentSchema parameter)

  3. Set the same schema in both Vector and DataIndex configurations

The Helm chart does not automatically create the schema. Either: - Pre-create the schema before deploying (recommended for non-Helm deployments) - Use Helm hooks or init containers to create it during deployment

The sinks use the default in-memory buffer (parity with the FluentBit MODE 1 setup). Do not set buffer.type: disk on the postgres sink: in Vector 0.54 it is not drained until graceful shutdown, stranding low-volume events. The kubernetes_logs checkpoint plus idempotent triggers cover a restart.

Row shape maps directly to the raw table columns (tag TEXT, time TIMESTAMPTZ, data JSONB):

  • tag — informational only (indexed); the triggers never read it.

  • time — ingestion time; feeds only the created_at/updated_at audit columns.

  • data — the untouched Quarkus Flow event. normalize_workflow_event() / normalize_task_event() extract instanceId, status, startTime/endTime (epoch seconds), input/output, error, and taskPosition from this JSONB.

A Vector batch is one multi-row INSERT. If a row makes a trigger raise an uncaught exception the whole batch fails and is retried. The workflow trigger is INSERT …​ ON CONFLICT and the task trigger catches foreign_key_violation, so this is tolerable. If a poison-pill row appears, set batch.max_events: 1 for strict row-by-row parity with FluentBit.

Required environment (set on the DaemonSet): WORKFLOW_NAMESPACE, NODE_NAME, POSTGRES_HOST, POSTGRES_PORT, POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD, optional DEBUG_EVENTS.

MODE 2 (Elasticsearch)

See Elasticsearch Production. The elasticsearch sinks write date-rolled raw indices; ES Transforms aggregate them into workflow-instances / task-executions.

Required environment: WORKFLOW_NAMESPACE, NODE_NAME, ELASTICSEARCH_HOST, ELASTICSEARCH_PORT, optional DEBUG_EVENTS.

Deployment (Helm)

Vector is deployed by the Data Index Helm chart. It is enabled by default for MODE 1 and MODE 2:

# MODE 1
helm upgrade --install data-index data-index/helm/data-index \
  -f data-index/helm/data-index/values-mode1.yaml

# MODE 2
helm upgrade --install data-index data-index/helm/data-index \
  -f data-index/helm/data-index/values-mode2.yaml

vector.config (mode1-postgresql | mode2-elasticsearch) selects configs/vector/vector-<config>.yaml for the ConfigMap and the DaemonSet’s storage environment variables.

Troubleshooting

# Vector pods
kubectl get pods -n logging -l app=vector
kubectl logs -n logging -l app=vector --tail=100

# Trace every workflow/task event to stdout (high volume - troubleshooting only)
kubectl set env daemonset/vector -n logging DEBUG_EVENTS=true
kubectl logs -n logging -l app=vector -f
kubectl set env daemonset/vector -n logging DEBUG_EVENTS-

# Prometheus metrics (port 9598)
kubectl port-forward -n logging ds/vector 9598:9598
curl -s localhost:9598/metrics | grep vector_component_sent_events_total

Common metrics: vector_component_received_events_total (from Kubernetes logs), vector_component_sent_events_total (to the sink), vector_component_errors_total, vector_sink_retries_total.

Validation

# Runs `vector validate` against both configs in a real Vector container,
# plus the Helm-copy identity check.
mvn -pl data-index/collectors verify

See also