Data Index Documentation

What is Data Index?

Data Index is a read-only query service for Serverless Workflow 1.0.0 runtime execution data. It provides a GraphQL API for querying workflow instances and task executions from Quarkus Flow applications.

Key Features:

  • 📊 GraphQL API - Query workflow instances and task executions

  • 🚀 Real-time Processing - PostgreSQL: <1ms triggers, Elasticsearch: ~1s transforms

  • 🔍 Flexible Filtering - Filter by status, name, namespace, time ranges

  • 🎯 Production Ready - Both PostgreSQL and Elasticsearch backends fully tested

  • 📝 Structured Logging - Captures events from Quarkus Flow apps via FluentBit

  • 🔄 Multiple Storage Backends - Choose PostgreSQL (MODE 1) or Elasticsearch (MODE 2)

How It Works

Data Index captures workflow execution events and makes them queryable via GraphQL:

Quarkus Flow App
    ↓ (structured logging to stdout)
FluentBit DaemonSet
    ↓ (tail container logs)
Storage Backend (PostgreSQL or Elasticsearch)
    ↓ (normalization)
GraphQL API

Storage Backends

Data Index supports two production-ready storage backends:

Backend Best For Status

PostgreSQL (MODE 1)

Most deployments - ACID transactions, simple operations, <50K workflows/day

✅ Production Ready

Elasticsearch (MODE 2)

High throughput (100K+ workflows/day), full-text search, advanced analytics

✅ Production Ready

Both backends provide:

  • Same GraphQL API (backend-agnostic queries)

  • Same data model (WorkflowInstance, TaskExecution)

  • Production-grade reliability and performance

Choose based on your requirements - see Decision Matrix for detailed comparison.

Quick Links

What Data Index Does NOT Do

Data Index is a read-only query service. It does NOT:

  • Execute workflows (that’s Quarkus Flow’s job)

  • Modify workflow state

  • Provide workflow management operations (start/stop/retry)

System Requirements

Core:

  • Kubernetes cluster (or KIND for local development)

  • FluentBit 2.0+

  • Quarkus Flow applications with structured logging enabled

Storage backend (choose one):

  • PostgreSQL 13+ (MODE 1)

  • Elasticsearch 7.10+ (MODE 2)

Next Steps

Ready to get started? Check out the Getting Started Guide.

Already have Data Index running? Learn how to structure and deploy your Quarkus Flow applications.