Real-Time Analytics: Trade-offs and Best Practices

Part of the Distributed Systems Guide: This post explores architectural patterns for real-time analytics. For distributed database fundamentals, start with Getting Started with Distributed Databases.

Building real-time analytics systems requires balancing speed, accuracy, and cost. Here's what I've learned from building and supporting real-time data pipelines at scale.

What is "Real-Time"?

Let's define our terms:

• True Real-Time: < 100ms latency (streaming analytics)
• Near Real-Time: 1-5 seconds (micro-batching)
• Fast Batch: 5-60 seconds (mini-batches)

Most "real-time" systems are actually near real-time, which is perfectly fine for 90% of use cases.

Architecture Patterns

Pattern 1: Lambda Architecture

Separate batch and stream processing:

textCopy
Events → Kafka → Stream Processor → Real-time View
       ↓
       → Batch Processor → Batch View
                         ↓
                    Serving Layer (merge views)

Pros:

• Fault-tolerant (batch layer recomputes if stream fails)
• Accurate (batch layer ensures correctness)

Cons:

• Complex (maintain two codebases)
• Expensive (duplicate processing)

Pattern 2: Kappa Architecture

Stream-only processing:

textCopy
Events → Kafka → Stream Processor → Materialized View
                       ↓
                 Replayable for corrections

Pros:

• Simpler (one codebase)
• Lower latency
• Easier to reason about

Cons:

• Harder to correct historical data
• Requires replayable event log

Data Ingestion

High-Throughput Ingestion

pythonCopy
# Bad: Individual inserts
for event in events:
    db.execute("INSERT INTO events VALUES (?)", event)

# Good: Batch inserts
db.execute_many("INSERT INTO events VALUES (?)", events)

# Better: Bulk load for large batches
db.bulk_load("events", events_file)

Schema Evolution

Plan for schema changes from day one:

jsonCopy
{
  "schema_version": "v2",
  "event_id": "abc123",
  "user_id": 12345,
  "properties": {
    "action": "click",
    "target": "buy_button"
  },
  "metadata": {
    "timestamp": "2026-01-07T10:00:00Z",
    "source": "web_app"
  }
}

Query Optimization

Pre-Aggregation

Don't aggregate at query time:

sqlCopy
-- Slow: Aggregate 1B events on every query
SELECT DATE(timestamp) as day, COUNT(*) as events
FROM raw_events
WHERE user_id = 12345
GROUP BY DATE(timestamp);

-- Fast: Pre-aggregate into rollup table
SELECT day, events
FROM daily_user_events
WHERE user_id = 12345;

Columnar Storage

Use columnar databases for analytics:

sqlCopy
-- Row-oriented: Reads entire rows (slow for aggregations)
-- Columnar: Reads only needed columns (fast for aggregations)

SELECT AVG(revenue), SUM(quantity)
FROM sales
WHERE product_id = 'ABC123';

-- Columnar DB reads only: revenue, quantity, product_id columns
-- Row DB reads: ALL columns

Trade-offs

1. Freshness vs. Accuracy

Problem: Real-time data might be incomplete due to:

• Late-arriving events
• Network delays
• Processing failures

Solution: Use watermarks and restatements:

pythonCopy
# Process events with 5-minute watermark
watermark = current_time - timedelta(minutes=5)

# Recompute recent windows when late data arrives
if event.timestamp > watermark:
    recompute_window(event.timestamp)

2. Latency vs. Cost

Lower latency = higher cost:

Latency	Approach	Cost (relative)
< 100ms	In-memory stream	10x
1-5s	Micro-batching	3x
30-60s	Mini-batches	1x

3. Consistency vs. Availability

CAP theorem strikes again:

pythonCopy
# Strong consistency (slower)
result = db.query("SELECT COUNT(*) FROM events", consistency="strong")

# Eventual consistency (faster)
result = db.query("SELECT COUNT(*) FROM events", consistency="eventual")

Practical Example: E-Commerce Dashboard

Requirements:

• Display orders/minute in real-time
• Show top products by revenue (last hour)
• Track conversion funnel

Solution:

typescriptCopy
// 1. Ingest events into Kafka
interface OrderEvent {
  order_id: string;
  user_id: string;
  product_id: string;
  amount: number;
  timestamp: Date;
}

// 2. Stream processing (Flink/Spark Streaming)
const ordersPerMinute = eventStream
  .window(Tumbling.size(Time.minutes(1)))
  .aggregate(new Count())
  .store("orders_per_minute");

// 3. Materialized view for fast queries
const topProducts = eventStream
  .window(Sliding.size(Time.hours(1), Time.minutes(5)))
  .groupBy('product_id')
  .sum('amount')
  .top(10)
  .store("top_products_hourly");

// 4. Query serving layer
app.get('/dashboard/metrics', async (req, res) => {
  const metrics = {
    ordersPerMinute: await redis.get('orders_per_minute'),
    topProducts: await redis.get('top_products_hourly'),
    conversionRate: await db.query('SELECT * FROM conversion_funnel')
  };
  res.json(metrics);
});

Common Pitfalls

❌ Over-Engineering

Don't build a Kafka cluster if daily batch jobs suffice.

❌ Ignoring Data Quality

Garbage in, garbage out. Validate at ingestion:

pythonCopy
def validate_event(event):
    assert event.timestamp < datetime.now()
    assert event.amount >= 0
    assert event.user_id is not None

❌ No Backpressure Handling

System crashes when ingestion > processing capacity:

pythonCopy
# Implement backpressure
if queue_depth > threshold:
    slow_down_producers()
    alert_operators()

Monitoring

Essential metrics:

pythonCopy
metrics = {
    'ingestion_rate': 'events/second',
    'processing_lag': 'seconds behind real-time',
    'error_rate': 'failed events/total events',
    'query_latency_p99': 'milliseconds'
}

Conclusion

Real-time analytics requires:

1. Clear latency requirements (don't over-optimize)
2. Right architecture (Lambda vs Kappa)
3. Pre-aggregation strategy
4. Monitoring and alerting
5. Plan for failure and late data

Start simple, measure, and scale incrementally.

Related Resources

• Getting Started with Distributed Databases - Foundational concepts for distributed systems
• How SingleStore Handles Real-Time Analytics at Scale - Concrete HTAP implementation example
• Why HTAP Systems Fail Quietly (And How to Notice Early) - Production failure modes

---

Building a real-time system? I'd love to hear about your challenges!