Log Aggregation Strategies

How do you implement centralized logging in a distributed system? What are the key components?

Monitoringintermediate

// interview question

How do you implement centralized logging in a distributed system? What are the key components?

Sample answer

Centralized logging collects logs from all services into one searchable system. Key components: 1) Collection - agents like Fluentd, Fluent Bit, or Filebeat. 2) Transport - message queues (Kafka) for buffering. 3) Processing - parsing, filtering, enriching (Logstash). 4) Storage - Elasticsearch, Loki, or cloud services. 5) Visualization - Kibana, Grafana. Best practices: use structured logging (JSON), include correlation IDs for tracing requests, set retention policies, and implement log levels appropriately.

Why this matters

In distributed systems, logs scattered across hundreds of containers are useless. Centralized logging enables searching across all services, correlating events, and debugging issues. The ELK stack (Elasticsearch, Logstash, Kibana) is traditional; newer options like Loki (Grafana) are more cost-effective. Structured logging is crucial - parsing unstructured text at scale is expensive.

Code examples

Fluent Bit DaemonSet

yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:latest
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: config
          mountPath: /fluent-bit/etc/
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: config
        configMap:
          name: fluent-bit-config

Structured log format

json

{
  "timestamp": "2024-01-15T10:30:00.000Z",
  "level": "error",
  "service": "payment-api",
  "trace_id": "abc123xyz",
  "span_id": "def456",
  "user_id": "user_789",
  "message": "Payment processing failed",
  "error": {
    "type": "PaymentGatewayError",
    "message": "Connection timeout",
    "stack": "..."
  },
  "duration_ms": 5023
}

Common mistakes to avoid

Logging sensitive data (passwords, PII) that violates compliance
Using DEBUG level in production, creating massive storage costs
Not including correlation IDs, making distributed tracing impossible

Likely follow-ups

How do you handle high-volume logging without impacting application performance?
What is the difference between logs, metrics, and traces?
How do you implement log retention and comply with data regulations?

Answer out loud first, then check yourself against the model answer.

Practice all Mid-Level questions More Monitoring questions

#logging#observability#elk#fluentd#monitoring

Also worth your time on this topic

Checklist

Monitoring & Observability Checklist

Comprehensive checklist for implementing monitoring, logging, tracing, and alerting across your infrastructure and applications.

60-90 minutes

Article

Your Logs Cost More Than the Service That Emits Them

Log ingestion is priced per gigabyte and indexing is priced per event, which means a service emitting small lines pays far more than its byte count suggests. Here is the arithmetic on a real server, why the usual advice to "log less" misses, and the five levers that actually move the bill.

Interview

Four Golden Signals of Monitoring

What are the four golden signals of monitoring and why are they important?

junior

Log Aggregation Strategies

More Monitoring interview questions

Also worth your time on this topic

Monitoring & Observability Checklist

Your Logs Cost More Than the Service That Emits Them

Four Golden Signals of Monitoring