Velocity Analytics

The existing analytics solution could not handle the volume of events generated by a growing user base. Queries that should take milliseconds were timing out at 30+ seconds, dashboards were unreliable, and the team had no visibility into real-time user behavior during launches.

I architected a new analytics pipeline from ingestion to visualization. Events flow through a Kafka-based streaming pipeline into ClickHouse, a columnar database optimized for analytical queries. The API layer uses Go for maximum throughput, with a React dashboard that supports real-time streaming updates via Server-Sent Events.

The system consists of three layers: an ingestion tier (Go HTTP collectors writing to Kafka), a processing tier (Kafka Streams for enrichment, sessionization, and aggregation into ClickHouse), and a serving tier (Go API with materialized views for common queries and a React dashboard with WebSocket streaming). Infrastructure is managed with Terraform across multiple availability zones.

The primary challenge was maintaining query performance as data volume grew to billions of rows. I implemented a tiered storage strategy with hot/warm/cold partitions, combined with aggressive materialized views for dashboard queries. Another challenge was ensuring exactly-once semantics in the streaming pipeline, which I solved using Kafka transactions with idempotent writes to ClickHouse.

Query response times dropped from 30+ seconds to under 200ms for 95th percentile queries. The system now processes 50M+ events daily with 99.99% uptime. Real-time dashboards became the primary tool for product launches, enabling data-driven decisions within minutes instead of days.