Cartrack
Two-plus years building Cartrack's data platform from the ground up — migrating terabyte-scale data lakes off Spark, Trino, and Iceberg onto a lean, high-throughput ClickHouse cluster on Kubernetes. The work started in data engineering and grew into fully agentic AI systems that now write, test, and ship their own code.
Senior Data/AI Engineer
Oct 2023 — PresentLeading the data platform transformation from Spark + Iceberg + Trino to ClickHouse on Kubernetes. Simplified the data stack to be as lean and stable as possible. Designed internal ClickHouse table transformations, implemented custom User-Defined Functions with clickhouse-udf-rs, integrated DuckDB for fast ad-hoc processing, and built AI Agents utilizing LangGraph, LlamaIndex, Qdrant, Next.js, and TanStack.
Resolving Data Lake bottlenecks
Began by addressing the limitations of Spark, Trino, and Iceberg. Tuned Spark parameters and optimized metadata footprint by cleaning up Iceberg snapshots to improve query performance.
On-premise Kubernetes (RKE) cluster
Wired up a Rancher Kubernetes Engine (RKE) cluster from scratch using on-prem VM machines, deploying namespaces, resource limits, and ingress policies as a team of two. Learned to resolve complex infrastructure and networking issues independently without cloud services.
ClickHouse migration & simplification
Deprecated legacy storage systems and successfully migrated all historical data to a very large, high-throughput 3-node ClickHouse cluster. Replaced complex pipelines with internal ClickHouse transformations, drastically simplifying the data stack for maximum stability.
Rust UDFs & DuckDB integrations
Engineered high-performance data operations by writing custom User-Defined Functions in Rust (clickhouse-udf-rs) and utilizing DuckDB for efficient file formats and local analysis.
Database tuning & performance
Tuned tables, engines, and columns. Optimized compression types and codec settings on each column to yield the maximum performance and query speed.
Observability & chmonitoring
Built chmonitoring and developed multiple custom internal tools to automate database operations, query analysis, and performance monitoring.
First LLM workflows
Created the initial Text-to-SQL workflows and response synthesis engines leveraging LlamaIndex and custom LLM integrations.
LangGraph agentic systems
Evolved the LLM systems into multi-agent workflows using LangGraph in fast/agent modes. Equipped agents with skills, tool calling, Firecrawl, Qdrant vector database, and Cube.js.
Automated AI agents working 24/7
Moved from scary vibe-coding to real agentic engineering — coding agents now ship most of the work without me touching the codebase. Every change runs through 3,000–5,000 unit and end-to-end tests before it reaches production, and a sandboxed agent listens for runtime exceptions and patches them on its own.