Data Engineering Weekly #247

If your team is relying on AI-driven insights, this upcoming webinar will show you how to make those insights more dependable, transparent, and explainable. In this 12/9 Deep Dive with our friends at Cube, you'll learn:

- Why AI analytics fails without governance (and what that actually means)

- How semantic layers provide the guardrails AI needs to be trustworthy

- Technical implementation: how Compass + Cube work together to prevent hallucinations

- Live demo: governed self-service analytics that data teams can actually trust

Save your spot now

DoorDash highlights the challenge of extracting reliable insights from fragmented knowledge systems and the limitations of single agents constrained by context, determinism, and long-horizon reasoning. The article details an evolutionary architecture that progresses from deterministic workflows to adaptive agents, hierarchical deep-agent systems with shared memory, and exploratory swarm-based A2A collaboration, all built on a unified platform featuring hybrid search, schema-aware SQL generation, multi-stage validation, and integrated guardrails.

https://careersatdoordash.com/blog/beyond-single-agents-doordash-building-collaborative-ai-ecosystem/

LinkedIn writes about the challenge of scaling Venice ingestion to support massive bulk loads, hybrid Lambda-style stores, partial updates, and active/active replication while avoiding bottlenecks in producing, consuming, persisting, and compaction. The article details the end-to-end evolution of the ingestion pipeline, including partition scaling, shared consumer and writer pools, SST-based ingestion, RocksDB tuning with leveled compaction, BlobDB, Fast-Avro adoption, parallelized DCR processing, and adaptive throttling for deterministic latency.

https://www.linkedin.com/blog/engineering/infrastructure/evolution-of-the-venice-ingestion-pipeline

Dropbox writes about the transition from a traditional RAG search system to an agentic AI that must reason, plan, and act without