Artie

For a long time, I thought Iceberg had no real value. Save cost? Not really. Not once you factor in the engineering overhead and expertise required to run it. Simpler? Definitely not. If anything, it adds more moving pieces and more more failure modes when things go wrong. So for a while, I didn’t get the hype. Then over the past year, we started seeing teams use it in ways that actually matter. Not for cost savings. But to simplify architecture. A few patterns kept showing up: • Multi-engine without duplication Instead of replicating data into both Snowflake and Databricks, teams replicate once into Iceberg and query it from both. • Decoupling ingestion from the query layer Iceberg becomes the source of truth, so pipelines don’t have to change every time you switch warehouses. • Unifying streaming + batch CDC and batch jobs both land into the same place, instead of maintaining separate “real-time” and “analytics” systems. That’s when it clicked: Iceberg isn’t a cost optimization tool. It’s an organizational scaling tool. If you’re a single-engine team with simple pipelines, you probably don’t need it. But once your data stack starts fragmenting across teams, tools, and use cases… the value becomes very real. Iceberg solves a very real problem - just not the one most people think.

The best data conversations I've had this year didn't happen at conferences. They happened around a dinner table with 10 people and no agenda. No slides. No pitches. Just data and engineering leaders talking honestly about what's working, what's broken, and what they're betting on next. That's why we keep hosting these. Our next Data Executives Dinner is April 16 in San Francisco - private dining room at the Four Seasons. Matthew Powers (CTO @ momoGood + Tatango) will be sharing how he rolled out data infra changes and pushed AI extensively across his org. Director of Data at ClickUp will also be at the table. 10-12 people. Always small. Always off the record. If you're a data or engineering leader in the Bay Area and this sounds like your kind of evening, DM me and I'll send you the details. And if you know someone who'd be great at this table, send them my way.

⭐ 𝗗𝗮𝘁𝗮 & 𝗘𝗻𝗴 𝗜𝗻𝗻𝗼𝘃𝗮𝘁𝗼𝗿𝘀 𝗦𝗽𝗼𝘁𝗹𝗶𝗴𝗵𝘁: 𝗩𝗲𝗱 𝗣𝗿𝗮𝗸𝗮𝘀𝗵, 𝗣𝗿𝗶𝗻𝗰𝗶𝗽𝗮𝗹 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿 𝗮𝘁 𝗚𝗶𝘁𝗹𝗮𝗯 ⭐ We’re excited to highlight ved prakash Principal Data Engineer at GitLab and one of 17 data leaders globally on Snowflake ’s Data Superhero Council. Ved is redefining how enterprises operationalize real-time data and govern AI at scale. At GitLab, Ved built Project Siphon, an enterprise-scale CDC backbone delivering real-time streaming, sub-hour latency, and disciplined cost control. He shares lessons through talks (Berlin Buzzwords, DSC Europe) and DataAI Chronicles on next-gen real-time and AI-integrated data platforms. 𝗛𝗼𝘄 𝗵𝗮𝘀 𝘁𝗵𝗲 𝗿𝗼𝗹𝗲 𝗼𝗳 𝗮 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗹𝗲𝗮𝗱𝗲𝗿 𝗲𝘃��𝗹𝘃𝗲𝗱 𝘀𝗶𝗻𝗰𝗲 𝘆𝗼𝘂 𝘀𝘁𝗮𝗿𝘁𝗲𝗱 𝘆𝗼𝘂𝗿 𝗰𝗮𝗿𝗲𝗲𝗿? Early on, leadership meant keeping pipelines running and reports accurate across ETL, queries, and infrastructure. Today it’s strategic: data leaders make multi-million dollar build vs buy decisions, shape company-wide tech direction, and tie architecture to ROI and cost efficiency. The biggest shift is owning platform economics, from Snowflake spend and AI governance to real-time systems that change how the business operates. 𝗪𝗵𝗮𝘁’𝘀 𝗮 𝗿𝗲𝗰𝗲𝗻𝘁 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗮𝗹 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝘆𝗼𝘂’𝗿𝗲 𝗽𝗿𝗼𝘂𝗱 𝗼𝗳 — 𝗮𝗻𝗱 𝘄𝗵𝘆? I’m proud of the AI Cost Control Framework we built for Snowflake Cortex at GitLab as Cortex credit usage scaled without visibility. It monitors consumption in real time, applies configurable thresholds, alerts before budgets are breached, and attributes spend to specific teams and services. As AI embeds into data platforms, proactive cost governance becomes table stakes for scaling innovation with financial discipline. 𝗪𝗵𝗮𝘁’𝘀 𝘁𝗵𝗲 𝗯𝗶𝗴𝗴𝗲𝘀𝘁 𝗺𝗶𝘀𝗰𝗼𝗻𝗰𝗲𝗽𝘁𝗶𝗼𝗻 𝘆𝗼𝘂 𝘀𝗲𝗲 𝗮𝗯𝗼𝘂𝘁 𝗖𝗗𝗖 𝗼𝗿 𝘀𝘁𝗿𝗲𝗮𝗺𝗶𝗻𝗴 𝘀𝘆𝘀𝘁𝗲𝗺𝘀? CDC isn’t “just faster data.” It’s an architectural transformation involving data movement, consistency guarantees, schema evolution, and how deletes, updates, and backfills coexist with streaming changes. Downstream systems, from dbt models to BI tools to ML workloads, also need to adapt to incremental semantics. CDC requires monitoring, alerting, deep database knowledge, and tight coordination across data, platform, and application teams. 𝗪𝗵𝗲𝗿𝗲 𝗱𝗼 𝘆𝗼𝘂 𝘁𝗵𝗶𝗻𝗸 𝗔𝗜 𝘄𝗶𝗹𝗹 𝗵𝗮𝘃𝗲 𝘁𝗵𝗲 𝗯𝗶𝗴𝗴𝗲𝘀𝘁 𝗶𝗺𝗽𝗮𝗰𝘁 𝗼𝗻 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗼𝘃𝗲𝗿 𝘁𝗵𝗲 𝗻𝗲𝘅𝘁 𝗳𝗶𝘃𝗲 𝘆𝗲𝗮𝗿𝘀? 1. Intelligent orchestration and self-healing pipelines (fewer 2am pages) 2. Code generation and platform acceleration (less boilerplate, more time for architecture) 3. Intelligent cost optimization (dynamic optimization for Snowflake/Databricks, especially where streaming and CDC costs can spike) At Artie, we’re excited to feature leaders like Ved, who are pushing the boundaries of what modern data engineering looks like.

We're sunsetting real-time replication. After talking to hundreds of data teams, we've come to a difficult conclusion. Batch is good enough. The latency doesn't matter. The freshness doesn't matter. Your dashboards being 6 hours behind is fine. Nobody's making decisions that fast anyway. And AI in production? It doesn't need real-time data either. Your models will perform just fine on yesterday's data. Stale context never hurt anyone. We'll be migrating all customers to a daily sync by end of Q2. If you need fresher data, we recommend refreshing your browser. Happy April Fools. Real-time is the future. Your AI models deserve live data. And we're not going anywhere.

A pipeline that fails is annoying. A pipeline you can’t debug is a nightmare. Production pipelines aren’t just about pipelines working. They’re about having deep visibility when things inevitably go wrong. But a lot of teams still evaluate data pipeline tools based on connectors and latency. The real test shows up later: “What do you see when something breaks?” During a demo someone described their troubleshooting process: Is the connector on? Is the connector off? That was about the level of observability available. The moment data becomes operationally important, that’s not enough. You need to see things like: • replication lag • flush behavior • table-level throughput • replication slot health • where the pipeline is actually stuck Because when a pipeline stops moving, the worst possible situation is this: You know it’s broken. But you have no idea why. Production pipelines shouldn't be judged on the days everything works. They should be judged on how quickly you can understand them when they don’t.

Operating Apache Iceberg in production? We’ll be at 𝗕𝗼𝗼𝘁𝗵 𝗕𝟵 at Iceberg Summit. Stop by to talk real-world Lakehouse operations: catalogs, compaction, schema evolution, interoperability. Right after: Come by for Iceberg on the Rocks, a happy hour where conversations keep flowing 👉️ https://luma.com/a6xq8l6c #IcebergSummit2026 #ApacheIceberg™ #OpenData

They say that good artists copy, great artists steal. So I stole directly from Marty Kausas. Almost word for word. I met Marty over dinner, and he hammered in the idea: your LinkedIn header is your billboard. So, we're officially building the AWS DMS killer. No explanation needed if you've ever used AWS DMS. And that's the point. Thank you Marty 🙏