Ben Sullins

The Databricks Startups team answers founders’ most common AI questions—where to start without hype, how to stay compliant with sensitive data, what to build versus buy, how to cut hallucinations, and how to keep costs in check. 💡 Get step‑by‑step starting points, governance patterns for sensitive data, a pragmatic build‑versus‑buy lens, tactics to curb hallucinations, and cost controls that scale. 🔗 Full post: https://lnkd.in/ehRa3Yp5 #Databricks #DatabricksForStartups #AI #GenAI #MLOps

63

If your datasets are poorly labeled, you’re inviting a host of issues like noise, bias, and inconsistencies that can seriously hurt your model's accuracy and reliability when it’s out in the wild. That's why data labeling has evolved from being just a preliminary step into a vital part of engineering and operations in the AI lifecycle.

7

Databricks didn’t just release another AI feature. They made Custom Agents production-ready. That’s a big deal. Because most organizations aren’t struggling with ideas. They’re struggling with operationalization. The real question isn’t whether AI agents are powerful. It’s whether your data foundation is ready for them. Are you already building on Databricks, or still experimenting outside your core platform? #Databricks #AI #DataEngineering #Lakehouse #EnterpriseAI

4

🌍 Spatial data is more than a spatial join. You need complete spatial support. And now Databricks users can get that with Wherobots. Databricks has made great strides with Spatial SQL, but many of the hardest problems in spatial still remain: Working seamlessly with raster and vector at scale Balancing query cost and performance when data volumes explode Building actionable intelligence, not just visualizations Running computer vision and ML directly on imagery and sensor data That’s where Wherobots comes in. By integrating directly with Unity Catalog, Wherobots gives Databricks users the ability to: ✔️ Work with local to planetary scale datasets without friction ✔️ Use 300+ functions across SQL, Python, and Scala to transform data into intelligence ✔️ Enrich analytics with computer vision ML and spatial statistics ✔️ Keep governance with Iceberg or Delta, while lowering costs and boosting performance Customers like Leaf Agriculture, Dotlas, and Overture Maps Foundation are already showing what’s possible when spatial intelligence is a first-class citizen in the lakehouse. The physical world doesn’t fit neatly into rows and keys. But with the right tools, it can fit directly into your workflows. Comment DELTA and I will send you more info. 🌎 I'm Matt and I talk about modern GIS, earth observation, AI, and how geospatial is changing. 📬 Want more like this? Join 8k+ others learning from my newsletter → forrest.nyc

77

It would've taken 25 years for this Snowflake customer to get 100% business data quality coverage. With Anomalo, Discover now configures unsupervised data quality monitoring within 10-15 minutes for each dataset. Learn how Discover’s data team flipped the script using Anomalo on Snowflake to scale trust across petabyte-scale data: https://lnkd.in/gPjxDCFH

11

Excited to release my conversation with Alan Tu, CFA of WCM Investment Management on Databricks. Prior to our recording, I had a very surface-level understanding Databricks. So a major thank you to Alan for going into the weeds throughout this episode. We cover:  - How the Databricks platform can unlock value for customers and their data - Databricks' role in the AI ecosystem  - WCM's partnership formed around a pivotal point in its history Find it in your podcast feeds now + a link to the episode and transcript in comments.

59

4 Comments

RAG Systems Are Evolving And They Are Becoming Agentic. Most Retrieval Augmented Generation (RAG) systems work well until they do not. Once you introduce multiple design dimensions like retrieval strategy, chunking, prompts, agents, workflows, and evaluation metrics, the system becomes too complex to tune by hand. That is why I found Fareed Khan’s recent article on self improving agentic RAG systems so compelling. The core insight: Do not build a static pipeline. Build a system that can analyze its own weaknesses and evolve in response to real world performance. Here is the model: 1. Specialist agents Instead of a single LLM call, use a team of role based agents such as a retriever, generator, evaluator, diagnostician, and architect. Each focuses on one part of the workflow. 2. Multi dimensional evaluation Responses are scored across accuracy, feasibility, compliance, clarity, cost, and more. This produces a performance vector instead of a single metric. 3. Root cause diagnosis A diagnostician agent identifies what is going wrong, such as retrieval issues, hallucinations, missing context, or prompt failures. 4. SOP evolution An architect agent proposes variations to the system SOPs. The system tests them, compares performance, and moves toward the Pareto optimal configurations. 5. Continuous improvement loop As production data changes, the system adapts instead of drifting. Why this matters: In complex domains like healthcare, legal, enterprise knowledge, and customer support, traditional RAG systems break down under real world variability. Agentic RAG can lead to: More accurate results More resilient pipelines Continuous learning without manual re tuning Scalable knowledge workflows The future of RAG is not retrieval plus generation. It is retrieval plus generation plus agents plus evaluation plus evolution. If you are building AI products or internal knowledge tools, this shift is worth paying attention to. #AI #RAG #AgenticAI #GenerativeAI #MachineLearning #EnterpriseAI #AIDevelopment #LLM #ArtificialIntelligence #FutureOfWork #TechInnovation #KnowledgeManagement #HealthcareAI #AIEcosystem #AITools

7

Data gets messy long before it gets valuable. That’s why platforms like Databricks matter. Databricks brings data engineering, analytics, and machine learning into a single workspace built on Apache Spark. No juggling tools. No fragile pipelines. Just one place to ingest data, transform it, analyze it, and train models at scale. The Lakehouse approach is the real shift here. You get the flexibility of data lakes with the reliability of data warehouses, powered by Delta Lake. That means ACID transactions, schema enforcement, and the ability to trust your data as it grows. For data teams, this translates to faster iteration, fewer handoffs, and less time spent managing infrastructure. If you’re working with large, complex data and still stitching together too many systems, Databricks is worth understanding. #Databricks #BigData #DataEngineering #Analytics #MachineLearning #Lakehouse

41

🚀 Looking to build a high-performing data team with Databricks? Not all Data Engineers are created equal—and when it comes to Databricks, there are five essential skills that separate the good from the great. Whether you're hiring or upskilling, this guide from Revolent breaks down what to look for in top-tier Databricks talent. 👉 Discover the must-have skills: https://lnkd.in/eVe3UNFy #Databricks #DataEngineering #CloudData #ApacheSpark #DeltaLake #TechTalent #DataSkills #BuiltOnDatabricks #PartnerSuccess #RevInsights

3

[Demo] Databricks Senior Product Manager Alex L. joins Josue “Josh” Bogran to walk through Databricks One, a simplified analytics and AI experience built for business users. In the demo, you’ll see how users can: - Find trusted dashboards and Genie spaces, and ask questions in natural language - Use dashboards for core KPIs, with Genie handling follow-up analysis - Drill down from dashboard visualizations using Genie - Explore Research Agent mode for open-ended questions, with built-in explainability and governance https://lnkd.in/gRMvvp2S

290

14 Comments

Tomorrow at 4pm. Kythera Labs goes live at the Databricks Booth (5453). Kythera Labs will demonstrate something most health systems haven’t seen yet: AI agents investigating revenue movement in real time. Not static dashboards. Live workflows that: - Surface referral leakage - Identify service line growth opportunities - Reconstruct patient journeys across claims + EHR All built on a governed Databricks lakehouse foundation. If you’re responsible for data, strategy, or growth — this session is worth seeing.

8

1 Comment

I’ve been digging into Palantir Technologies Foundry’s ontology layer, and here’s my take: If you’re working with well-modeled datasets where relationships are already clear — the “hype” around ontology can feel a bit overblown. After all, as developers, we already know how tables relate, and querying them isn’t rocket science. Where the ontology truly shines, however, is enterprise-scale governance: RBAC (Role-Based Access Control) can be applied at the object or property level. Permissions automatically propagate across all related tables and views. You avoid manually applying access rules across dozens (or hundreds) of tables in Snowflake or other warehouses. Centralized semantic definitions reduce errors, enforce consistency, and make data easier to consume for analysts, apps, and AI models. In short, for a single developer, the ontology may feel like just another abstraction. For large organizations with many teams, strict compliance, and complex data pipelines, it becomes a powerful tool for governance and consistency. I’m curious: how do other developers feel about ontology-style semantic layers? Valuable abstraction or over-engineered marketing? 🤔 #DataGovernance #DataEngineering #PalantirFoundry #Ontology #DataOps #RBAC #S2DataSystems

12

1 Comment

What 'makes' the AI data stack? How do you architect it well? When do you start using Snowflake vs. DuckDB? Here's what hit for me at tonight's Vancouver Agentics Foundation Meetup (thanks Robert Ranson and Bill McGraw for hosting): 3 things that'll save backend and AI engineers hours of bad architecture decisions: 1. DuckDB vs Snowflake isn't about preference, it's about thresholds. Stay local-first until your pipelines regularly exceed 20-30 minutes, modeled data crosses 50-100GB, or you hit 3+ concurrent contributors. Before that? Snowflake is just wasted resources. 2. Business logic belongs in dbt models, not orchestration code. Dagster owns the run order. dbt owns the logic. Mixing them is how "temporary" models become production dependencies nobody can touch. 3. The real "modern" in modern data stack isn't the tools. It's the operating model: code-defined pipelines, automated validation before consumption, and modular components you can swap as scale changes. We built a game to make this stick: https://lnkd.in/g3vgh6XJ 6 zones. 18 lessons. 15 questions. If you're onboarding data engineers or just want to test yourself, try it. #ai #dataengineering #analyticsengineering #snowflake #duckdb #dbt #aiengineering #yvr #aijobs #fde

1

“The AI-powered system is anti-fragile. Every user interaction, especially corrections, becomes a signal that enriches the system for everyone. This model enables continuous improvement. Traditional semantic layers are most accurate the day they're built, then slowly decay until someone rebuilds them.” In the very near future hopefully we’ll stop talking about these traditional semantic layers altogether. MotherDuck has come to the same conclusion as us at Fabi.ai : traditional semantic layers are a story told by vendors and they’re expensive, fragile and not all that useful in an AI world. True story: I was chatting with the head of engineering at one of the big BI firms peddling semantic layers, and he told me “to be honest, we see very little adoption, we’re mostly hoping people adopt it for AI” (so basically hoping AI needs it so they can sell their product)

7

7 Comments

🚀 Unleashing the Power of Spatial Analytics in Databricks! 🗺️ Did you know you can now bring advanced spatial analytics directly into your Databricks workflows using ArcGIS GeoAnalytics Engine and the ArcGIS for Python API? Whether you're: ✅ Running large-scale spatial joins across millions of records ✅ Enriching your data with proximity, density, or drive-time analysis ✅ Integrating real-time location intelligence into ML pipelines ✅ Visualizing results interactively with ArcGIS Online or Enterprise …you can do it all seamlessly within your Databricks environment. 💡 ArcGIS GeoAnalytics Engine brings over 120 spatial functions to Spark 🐍 ArcGIS for Python API connects your spatial data to the full ArcGIS ecosystem. 🔥 Combine them to scale geospatial workflows across big data This is a game-changer for data scientists, GIS analysts, and ML engineers working with spatial data in the cloud. Curious how this works? Happy to share examples or use cases! https://lnkd.in/g5icgSte https://lnkd.in/giFdEbqK Matthew Perry Brady Cline Sarah Battersby Carsten Piepel Nikki Paripovich Stifle Michael Johns Kent Marten Olivia R. Szehon Ho #Databricks #ArcGIS #GeoAnalytics #Esri #SpatialDataScience #BigData #GeospatialAI #Python #GIS #LocationIntelligence

64

6 Comments

Automate. Accelerate. Analyze. Data teams spend too much time on manual coding and complex workflows. That’s why we created “WhereScape + Databricks: Automate Your Data Infrastructure and Analytics.” In this free white paper, discover how to: ✔ Cut data pipeline development from months to days ✔ Automate documentation and governance with full lineage ✔ Deploy Data Vault 2.0 on Databricks seamlessly ✔ Leverage Delta Lake, Unity Catalog, and the Medallion Architecture with automation Build smarter, not harder. Download now and transform how you deliver analytics: https://ow.ly/csY950WxoVx #Databricks #DataAutomation #DataEngineering #DataVault #Lakehouse #ETL

7

New A16Z article about popular "Context Graph" or "Meta Context" ideas for the AI-agentic applications. "...in fact, it’s difficult today to have a conversation with an organization working with data and AI and not have the topic of context come up." 👇 https://lnkd.in/dgxshqmb

1