Lila Ibrahim

This was probably one of the most inspiring discussions on the future of robotics and AI. It's the 2nd roundtable discussion I cohosted with Leo Lu -We’re pushing harder to connect the top minds across AI domains. This time, we brought robot learning and LLM researchers into one closed-door, off-record discussion. Why now? As scaling laws move to the center of “physical-world AGI,” the question shifts from if to how: When we can scalably obtain 1000× more robotics data, where do scaling laws actually hold for robotics foundation models—and where don’t they? What can we borrow from LLM scaling—and what must be different for embodiment? My (belated) takeaways (and with my personal POV): 1. Data volume ≠ performance. In robotics, pouring in more data doesn’t guarantee lift. Narrow distributions can lower loss while hurting generalization. We still lack pre-training metrics to target data/task diversity. 2. Is perfect action data overrated (early on)? For the foundation stage, do we really need massive, replayable teleop/UMI/DexCap data—or is rough next-step prediction (egocentric) enough? Perhaps RL/BC comes too early when the base model is still “a toddler.” Should we build the brain first; specialize later? 3. Human video, with caveats. VLMs already encode human action priors. Egocentric video is noisy, but might transfer better—if scaled sufficiently. The open question: how much scale before we see dependable gains? 4. Benchmarks are missing (and definitions too). What’s “success rate” in robotics—completion, speed, robustness? Without shared, task-relevant metrics, “scaling laws” reduce to anecdotes. 5. Sim2Real: ML-first over graphics-first. The room leaned toward ML-focused approaches (e.g., domain randomization) rather than photorealism. Simulation for evaluation is very real; training will lean increasingly on real-world data as collection and types of data scales. Plenty more to test, but one certainty: a wave of trainable robotics data is coming, and we need tighter feedback loops between LLM and robot learning communities. Thank you all for bridging this gap: LLM researchers from Google DeepMind OpenAI and Databricks, and; Robot learning researchers from Dyna Robotics Google DeepMind Robotics, OpenAI Robotics, NVIDIA, Amazon FAR, Stanford University University of California, Berkeley and UC San Diego (including Xingyu Lin Lirui (Leroy) Wang) If you’re working on algorithm / data / infra for embodied AI, LLMs, video-gen, or world models and want to compare notes on what it takes to scale a robotics foundation model, DM me. Let’s keep the dialogue going. #Robotics #AI #ScalingLaws #FoundationModel #AGI #LLM #Meetup #SFevents #researchers

59

3 Comments

It's an AI ecosystem most of us in the West don't hear as much about, but one that is critical for us to know and understand. This podcast gives great insight into the Chinese AI ecosystem. Smart teams can arbitrage and get the best of both AI worlds. https://lnkd.in/g6Qdj5Jy Thanks Wenchi Yu for this podcast.

9

1 Comment

NVIDIA CEO Jensen Huang: “I really discourage 1-on-1s” Jensen famously has 60 direct reports. When Stripe founder Patrick Collison points out that this isn’t conventionally considered best practice, Jensen shares his reasoning: “I don’t do 1-on-1s, and almost everything I say, I say to everybody all the time. I don’t really believe there’s any information that I operate on that only one or two people should hear about… I believe that when you give everybody equal access to information, that empowers people. And so that’s number one… Number two, if the CEO’s direct staff is 60 people, the number of layers you’ve removed in a company is probably something like seven.” Patrick offers to steal man the other side of the argument: “1-on-1s are where you provide coaching, where you maybe talk through personal goals and career advancement, where maybe you give feedback on something that you see somebody systematically not doing so well… Do you not do those things or do you do them in a different way?” Jensen responds: “I give you feedback right there in front of everybody. In fact, this is a really big deal. First of all, feedback is learning. For what reason are you the only person who should learn this?… We should all learn from that opportunity… Half the time I’m not right, but for me to reason through it in front of everybody helps everybody learn how to reason through it. The problem I have with 1-on-1s and taking feedback aside is you deprive a whole bunch of people that same learning. Learning from other people’s mistakes is the best way to learn.” I broke down the full operating mindset Jensen Huang chose to build NVIDIA, and the price he paid to make it last. Full breakdown here 👇 https://lnkd.in/ePbQi9ig

32

1 Comment

What is it really like working at a top AI foundation lab? Chong Ming Lee spoke with Prakhar Agarwal at Meta's Superintelligence Labs, in a fascinating conversation about the role. "What I would tell my younger self is to be comfortable exploring new avenues and new ideas. What I've seen is that we try to play to our strengths or stay in a deterministic setting where we know we'll do fine. But in these domains, the speed at which things are moving is so fast that you need to be able to switch to a new topic," he said. "Build the muscle to handle being thrown into something completely new. Sometimes, it's more psychological than a skill issue." On Business Insider https://lnkd.in/eeRnTKTT

81

3 Comments