Altan Alpay

https://lnkd.in/dWhV_txb The latest challenge that LLMs need to overcome is how to use memory. In this article, an LLM is taught what to remember and how to use this memory.

11

The New Stack spotlights the PyTorch Foundation’s evolution into an umbrella foundation and the addition of vLLM and DeepSpeed as newly hosted platform projects. “Since its inception at the Linux Foundation two and a half years ago, the PyTorch Foundation has grown to include over 30 member organizations and 120 ecosystem projects. These newest moves are a natural evolution of its rapid growth and global momentum.” 🧠vLLM: Built for fast, efficient LLM inference with innovations like PagedAttention and deep PyTorch integration. ⚙️ DeepSpeed: A go-to library for scalable training, leveraging technologies like ZeRO and 3D parallelism—all built on PyTorch. 📖Read the full article from Steven J. Vaughan-Nichols: https://hubs.la/Q03lFYd00 #PyTorch #vLLM #DeepSpeed #OpenSourceAI #LLMInference

84

1 Comment

🆕 The Ultimate Guide to Open-Source RL Libraries for LLMs: 9 Frameworks Compared 🎯 Three primary RL applications for LLMs: 1️⃣ RLHF for human preference alignment 2️⃣ reasoning models for problem-solving 3️⃣ multi-turn agentic interactions [via Philipp Moritz of Anyscale ] https://lnkd.in/gcF2dNGq

57

1 Comment

#MLModel pipelines ≠ traditional dev. Bridge the gap with JFrog + NVIDIA NIM: https://jfrog.co/41Gr7q6 And now you're process will also cover: model versioning, GPU allocation, AI-specific security. #MLOps #DevSecOps #AI

33

To address the demand for sophisticated AI workloads, Anyscale and Google Cloud enable developers to deploy AI applications at a greater scale, supercharging their workloads with enhanced performance and reliability. Check out the full blog post by Matthew Connor and Dominic Catalano to learn more about how the companies are providing developers with consistent, cost-effective infrastructure.  #AI #Compute #Scalability

18

🔥 Huge milestone — WEKA just open-sourced their software-defined memory for PyTorch and LLM KV Cache! Proud to be a launch partner alongside vLLM, NVIDIA, PyTorch, and Oracle cloud — enabling faster, leaner LLM inference at scale with LMCache Lab as the caching layer. Open infra for the GenAI era is here. Learn more about us here: 🔗 https://lnkd.in/gvsbCG5p #LMCache #WEKA #LLMInference #SoftwareDefinedMemory #OpenSourceAI #PyTorch #vLLM #KVCache #AIInfra

12

Many enterprise GPUs run a single model during inference — even when it uses only ~30% of memory. So how much capacity is being left on the table? In our latest benchmark with Nebius, we used NVIDIA Run:ai fractional GPU allocation and NVIDIA NIM to measure real-world impact on throughput, latency, and concurrency. What we found: ✅ 86% of full GPU concurrent user capacity using just 0.5 slice ✅ 3× more users with mixed workloads on shared GPUs ✅ Near-linear scaling down to 0.125 slices ✅ Zero latency cliffs during autoscaling Stop GPU fragmentation. Start maximizing throughput. 🔗 Read the full deep dive: https://nvda.ws/3ZJgIbo

336

8 Comments

On top of the previous two articles, I wrote a new article on deploying Jupyter Notebook with GPU attached. The additional step is straightforward. We deployed a Kubernetes cluster on VMs. A GPU device is attached to a VM using device pass-through. We installed Nvidia operator on the Kubernetes cluster and enabled GPU sharing. The shared GPU resources are available to attach to multiple pods. We modified our Jupyter Hub deployment so that a new profile with GPU is added. We verified the GPU is used for computation in torch and Hugging Face code snippets. https://lnkd.in/g8paXZjf

4

Nvidia GTC presentation about structured & unstructured data being foundation for AI/LLMs used for business needs. We at OLake strongly believe the same and developing features around Iceberg. Apache Iceberg as per Feb 2026 release will soon be supporting unstructured data along side with structured and semi. https://lnkd.in/geYYF--6

20

GPU fragmentation is real. Most inference clusters run a single model per GPU and leave memory and throughput on the table. Fractional allocation is a meaningful step forward. But there’s a second dimension people are underestimating: time. Even if you slice a GPU well, utilization collapses when models take seconds or minutes to hydrate under bursty traffic. The real inefficiency isn’t just memory fragmentation. It’s lifecycle fragmentation. If we want true serverless inference economics, we need to optimize for both.

7

NVIDIA’s latest RunAI benchmark is a masterclass in infrastructure optimization. Achieving 86% of full GPU concurrent capacity on just a 0.5 slice is a massive leap forward in solving the idle compute crisis. But what happens when you need to scale beyond the physical limits of OS-level virtualization? What happens when you are managing a 10,000-unit autonomous drone swarm or filtering millions of cryptographic signals, and you can't afford the microsecond latency of context-switching between Kubernetes pods? You have to move the solution from the Infrastructure Layer to the Algorithmic Layer. At Cascade Space Systems, we took a fundamentally different approach to GPU fragmentation: Tensor-Native Multiplexing. Instead of partitioning the hardware into "logical fractional boundaries" (which eventually hit a hard limit via MIG or Time-Slicing), our Omni-Sieve architecture weaves infinite parallel compute streams into a single mathematical fabric. By natively interleaving memory-heavy filtering with compute-heavy arithmetic inside the exact same CUDA/MPS kernel, we bypass OS virtualization entirely. The GPU doesn't switch contexts between users; it evaluates thousands of concurrent telemetry streams in a single, massive Tensor Warp. The results change the paradigm for defense and cyber-physical AI: 🚀 Infinite Terminals per Node: Bypassing the 7-slice MIG limitation. ⚡ Zero Context Switching: True synchronous, hard real-time execution. 🔒 100% Silicon Saturation: Forcing deterministic safety without sacrificing throughput. #NVIDIA #Runai brilliantly solved how to efficiently host the AI brain. Cascade Omni-Sieve solves how to efficiently enforce the physics and logic guardrails for that brain at a civilizational scale. Stop dividing the hardware. Start weaving the math. #DeepTech #NVIDIA #SpaceForce #CyberSecurity #EdgeComputing #ArtificialIntelligence #JADC2 #CascadeSpaceSystems Eric Schmidt Schmidt Futures Timothy Costa

1

In this post, we demonstrate how to integrate Amazon SageMaker HyperPod with Anyscale platform to address critical infrastructure challenges in building and deploying large-scale AI models. The combined solution provides robust infrastructure for...

1

As Ikroop Dhillon shared, #NVIDIA #Nemotron 3 Super is now available on OCI Generative AI Model Import on day one. Huge thanks to the #NVIDIA team for advancing open and efficient foundation models, and a big shoutout to our OCI Generative AI engineering team who made day-0 availability possible for customers. Proud of the collaboration that brings the latest AI models quickly into developers’ hands.

11

Use case: scaling videogen with Baseten Inference Stack on Nebius ↔️ Generative video workloads run longer than others, stress GPU memory and magnify the impact of latency. Today’s blog shows how Baseten’s optimized runtime (including topology-aware parallelism) and orchestration operate on our clusters: https://lnkd.in/e23Bj7qc. Since last month, Baseten has been running on Nebius across our cloud regions in the US, Finland and France. #Partnerships #AIInfrastructure #ModelInference #VideoGen

63