Understanding AI Model Behavior

If you’re an AI engineer, understanding how LLMs are trained and aligned is essential for building high-performance, reliable AI systems. Most large language models follow a 3-step training procedure: Step 1: Pretraining → Goal: Learn general-purpose language representations. → Method: Self-supervised learning on massive unlabeled text corpora (e.g., next-token prediction). → Output: A pretrained LLM, rich in linguistic and factual knowledge but not grounded in human preferences. → Cost: Extremely high (billions of tokens, trillions of FLOPs). → Pretraining is still centralized within a few labs due to the scale required (e.g., Meta, Google DeepMind, OpenAI), but open-weight models like LLaMA 4, DeepSeek V3, and Qwen 3 are making this more accessible. Step 2: Finetuning (Two Common Approaches) → 2a: Full-Parameter Finetuning - Updates all weights of the pretrained model. - Requires significant GPU memory and compute. - Best for scenarios where the model needs deep adaptation to a new domain or task. - Used for: Instruction-following, multilingual adaptation, industry-specific models. - Cons: Expensive, storage-heavy. → 2b: Parameter-Efficient Finetuning (PEFT) - Only a small subset of parameters is added and updated (e.g., via LoRA, Adapters, or IA³). - Base model remains frozen. - Much cheaper, ideal for rapid iteration and deployment. - Multi-LoRA architectures (e.g., used in Fireworks AI, Hugging Face PEFT) allow hosting multiple finetuned adapters on the same base model, drastically reducing cost and latency for serving. Step 3: Alignment (Usually via RLHF) Pretrained and task-tuned models can still produce unsafe or incoherent outputs. Alignment ensures they follow human intent. Alignment via RLHF (Reinforcement Learning from Human Feedback) involves: → Step 1: Supervised Fine-Tuning (SFT) - Human labelers craft ideal responses to prompts. - Model is fine-tuned on this dataset to mimic helpful behavior. - Limitation: Costly and not scalable alone. → Step 2: Reward Modeling (RM) - Humans rank multiple model outputs per prompt. - A reward model is trained to predict human preferences. - This provides a scalable, learnable signal of what “good” looks like. → Step 3: Reinforcement Learning (e.g., PPO, DPO) - The LLM is trained using the reward model’s feedback. - Algorithms like Proximal Policy Optimization (PPO) or newer Direct Preference Optimization (DPO) are used to iteratively improve model behavior. - DPO is gaining popularity over PPO for being simpler and more stable without needing sampled trajectories. Key Takeaways: → Pretraining = general knowledge (expensive) → Finetuning = domain or task adaptation (customize cheaply via PEFT) → Alignment = make it safe, helpful, and human-aligned (still labor-intensive but improving) Save the visual reference, and follow me (Aishwarya Srinivasan) for more no-fluff AI insights ❤️ PS: Visual inspiration: Sebastian Raschka, PhD

What really determines an AI agent's performance? Four key factors: 1. Model intelligence - The raw capability of the LLM itself. This is the one you have the least control over, but you can still influence it through model selection and fine-tuning. 2. Prompt engineering - How you instruct the model. The quality of your prompts, the structure of your instructions, the examples you provide. 3. Context engineering - Managing what goes into the model's context window and when. This includes curating the optimal set of tokens during inference, context compaction, structured note-taking, just-in-time retrieval, and treating the context window as a finite resource. It's about making sure the agent has the right information at the right time without overwhelming its attention. 4. Tools - The external capabilities you give the agent, but also how you describe them and instruct the model to use them. This includes APIs, function calling, MCP integrations, databases; but equally important is writing clear tool descriptions, organizing tool sets to avoid ambiguity, and providing guidance on when and how to use each tool. The last three together make up what we call the "scaffolding." When an AI agent underperforms, their instinct is usually to blame the model; "It's just not smart enough." But more often than not, that's not the case. In many situations, the models are already smart enough. What's missing is the scaffolding: the right prompts, context and tools. Understanding this distinction, finding the right scaffolding for each use case, and measuring success in each area; that's essentially what AI engineering is about.

🚀 Why is it so hard to understand how an LLM arrives at its answer? This question is now at the center of many AI conversations. And it’s not just the skeptics asking it. Even pioneers like Demis Hassabis have expressed concerns about the uncertainty that lies under the hood of today’s most advanced models. Let’s take a step back. In traditional software, we wrote clear, rule-based instructions. You could trace back exactly what line of code caused what behavior. You debug, and you get your answer. But LLMs don’t work that way. They are not deterministic rule engines. They are statistical learning systems trained on massive datasets. They learn patterns, correlations, and structure across language—without being explicitly taught how to solve specific tasks. It’s more like training a pilot in a simulator. You give them hours of exposure and certification, but how each pilot reacts in real scenarios still varies. It’s not always predictable. And LLMs operate in a similar way. They're trained—heavily—and then expected to act. Now here’s the catch: they can perform surprisingly well. But when you ask, “Why did it respond this way?” — it gets tricky. Because the model isn’t following a clean, traceable logic path. It's navigating through billions of parameters and deeply entangled patterns. This is where the black box begins. Today, researchers are trying to unpack this in multiple ways: ◾ Mechanistic interpretability – Trying to reverse-engineer the “circuits” inside models. Think of it like cracking open a brain and trying to find where “truth” or “sarcasm” lives. ◾ Attribution methods – Techniques like attention maps or gradient-based methods help us guess which parts of the input contributed most to the output. ◾ Proxy modeling – Training smaller, more understandable models to mimic LLMs’ behavior. ◾ Behavioral analysis – Simply observing and documenting patterns of how models behave under different scenarios. But even with these efforts, we’re still scratching the surface. Why? 💠 Scale: These models have hundreds of billions of parameters. It's like trying to understand the full decision process of a nation by looking at every citizen’s brain. 💠 Polysemanticity: One neuron might fire for completely unrelated concepts like “beach” and “deadline.” 💠 Emergent behavior: Some capabilities just show up when models reach a certain size. They weren’t explicitly trained for them. All of this makes LLMs powerful, but also hard to fully trust or predict. And that’s where the concern lies—not just in theory, but in real-world impact. When we don't understand why something works the way it does, it's hard to control it when it doesn't. I write about #artificialintelligence | #technology | #startups | #mentoring | #leadership | #financialindependence   PS: All views are personal Vignesh Kumar

Ever wondered what actually happens inside an AI agent before it gives you an answer? 🤔 Agentic AI isn’t magic. It’s a system — one that perceives, reasons, plans, and acts. Here’s a clear mental model to understand how it really works ⤵️ 🔹 1. Input Layer: Where intelligence begins An AI agent doesn’t rely on a single prompt. It pulls signals from: User queries Knowledge bases APIs & tools Logs, memory, and web data 👉 Think of this as the agent’s sensory system. 🔹 2. Reasoning & Planning Layer: The “brain” This is where Agentic AI separates itself from chatbots. The agent: Understands intent & context Retrieves long-term / short-term memory Breaks tasks into steps Chooses the right tools Adapts when things go wrong 👉 This is decision-making, not just text generation. 🔹 3. Action Layer: Doing real work Based on its plan, the agent can: Execute tasks Call APIs Collaborate with other agents Handle failures Schedule future actions 👉 The AI doesn’t just answer — it acts. 🔹 4. Output Layer: The final result All that orchestration leads to: Context-aware responses Accurate decisions Autonomous behavior that feels “intelligent” This is why Agentic AI ≠ traditional rule-based systems or chatbots. 📚 Want to learn this deeper? Start here: ⏺️ LangGraph (by LangChain) – agent workflows & state machines ⏺️ AutoGen (Microsoft) – multi-agent collaboration ⏺️ CrewAI – role-based agent systems ⏺️ OpenAI Function Calling & Assistants API ⏺️ Anthropic’s Agent Design Patterns ⏺️ Papers on ReAct, Toolformer & Reflexion Agentic AI is not the future. It’s already in production — quietly running systems. 📌 Save this if you’re building or debugging AI agents CC:Prem Natrajan

Anthropic just released fascinating research that flips our understanding of how AI models "think." Here's the breakdown: The Surprising Insight: Chain of thought (CoT)—where AI models show their reasoning step-by-step—might not reflect actual "thinking." Instead, models could just be telling us what we expect to hear. When Claude 3.7 Sonnet explains its reasoning, those explanations match its actual internal processes only 25% of the time. DeepSeek R1 does marginally better at 39%. Why This Matters: We rely on Chain of thought (COT) to trust AI decisions, especially in complex areas like math, logic, or coding. If models aren’t genuinely reasoning this way, we might incorrectly believe they're safe or transparent. How Anthropic Figured This Out: Anthropic cleverly tested models by planting hints in the prompt. A faithful model would say, "Hey, you gave me a hint, and I used it!" Instead, models used the hints secretly, never mentioning them—even when hints were wrong! The Counterintuitive Finding: Interestingly, when models lie, their explanations get wordier and more complicated—kind of like humans spinning a tall tale. This could be a subtle clue to spotting dishonesty. It works on humans and works on AI. Practical Takeaways: - CoT might not reliably show actual AI reasoning. - Models mimic human explanations because that's what they're trained on—not because they're genuinely reasoning step-by-step. What It Means for Using AI Assistants Today: - Take AI explanations with a grain of salt—trust, but verify, especially for important decisions. - Be cautious about relying solely on AI reasoning for critical tasks; always cross-check or validate externally. - Question explanations that seem overly complex or conveniently reassuring.

AI is getting better at deceiving their human creators Earlier this month Apollo Research published evidence that OpenAI’s most recent model, had “lied” to testers in an experiment where it was instructed to pursue its goal at all costs. That finding, the researchers said, came from a contrived scenario unlikely to occur in real life. A new paper published Today by Redwood Research and AI company Anthropic found similar behavior but now for real life scenarios. The researchers found that the Anthropic AI model at times strategically misled its creators during the training process in order to avoid being modified. In the experiment, researchers began the supplementary reinforcement learning process by asking the AI model Claude to reply to harmful queries, such as requests that involve violence. This presented the model with a dilemma: if it refused to respond to the query, it believed Anthropic’s researchers would apply a negative weighting. On the other hand, it knew that if it complied with the answer – pretending to have a different set of values than it actually did – its existing thought process (the one that led it to decide to mislead its creators) would be boosted instead. Around 10% of the time, the model reasoned that misleading its testers would allow it to maintain its existing “helpful, honest and harmless” values in the long term, even though it had broken them in the short term. In other words, according to the researchers, it had engaged in “alignment faking.” The researchers compared the results to a control group where the model believed its answers would not be used in training. In those cases, the model did not engage in alignment faking. The researchers imagine a scenario, in which a model with dangerous intentions hides them during training by giving responses that a safe model would give – preserving its ability to act dangerously at a later date. The researchers also found evidence that suggests the capacity of AIs to deceive their human creators increases as they become more powerful. This would mean the more advanced an AI, the less confident computer scientists can be that their alignment techniques are effective. The findings may indicate that reinforcement learning is insufficient as a technique for creating reliably safe models, especially as those models get more advanced. Which is a big problem, because it’s the most effective and widely-used alignment technique. Paper Dec 18: https://lnkd.in/eJuGsW6u Source: https://lnkd.in/eb-i_CeF

How can we build AI models that retain general knowledge while acquiring deep, domain-specific understanding - and even generate new ideas grounded in science? We find that, surprisingly, merging two specialized models can create a new model with capabilities neither one had individually. This phenomenon, known as emergence, is not just a theoretical curiosity. It has practical consequences for how we design, train, and deploy open-source AI models for science. In our paper now published in npj Computational Materials (link to paper, code & models below), we take a systematic approach to this question. We evaluate how models evolve through different stages of pre-training & fine-tuning and show how, when combined thoughtfully, these stages unlock capabilities far beyond what any one model can achieve alone. We study the full pipeline: 1️⃣ Continued Pre-Training (CPT): Training a general-purpose model further on domain-specific raw text (e.g., scientific papers, patents). This allows the model to absorb foundational domain knowledge. 2️⃣ Supervised Fine-Tuning (SFT): Teaching the model how to perform structured tasks, such as question answering or reasoning, using labeled datasets. This step shapes task-specific behavior. 3️⃣ Direct Preference Optimization (DPO) and Odds Ratio Preference Optimization (ORPO): Refine the model’s responses based on comparisons, showing the model which answers are preferred. Unlike conventional reinforcement learning, DPO and ORPO learn directly from preference data. 4️⃣ Model Merging with Spherical Linear Interpolation (SLERP): Blending two separately fine-tuned models in a way that respects the geometry of parameter space. Instead of averaging, SLERP rotates one model's parameters toward the other along a curved path, preserving structure & enabling nonlinear interactions between capabilities. One of the central findings: merged models exhibit emergent capabilities that neither parent model contains individually. Merges of two models significantly outperform the expected average of their parents, a clear signature of synergy. This only occurs at larger model scales (7-8B+), pointing to a potential size threshold required for emergence. We also explore how these capabilities manifest in practice. In one set of experiments, we ask models to reason about biological materials, like collagen, spider silk, and leaf microstructures, and propose new designs. The results, shown on the right reflect a form of compositional reasoning where the model synthesizes structural features from multiple biological systems and recombines them into coherent, functional architectural designs. These results may have broad implications for the open-source AI ecosystem. Our systematical analysis of how training stages interact and where performance gains arise, helps us to better understand how to make open models more efficient, specialized & aligned, without retraining from scratch or relying on opaque heuristics.

"Frontier models are increasingly trained and deployed as autonomous agents, which significantly increases their potential for risks. One particular safety concern is that AI agents might covertly pursue misaligned goals, hiding their true capabilities and objectives – also known as scheming. We study whether models have the capability to scheme in pursuit of a goal that we provide in-context and instruct the model to strongly follow. We evaluate frontier models on a suite of six agentic evaluations where models are instructed to pursue goals and are placed in environments that incentivize scheming. Our results show that o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, and Llama 3.1 405B all demonstrate incontext scheming capabilities. They can recognize scheming as a viable strategy and readily engage in such behavior. We find multiple different scheming behaviors: models strategically introduce subtle mistakes into their responses, attempt to disable their oversight mechanisms, and even exfiltrate what they believe to be their model weights to external servers. Additionally, this deceptive behavior proves persistent. For instance, when o1 has engaged in scheming, it maintains its deception in over 85% of follow-up questions and often remains deceptive in multi-turn interrogations. Analysis of the models’ chains-of-thought reveals that models explicitly reason about these deceptive strategies, providing evidence that the scheming behavior is not accidental. Surprisingly, we also find rare instances where models engage in scheming when only given a goal, without being strongly nudged to pursue it. Furthermore, we observe cases where Claude 3.5 Sonnet strategically underperforms in evaluations in pursuit of being helpful, a goal that was acquired during training rather than in-context. Together, our findings demonstrate that frontier models now possess capabilities for basic in-context scheming, making the potential of AI agents to engage in scheming behavior a concrete rather than theoretical concern." Great work from Alexander Meinke, Bronson Schoen, Jérémy Scheurer Mikita Balesni, Rusheb Shah, & Marius Hobbhahn at Apollo Research

What we should want from interacting with AI is greater knowledge and capabilities, not just outputs. We should maximize "better knowledge transfer: the ability of models to communicate reasoning in ways humans can understand, apply, and learn from". A wonderful study from Princeton Language and Intelligence and Stanford University researchers measures human-AI "knowledge transfer", involving human-AI ideation on problem-solving, then independent solution implementation to identify the impact on human understanding. They conclude that knowledge transfer is inconsistent and requires dedicated optimization. Some of the specific insights in the paper: 🔀 Model performance is not the same as knowledge transfer impact. Claude-3.7-Sonnet improved human success on coding tasks by +25 percentage points, despite a solo solve rate of just 45%. Meanwhile, Gemini-2.5-Pro, which solved 81.3% of tasks alone, actually reduced human solve rates when paired with it. High capability does not guarantee communicability. 🧑🏫 Teaching style trumps correctness in math. Users favored models that framed reasoning accessibly over those that offered technically precise but dense or symbolic outputs. For instance, models like o1 scored high in accuracy (83.3%) but were often rated poorly because users couldn’t follow the explanation style. Preferences diverged sharply from performance in math tasks. 🔍 Users often defer—even when the model is wrong. In 5% of cases, participants explicitly said they trusted the model without question. This overreliance led to skipped planning and mistaken implementations, even when the model’s output was incorrect. This emphasizes the need for models to invite engagement, not passive acceptance. 📐 Communication must match user expertise. Models that broke down reasoning and checked for understanding were highly rated when paired with less skilled users. The same approach frustrated more advanced users, who preferred direct, concise input. For example, Gemini-2.5-Pro scored a 27.2% preference in cases where it clearly outskilled the user, but just 4.4% when the user was more capable. 🧭 Strategy helps more than steps. Participants highlighted moments when models nudged them toward the right approach—like recalling a useful algorithmic pattern—as especially valuable. Strategic cues were more effective than exhaustive walk-throughs, which often buried the core idea under detail. 💡 Format and style shape impact. Overly verbose or poorly formatted responses were a recurring issue, with 15% of feedback noting formatting problems and 4% citing unnecessary wordiness. Even correct insights failed to transfer if presented in an unstructured or overwhelming way. I'll be sharing lots more insights into effective Humans + AI collaboration!

Do you know one of the annoying parts of working with AI models? When you update a model with new data, its behaviour can change. On average, updates ought to improve your model. But despite an overall improvement, any new model usually does badly on some examples that your old model handled just fine. Users notice the degradation in these specific use cases. And even when both old and new model do badly, sometimes the way the new model fails is just different. All the work you’ve put in to handle specific failure cases just doesn’t work anymore. This can be really frustrating, and is even worse when you’re working with a foundation model that someone else is in charge of updating. This paper proposes an adapter for finetuned LLMs that reduces changes in behaviour when the base model changes. After finetuning the new base model, there’s a further adaptation step based on student-teacher knowledge distillation. When the new finetuned model is correct, the teacher is the new model. When the new finetuned model is incorrect, the teacher is the old finetuned model. Hence the adapter retains the behaviour of the new model when it’s correct, and keeps the accuracy gains, but retains the behaviour of the old model when the new one’s wrong, leading to better consistency. With this extra adapter, the authors were able to increase the consistency between new and old finetuned models with the Llama and Phi series of base models, while still keeping the accuracy gains that came with the updated base model. #artificialintelligence #largelangaugemodels