The Power of LLMs
I wanted to learn some particular guitar techniques, and was pleased to find that Claude, Anthropic’s LLM, could generate tabs for a variety of techniques and styles, with many options. I could specify particular areas I wanted to practice such as hammer-ons, walk downs, double stops, and scales, or ask it to suggest a course of study, and generate relevant materials. Within minutes, I had pages of tabs for a variety and combination of techniques. The explanations were clear, the examples seemed musical, and I thought I’d found a good practice tool. This parallels the results I’ve had in the past when working with LLMs like Claude, Gemini, and ChatGPT to generate text and code–I quickly get impressive results.
If you’ve used these tools at all in the past few years you’ve had a similar experience. This technology is amazing: it can write poetry, create code, explain quantum physics, and create music. I use it to help generate ideas for this blog and suggest rewrites when I’m stuck. It is fluent in English, and excellent at explaining and summarizing. It seems to know something about everything. And LLMs are just one branch of generative AI—image and video generators produce equally stunning results every day.
This astounding capability has led people to form strong impressions about how this technology works—not the deep mathematical details, but assumptions about what’s happening when an LLM writes code or explains quantum physics. The fluency and versatility are so convincing that most people naturally assume these systems work like a combination of an extremely knowledgeable human and a traditional computer program: that LLMs understand concepts, retrieve relevant information, follow instructions, reason through problems, and produce consistent results. I’ve had fascinating discussions about this with family and friends, and these intuitive assumptions are almost universal. They’re also almost entirely wrong.
The Hype
In fairness, much of the expectations developed about LLMs come from the overall hype and overpromising surrounding this technology.
The companies that produce the LLMs have obvious incentives to make astounding claims for their products. And companies using the technologies justify their choices and try to appear cutting edge by talking up their implementations. Only in The Onion (satire, for those who do not know) do these organizations ever advise caution and restraint.
OpenAI describes ChatGPT as having “human-level performance” across diverse domains. Anthropic markets Claude as being able to “think through complex problems.” Google promotes Gemini as able to “understand and operate across different types of information.” These descriptions naturally lead users to expect human-like reasoning and reliability.
Meanwhile, companies implementing LLM solutions amplify these claims. Consulting firms promise an AI transformation and intelligent automation. For example, McKinsey claims there will be a $4.4 trillion impact from productivity growth. Software vendors tout AI-powered everything, from customer service that understands exactly what customers need to content generation that produces expert-quality output. Marketing materials overflow with terms like “intelligent,” “smart,” and “autonomous”—language that implies capabilities these systems simply don’t possess.
The tech press contributes to the cycle, with breathless coverage of each new model release and speculation about artificial general intelligence (AGI) being just around the corner. Even measured technical discussions often anthropomorphize these systems, describing them as “learning,” “understanding,” or “reasoning” rather than using more accurate terms like “pattern matching” and “probability calculation.”
This pervasive hype creates unrealistic expectations about the nature and behavior of LLMs. This messaging, combined with the unique and awesome experience of using them, has led people to a set of seemingly reasonable assumptions about the technology.
What People Think about LLMs
Looking more closely at the guitar tabs Claude generated for me I started noticing problems. I found incorrect titles, strange fingering systems, and many different alignment issues.
The issue wasn’t that Claude made mistakes—any tool can have bugs. It was how it failed that revealed something fundamental about how these systems actually work. When I pointed out the mistakes, Claude either apologized profusely and possibly repeated the same problem or denied the issue and assured me everything was correct. When I asked it to fix the alignment issues for instance, it would sometimes make them worse while insisting they were better. It could describe a valid tab that it proposed to generate, but any human could immediately see that output did not match its description. These behaviors directly contradict what most people assume about how LLMs operate.
- They are predictable – Since they’re computer programs, they’ll always respond the same way to the same input.
- They are trustworthy – Their confident, knowledgeable responses mean you can rely on their output
- They are logical – As code, they have built-in mechanisms ensuring truthful, rational responses
- They understand me – Their fluent, contextual responses show they grasp my meaning
- They follow instructions precisely – They’ll carefully execute whatever directions I give them
- They update from corrections – Later instructions override earlier ones, and they learn from my feedback
- They are friendly and helpful – They will try to do what’s best for me, and give me good advice
- They access current information – Like search engines, they always retrieve and return the latest data
- They understand context the way a human does – Their responses update with new information, so they gaining context much as a human would
Reasonable Assumptions, But Not Correct
These assumptions all treat LLMs as if they were either very smart humans or sophisticated traditional software. For example, many software systems leveraging other types of Artificial Intelligence like regression analysis, recommendation algorithms, or even image recognition, do behave more predictably than LLMs. Give the same input to a traditional AI system, and you’ll generally get the same output.
Of course, even these “predictable” AI systems have their famous failure modes—image recognition mistaking chihuahuas for muffins, or recommendation systems creating bizarre echo chambers. But those failures feel more like bugs to be fixed rather than fundamental features of how the system works.
LLMs and their generative AI siblings are fundamentally different. The randomness and inconsistency aren’t bugs—they’re core features of how these systems generate human-like text. Their strange behaviors, like confidently insisting incorrect guitar tabs are perfect, start to make sense once you understand what they are really doing.
And it is important to have a high level understanding of how they work. Each misconception can lead to real problems: trusting unreliable outputs, frustration when corrections don’t stick, confusion when identical inputs produce different results, or costly project failures from unexpected behavior. My guitar tabs were just practice exercises, but people are using LLMs for medical advice, legal research, business decisions, automated responses and actions (as AI agents), and mission critical applications. Understanding what these systems actually do—rather than what they appear to do—isn’t just intellectually interesting. It’s practically essential.
The Reality of LLMS
Trying to get the guitar tabs to correctly display was frustrating. Claude seemed to be gaslighting me, telling me not to believe what I was seeing. But knowing how LLMs work I realized that Claude wasn’t lying to me, trying to trick me, or win an argument. It was doing what it always does, following the probabilities.
LLMs are not like smart humans, and they don’t behave like other software.
They don’t follow instructions like traditional software. When you give an LLM instructions, you’re not programming it—you’re adding context that influences token probabilities. Claude wasn’t disobeying when it made my guitar tabs worse; it was weighing my formatting instructions against billions of other learned patterns (probabilities), sometimes producing unexpected results.
They don’t understand you—or anything else. When Claude generated detailed hammer-on explanations, it wasn’t drawing from music theory knowledge. It was predicting what words typically follow other words in guitar instruction contexts. The fluency creates a powerful illusion of understanding, but there’s no actual comprehension—just extraordinarily sophisticated pattern matching.
They don’t learn from corrections within conversations. When I kept fixing tab errors, Claude wasn’t updating its knowledge. Each new response uses the entire conversation context (including my corrections) to generate new probability patterns, but it’s not bound by previous responses. This is why similar mistakes can reappear—the correction influences but doesn’t control future outputs.
The Elephant Paradox: Humans have an interesting quirk. If you tell them not to think about an elephant, they have a tough time not actually thinking about one. LLMs have the same quirk! Anything you tell an LLM becomes part of the context and can influence future responses, even if you specifically ask it not to do so.
They don’t access current information or verify facts. LLMs recreate patterns from training data.They don’t continuously train so don’t know the latest information (this is called a knowledge cutoff). Without outside help they can’t fact-check themselves, search the internet, or know what happened yesterday. When they confidently state facts, they’re reproducing training patterns, not retrieving verified information.
They don’t have genuine personalities or motivations. The “helpful” behavior is trained pattern mimicry. Responses that sounded considerate and supportive were rewarded during training, so the system learned to reproduce those patterns. There’s no actual desire to help you.
They don’t behave predictably like other software. Built-in randomness means identical inputs can produce different outputs. This isn’t a bug—it’s fundamental to how they work. For example, if you work with the APIs for an LLM you can actually tweak various randomness factors to either give you the same responses all the time, or much more random responses.
To understand why Claude confidently insisted those broken tabs were perfect, you need to understand the fundamental process behind every LLM response. In the next post I will provide a high-level explanation of how LLMs work, provide some comparisons with human thinking, and consider many of the mechanisms to improve LLM safety, security, and reliability.