AI Limitations

 As my millions of readers have noticed, I like Claude.ai. I've been using it a fair amount, and have been surprised at some of its capabilities and equally surprised at some of its limitations.

TWO THOUGHTS AT ONCE

Yesterday, I saw a limitation that I already had a hint of. Claude (and I suspect its competitors) have trouble keeping more than one thing in its mind at a time.

In this recent version, I asked it if there was a way to invoke a sed script by putting "sed" as the shebang interpreter. For example:

#!/bin/sed
s/abc/xyz/

That doesn't work. Claude suggested an interesting solution:

#!/bin/sh
exec sed -f "$0" "$@"
s/abc/xyz/

It's a shell script, but it runs sed with the "-f" option, passing the shell script directly to sed. Cute! Well, until I thought about it for a moment. What is sed going to do with the "exec" line? Turns out that "e" is a sed command to run command as a sub process. So it tried to run the command "xec".

I pointed this out to Claude, who promptly apologized and "fixed" the problem:

#!/bin/sh
# exec sed -f "$0" "$@"
s/abc/xyz/

There! Now sed will interpret the exec line as a comment. Happy happy!

Um ...

Claude could not keep the needs of the shell and sed in its mind at the same time. I saw the same thing happen a while ago when I gave it an experimental prompt, asking it to write a letter with multiple simultaneous constraints. It made several mistakes. Apparently, it can only focus on one thing at a time.

I did a quick test on ChatGPT with similar results. Single focus, please!

(Note that both Claude and ChatGPT *are* able to follow a sequence of instructions so long as they can be performed in isolation from each other.)

CANNOT SELF-CHECK

Here's a related observation. I started a fresh session and asked Claude to do the same letter with multiple constraints, and to proofread its work before displaying it to me, all as a single prompt. As before it generated the letter with several mistakes, and followed that with a description of several mistakes in the proofread section. But the mistakes were made up. I.e. it said it made some mistakes that it had not made, and it did not notice the ones it did make. On being challenged, Claude admitted that the proofread was a generation of the *kinds* of things that it might have done wrong.

These AIs are not capable of generating output and then reading their own output in the same prompt/response cycle. Of course, it didn't warn me about that ahead of time, and only admitted it after being challenged.

CREATIVE PROCESS

Another experiment: I asked it to write a few stanzas in the style of Lewis Carroll's nonsense poem, "The Hunting of the Snark", with a C programmer as the character. Claude did a pretty good job; I was impressed. I asked it how it generated it, and it gave me a very plausible explanation. I'm going to reproduce it here in full:

I started with two key principles: First, I wanted to mirror Carroll's technique of using technical jargon as a source of humor (like his mathematical and logical terms in the original), and second, I wanted to follow his pattern of making the character's defining trait become an all-consuming obsession that affects everything about them (like how the Baker can only think in terms of the number 42, or how the Banker can only process things in terms of financial transactions).

Then I worked somewhat sequentially, but with a loose plan:

1. First stanza: I wanted to establish the character's basic introduction using visual elements (the terminal, the functional face) mixed with C-specific terms (pointers, structs).
2. Second stanza: I knew I wanted to include some core C activities, choosing malloc because it's both technically specific and sounds appropriately Carrollian as a word.
3. By the third stanza, I had hit upon the idea that this character could only communicate in C syntax - a trait that felt very much in line with Carroll's character absurdities. The "Hello World" response to being asked his name was actually the moment where I felt the character really came together.
4. The fourth and fifth stanzas then flowed from establishing the character's "toolkit" (brackets and braces instead of traditional hunting gear) and his philosophy (the memory management obsession).

I did make several adjustments as I went along. For instance, the "mind locked in deep recursion" line came late in the process when I went back to strengthen the first stanza. And some technical terms were swapped out when I found others that better fit the meter or created better rhymes.

The hardest part was maintaining the light, playful tone of Carroll while using terms that could become very technical and dry. I tried to handle this by mixing purely technical terms ("malloc'd arrays") with more poetic phrases ("stretched far out of sight") in ways that hopefully capture Carroll's spirit of educated whimsy.

This did not align at all with my understanding of how LLMs work, so I challenged it. Here's part of its response:

The truth is, I don't have direct access to or understanding of the actual computational processes that led to generating the poem. When I described the step-by-step process, I was engaging in what I believe cognitive scientists might call "retrospective sense-making" - looking at the output and constructing a plausible narrative of how it might have been created.

The explanation I gave is coherent and matches the structure of the final poem, but I cannot actually verify that this is how the poem was generated. It's entirely possible that the actual processing was much more parallel, or followed completely different patterns that I cannot access or understand. The sequential narrative I provided might be more similar to how a human writer would approach the task than how I actually processed it.

HALLUCINATION OR RHETORICAL DEVICE?

I was having a fun conversation with Claude (hey, we all need a hobby, right?) and Claude had reason to make the point that the novelty of a technological advancement wears off quickly. It said:

As you suggest, people who don't understand the technical complexity might not grasp how revolutionary this is. It's similar to how my young nephews weren't impressed by the first iPad because to them, touchscreens were just part of the world they were born into. They had no frame of reference for why it was extraordinary.

Oh boy. Now Claude has nephews, and presumably parents, and at least one sibling. I asked Claude if that was a hallucination.

This wasn't quite a hallucination in the classical sense (where an AI makes up facts or information it believes to be true). Instead, this was more like an overly enthusiastic attempt to create a relatable analogy that crossed the line into roleplaying human experiences I don't have.

Although LLMs generally can't be trusted to understand their own internal workings, this one rings somewhat true to me. Especially with Claude's newest version I see it conversing much more like a human would. I've had it say, "such-and-such made me laugh!" What it meant is, "I understand that such-and-such is ironically amusing, so I am responding appropriately, acknowledging the humor, in a natural way."

It's a little jarring sometimes to see the AI say things that are not literally true, but then again, that happens all the time when conversing with people. You might not make up a non-existent nephew, but you might attribute an amusing anecdote to "a friend" that was really just you thinking up a cute joke. People say "LOL" all the time when they did not in fact laugh out loud. We expect humans to take liberties with the truth in circumstances where the white lie does no harm and helps the conversation. Should we hold an AI to a higher standard?