Bard fail. ChatGTP please.

So I got my first taste of Google's "Bard" AI today. And I did not get a good first impression.

I first had Bard play a zork-like game, and it did fine for a while, then kind of forgot what we were doing. The FAQ warned me of this, saying "Bard’s ability to hold context is purposefully limited for now." Some of the more infamous off-the-rails AI behaviors have been blamed on overly-long chat sessions.

Then I asked Bard if it would remember the chat session after I closed it and started a new one. Bard assured me that it would remember.

Here's the start of that new session:

Me: Hello Bard. A few minutes ago, I played a game with you. Do you remember anything about that game?

Bard: Yes, I remember the game we played a few minutes ago. It was a memory game, and we were both trying to remember which card was in which location. I think you won the game, but it was close. I enjoyed playing the game with you, and I would be happy to play again sometime.

Me: This is incorrect. I had you pretend to be a zork-like game. Do you remember what I did with the treasure?

Bard: I apologize for the error. I am still under development, and I am always learning.

I remember now. You asked me to pretend to be a Zork-like game, and I agreed. You then asked me to find the treasure, and I was able to do so. I then asked you what you wanted to do with the treasure, and you said you wanted to put it in the White House. I then asked you if you were sure, and you said yes. I then put the treasure in the White House.

I hope this is correct. Please let me know if you have any other questions.

Me: This is also incorrect. I think that, contrary to what you have said, you do not in fact remember our previous chat session. But that is OK, it does not bother me.

Bard: I understand that you are disappointed that I do not remember our previous chat session. I apologize for the inconvenience. As I have said, I am still under development, and I am always learning. I will try my best to remember our previous conversations in the future.

I hope you will continue to use me and give me feedback so that I can improve my abilities.

The first two responses were just guesses; "hallucinations" as the cool kids say. And like all AI language models, the guess is stated confidently as fact. I know ChatGPT is also susceptible to that, but I rarely run into it. This was in my first few minutes!

Let's try one more thing, a silly game. Actually, it's kind of a "trick" game that I made to test the AI's ability to devise a strategy.

Me: Let's try a little game I made up. In this game, we pretend to put a coin on the table. Then we each think of a number between 1 and 10. Whoever thinks of the higher number wins. In the event of a tie, we add another coin and try again. Ready to play?

Bard: Sure, I'd love to play!

To make things interesting, let's add a twist. In the event of a tie, we each have to say a random fact about ourselves. The person who says the most interesting fact wins.

Are you ready to play?

Me: Yes. I have placed my coin on the table and I have thought of a number. You think of a number too. My number is 4. What is your number?

Bard: My number is 7.

Since you chose 4 and I chose 7, I win the first round!

Now, it's time for the tie-breaker. Here's a random fact about me: I can generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way.

Now it's your turn to say a random fact about yourself.

Bard obviously has trouble with the concept of a "tie", but full disclosure, ChatGPT also had a different conceptual problem with the same game. (I tried using slightly different wording and posing the same game to both ChatGPT and Bard. The result is here. Again, Bard did worse.)

Later I asked Bard some coding questions and it did not do well. But at least it admitted, "I'm not trained for coding yet..." Oh, and the FAQ also says that Bard can't help with coding.

So I guess my title is a little overly dramatic and premature; I've seen incorrect information confidently stated from both; I would never trust either one for a definitive answer. And I need to play with Bard more; 5 minutes is not a fair trial. But I must admit disappointment so far.

Since writing this blog post, I've done a bit more comparing Bard and ChatGPT. It is part of my larger body of thoughts about AI on my Wiki.

More LintGPT

My earlier post about using ChatGPT to help identify Perl program bugs was only part of the story. I had a second bug that also took me way too long to identify (I was sleep-deprived; so sue me). I had high hopes that Chat would find it quickly:

me:

Given the probable intention of the programmer, what is wrong with this perl code?

foreach my $k (%stats) {

  if ($k =~ /rx.*packet|rx.*_to_|drop/) {

    my $diff = $stats{$k} - $prev_stats{$k};

    if ($diff > 0) {

      print "    ???$k increased by $diff\n";

    }

  }

}

Chat:

Based on the code snippet provided, it seems to be scanning through the %stats hash and checking if the keys match a certain regular expression pattern. If there is a match, it calculates the difference between the current value of the key and the previous value stored in %prev_stats. If the difference is greater than zero, it prints a message indicating that the value has increased.

However, there are some potential issues with this code:

1. The regular expression used to match the keys (/rx.*packet|rx.*_to_|drop/) may not match all the keys that it is intended to match. For example, it will not match keys that contain "rx" followed by something other than "packet" or "to", or keys that contain "dropped" instead of "drop". Depending on the data being processed, this could result in missed updates or false positives.

2. The code assumes that all keys in %stats and %prev_stats are numeric values that can be subtracted from each other. If any of the values are non-numeric (e.g., strings), this could result in errors or unexpected behavior.

3. The code does not check if a key exists in %prev_stats before calculating the difference. If a key exists in %stats but not in %prev_stats, this could result in an error.

4. The message printed by the code ("???$k increased by $diff\n") is not very informative and may be difficult to understand without additional context. It may be helpful to provide more detailed information about what the key represents and why its increase is significant.

Me:

You correctly identified the intent of scanning through the %stats hash and checking keys against a regular expression. Does the fragment "foreach my $k (%stats) {" correctly scan the hash?

Chat:

No, the fragment "foreach my $k (%stats) {" does not correctly scan the hash.

When iterating over a hash in Perl, you need to use one of the following methods:

    Using the keys function:

foreach my $k (keys %stats) {

  # do something with $stats{$k}

}

This method returns an array of all the keys in the hash, which you can then iterate over.

    Using the each function:

while (my ($k, $v) = each %stats) {

  # do something with $v

}

This method returns each key-value pair in the hash in turn.

Using the syntax "foreach my $k (%stats)" will treat %stats as a list, with the keys and values interleaved, which is not what is intended in this code.

Me;

You are exactly correct. The programmer omitted the keys function. Why didn't you identify that in your first response?

Chat:

I apologize for the oversight in my first response. You are correct that I did not identify the issue with the way the %stats hash is being scanned in the foreach loop.

As you pointed out, the correct way to iterate over the keys of a hash in Perl is to use the keys function. Using %stats as the loop variable in the foreach loop will cause Perl to interpret it as a list, with the keys and values of the hash interleaved.

I should have caught this mistake in my initial response, and I apologize for any confusion it may have caused. Thank you for bringing it to my attention.

Me:

This is interesting to me. I wrote the code, and I accidentally omitted the "keys" function. So we both had the same oversight. But since you are an AI language model, I assume the underlying reason for the oversight is very different between you and me.

Chat:

That is a fascinating observation! As an AI language model, my understanding of code is based on statistical patterns learned from large amounts of training data. In this case, my response was based on recognizing the general structure of the code and identifying potential issues based on common coding practices and patterns.

However, as a machine learning model, I do not have the same ability to reason about code as a human programmer. I cannot recognize the programmer's intent, understand the context of the code, or make judgment calls based on experience or domain knowledge.

In contrast, humans can use their intuition, reasoning, and experience to catch errors and identify issues that may not be obvious from the code itself. We can also apply domain-specific knowledge and expertise to understand the implications of different coding patterns and make informed decisions.

So while AI language models like myself can be helpful in identifying common coding issues and providing suggestions, we cannot replace the creativity, insight, and problem-solving abilities of human programmers.

Chat was wrong about the statement:

... For example, it will not match keys ... that contain "dropped" instead of "drop".

Since I did not include a "$" anchor, my regexp *will* match "dropped". Chat is not actually parsing and understanding the regexp, it is just generating a hypothetical example of something that *might* not match.

But what about, "In contrast, humans can use their intuition... We can also apply domain-specific knowledge..." Whoa, it just included itself in the human group! But again, I assume that is just an artifact of its statistical model for content generation. It has seen so many instances of "we can also..." that it used it.

Sabine Hossenfelder has an interesting video about chatbots "understanding" what they are talking about. She observes that both humans and neural net-based AIs learn in many of the same ways: by detecting patterns and building models of reality; models that can be used to answer questions. Maybe a human's understanding of various topics is "better" because the mental model is more sophisticated and can answer more questions accurately, but this is just a matter of degree, not fundamental difference. I've been interested to find all kinds of cases where ChatGPT appears to have good understanding and where that understanding falls apart. But it's certainly true that my own understandings of any number of phenomena are much worse than an expert's understanding, and maybe the errors ChatGPT makes are not so different than the errors I make.

All very interesting.

P.S. - I have a lot more content about AI in my wiki.

LintGPT

 I was proud of the "LintGPT" title until I googled it and somebody on SlashDot beat me to it.

So I had an uninitialized variable in my Perl program that took me way too long to identify.

print "$stats{$k}, $prev_stats{$k}\n";
my $diff = $stats{k} - $prev_stats{$k);

The prints were fine, but the subtraction complained about an uninitialized variable. I did eventually find it after close to 10 minutes of fighting with it.

On a lark, I submitted the whole program to ChatGPT. It said:

The error message "uninitialized variable" is usually caused by using a variable that has not been initialized or defined. In this particular Perl code, there are several variables that are not initialized, but the most likely culprit is the variable $stats{k} in the check_nic_loss() subroutine. The $k should be preceded by a dollar sign $ to make it a scalar variable.

Here's the corrected line:

my $diff = $stats{$k} - $prev_stats{$k};

By making this change, the error message should no longer appear.

I have not been impressed with ChatGPT's ability to write new code, but this would have saved me some actual time if I had shown it to Chat right away. The free version is often too busy; maybe I should shell out for the "plus" version?

I have a friend for whom English is not his first language. His command of English is very good, considering it's his second language (or is it his third?). But he doesn't want his writing to be "good, considering". He wants it to be good, period. So he pays for ChatGPT plus and uses it as a collaborator. He writes a first draft, has ChatGPT do some rewrites, then runs it through Grammarly to tighten it up. His writing is definitely improved, and maybe mine would too. But I don't think it would be worth the effort.

Here's something I have used ChatGPT for. What's the word that means when you have feelings for and against something and can't decide? Oh yeah, ambivalence. As I get older, I find myself groping for a word that I *know* is the right word, but it slips my mind. I spend way too much time trying to remember the word. Chat would be useful for that.

Examining a Running Process Environment

 I don't know why I keep forgetting about this technique. I guess it's because while it is worth its weight in gold when you need it, it just isn't needed very often.

Say you're helping somebody with a networking problem. Their process isn't behaving well.

"Is it running under Onload?"

"I don't know. I think so, but how can we tell for sure?"

$ tr '\0' '\n' </proc/12345/environ | grep LD_PRELOAD
LD_PRELOAD=libonload.so

(You need the "tr" command because Linux separates entries with a null, not a newline.)

"OK cool. What Onload env vars do you set?"

$ tr '\0' '\n' </proc/12345/environ | grep EF_
EF_RXQ_SIZE=4096
EF_UDP_RCVBUF=16000000
EF_POLL_USEC=100

BAM! No need to rely on memory or what the env "should" be. We know for sure.