In my previous post, I described an experiment in which I had Claude.ai use its new web searching capability to become a news curator for me. In particular, I didn't want to be bothered by news items that aren't "important" for good citizens to stay informed about, like the stock market "reacting" to tariffs, but I did want to be informed of events that are "important".
I went thought it was working pretty well until this morning (Monday, April 21, 2025). Claude completely missed the news item that Pope Francis died. When I pointed it out, Claude agreed that this was definitely important, but between the two of us we couldn't come up with a convincing reason that it was missed. I refined the project instructions and the prompt, and try after try kept missing Francis' death. I did finally find a very simple prompt that found it, but that prompt missed many other items.
My conclusion is that LLMs go a much better job when given a specific thing to report on. Asking for death of the pope certainly finds it. Asking for death of notable people also found it. But I don't want to have to tell Claude all of the things I want to include. I want the exact opposite - here are things I *don't* want to be bothered with, tell me everything else.
Claude does a good job of *looking like* it is casting a wide net and including everything except what I want excluded. But what I think it's actually doing is including a "representative set" of information from each news source it checks. Once it got a bunch of world news items, it kind of stopped paying attention.
All that said, I also wonder if maybe the issue was web page format. Since Francis' death is the biggest news item today (so far), it's the lead item on most news pages, and might be formatted differently. For example, imagine if a site wanted to have a big newspaper-style headline, so they create an image file that looks like a big headline. Claude wouldn't be able to read it. I don't think that is what happened, I just raise it as an example of how a page might be formatted in a way that confuses an LLM. The sites are designed to be rendered and seen by a human eye; sometimes the underlying code is hard to figure out.
And I guess I don't care enough to look any further. I'm sad my experiment failed, but oh well.
No comments:
Post a Comment