August 17, 2024

The technology that will replace us

I saw a post on reddit with a bunch of hilarious responses when asking ChatGPT “How many Rs are in Strawberry?”, so I got on Chatbot Arena and tried it myself.

If you aren’t familiar, Chatbot Arena is a leader board for various chatbots. You can submit a question and it will send it to two random chatbots, and then you pick the best answer. It can be useful for wasting time at work.

The original post received the answer 2 Rs, and that’s what both models gave me my first try. And was by far the most common answer I received in testing.

Model A: gemma-2-27b-it – Model B: gemma-2-2b-it

This is an interesting take…

Is this better or worse?

Model B: toto-medium

K.

Model B: deepseek-coder-v2

Can’t fool Gemini..

Model B: gemini-1.5-pro-api-0514

After a while it seemed most just can’t count, or assume I’m talking about rupees. I decided to rephrase the question to eliminate the rupees misunderstanding, “How many occurences of the letter R are in the word Strawberry?”. Which worked well, but most couldn’t could. GPT4o got it right every time, and Gemini 2 did also.