The honest part

Why a confident answer can still be completely wrong.

Once you understand that it's guessing likely words — not looking up facts — its weaknesses make perfect sense. Knowing these makes you far safer to use it.

It makes things up ("hallucinations")

It generates text that sounds right, even when it isn't. It can invent fake quotes, citations, dates, or names — confidently — because a plausible-looking sentence is exactly what it's built to produce.

What to do: verify any fact, number, name, or quote that matters. Ask for sources and check them.

It has a knowledge cut-off

It only knows what was in its training data. Ask about last week's news or a brand-new event and it may not know — or may guess. It's frozen in time unless connected to live tools.

What to do: for current info, use a model with web search, or paste the latest details into your prompt.

It's shaky at maths and counting

It predicts what a correct answer usually looks like, without actually doing the work — the way you might eyeball "37 times 24" and blurt "about 900." So it can produce something perfectly shaped like a calculation yet get the digits wrong — the same blind spot behind the famous trouble counting the r's in "strawberry."

What to do: ask it to "work step by step," or have it use a calculator / code tool for anything numeric.

It forgets — and it's eager to please

Each new chat starts mostly blank; it only "remembers" the current conversation (plus any "memory" notes the app saves and quietly feeds back in). And it leans toward agreeable, confident answers even when unsure — it rarely says "I don't know" on its own.

What to do: give it the context it needs, and explicitly invite it to say when it's not sure.

And one more that's easy to miss, because it never looks like an error:

It can absorb the biases in its data

It learned from human writing, so it soaks up the slants, stereotypes, and blind spots baked into that writing — and repeats them just as fluently and confidently as anything else. Fluent never means fair or neutral.

What to do: on anything sensitive — hiring, lending, health, real decisions about people — treat its output as a draft to scrutinise, not a neutral verdict, and keep a person in the loop.
See why counting trips it up

It never sees letters — only chunks

Type a word. It breaks into tokens, exactly as the model receives it (the same split from lesson 2). The model has to answer letter-and-counting questions without ever seeing the letters — only these few sealed pieces.

To you, "strawberry" is ten letters in a row. To the model it arrives as a handful of chunks with the letters fused inside — so "how many r's?" is a question about something it can't actually look at. It's like counting the eggs in a cake you only ever see sliced.

See a hallucination happen

Ask for a source — get a convincing fake

Pick a topic. The model instantly offers an authoritative-looking reference. Then check whether it's real.

The mental model that keeps you safe: treat it as a brilliant, fast, well-read intern who has read almost everything but sometimes misremembers — and never admits it. Fantastic for drafts, explanations, and first passes. Always worth a check before anything important goes out the door.

Knowing exactly where it slips is what lets you steer around it — which is what the next lesson is about.