Why AI sounds smarter than it is, and what schools should do

Fluency is not accuracy

A language model predicts plausible text. It has no mechanism for being unsure in the way a person is unsure, and nothing in the writing signals the difference between a sentence it has strong grounds for and one it assembled because it reads well.

Humans use fluency as a proxy for competence, because for humans it usually is one. Confident, well-organised prose from a colleague generally means they know the subject. That heuristic is what these tools break, and it breaks hardest on readers who are tired and in a hurry.

Where it costs a school

• A confident wrong answer about entitlement, procedure or a statutory deadline, repeated to a parent as though it were checked.
• An invented citation — a circular number, a section of an Act, a guidance document — that looks exactly like a real one.
• Plausible detail in a report about a child that nobody wrote and nobody noticed.
• Advice that is accurate somewhere else. A great deal of material online describes English or US practice, and a model trained on it will answer an Irish question in English terms without flagging the swap.

The habit that catches it

Checking everything defeats the purpose; checking nothing is how the wrong thing reaches a parent. The workable rule is to scale the check to the consequence.

Anything that will be read by a pupil, a parent or an inspector, anything that states a rule or a deadline, and anything citing a source gets verified against the source. Lesson ideas, phrasing and first drafts of internal text do not.

Two specifics worth building in: never accept a reference without opening it, and treat any answer about Irish education law as unverified until it has been checked against the Department of Education or the relevant Act. A tool grounded in Irish material narrows this considerably, but it does not remove the step.