Ask LLMs to Teach You, Not Tell You

A large language model can help you understand a subject, but usually people just use it to get an answer. This won’t help you. It matters who is doing the work and thinking. You should try to use assistants to provide hints, encourage effort, and give you feedback and not just a quick answer. That is if you want to learn.

TLDR; To use LLMs for learning:

1. attempt first,
2. then get assistance,
3. then review without the tool

If you skip the first step of trying yourself, then the model replaces your thinking rather than helping it. You need to start with your mind engaged.

Search Engine Knowledge

Search engines had similar concerns before LLMs arrived and Adrian Ward’s research on them supplies an intervention worth copying.¹

In eight experiments, Ward found a consistent pattern: people who used Google to answer general-knowledge questions became more confident in their memory and predicted better performance on a later test. Spoiler: they didn’t actually perform better. Their confidence was misguided as they often misattributed Google’s result knowledge to their own internal knowledge.

One of Ward’s experiments was for the participants to write their answer first before searching, making their personal knowledge clear ahead of time. Those who wrote first were more accurate attributing the source of knowledge, and calibrated confidence.

Chatbots have the same problem, an LLM quickly turns any question to an answer in a confident voice. This is often faster than our own internal memory search, which is why we use it, but Ward’s study shows people lose sight of what is their knowledge vs. the machines.

Love Island and Sesame Street

For decades, people argued whether TV was good or bad. But it’s not really about the technology but how you’re using it.

• Eight hours of Love Island: a continuous, frictionless show engineered to keep you watching and to ask nothing of you.

• Sesame Street, designed by researchers who tested segments on children, paced content around measured attention spans, and built in pauses where a kid can answer before the character.

These two programs use the same delivery technology, but require opposite cognitive demand. A program built to prompt thinking produces different results than one built to prevent you from changing the channel.

LLMs can be either. The same chatbot can walk you through a problem asking you questions and challenging you as you go, or it can just hand you an answer, which you’ll likely misattribute to your own knowledge. It all depends on how you approach and use the tool.

Performance is not learning

A student can submit a better essay with a chatbot and still understand less of it. The document alone isn’t a measure of the student, it measures both student plus model. Real learning shows up later, when a student has to retrieve, explain, or apply the idea without help.

A large field experiment makes the distinction. Hamsa Bastani and colleagues assigned nearly 1,000 high-school math students in Turkey by classroom to ordinary resources, a standard GPT-4 interface, and an AI tutor that offered teacher-written hints instead of answers.²

During practice, the standard GPT interface raised scores by 48 percent relative to the control condition, while the AI tutor raised them by 127 percent. On the immediately following closed-book exam, students who had used the standard chat interface scored 17 percent below controls. Students who had used the AI tutor did not differ significantly from controls.

Conversation logs showed students often asked the standard chat interface for an answer and copied it, while those using the AI tutor elicited more attempts and more requests for help.

A second experiment shows AI supporting learning when designers build it into the interaction. Greg Kestin and colleagues ran a crossover trial with 194 students in a Harvard introductory physics course.³ Students learned one lesson through in-class active learning and another through a structured AI tutor. The tutor paced students through expert-written activities, used supplied step-by-step solutions, and prompted active engagement rather than free-form answer generation. Students scored higher on immediate post-tests and spent a median of 49 minutes with the tutor, compared with 60 minutes in class.

The AI tutor also benefited from several advantages: self-pacing, videos, personalized feedback, a custom platform, and extensive instructor preparation.

Tools that just provide an answer can improve immediate performance, but hurt later unaided work. Using structured tutoring can improve real learning by supporting later recall.

Try First

1. Answer before searching or prompting. Ward’s Google study found less knowledge misattribution when participants wrote an answer before they searched.

2. Withhold complete solutions during practice. Bastani and colleagues’ guarded tutor prevented much of the harm found with the answer-giving interface.

Teachers can require an initial answer before opening AI for help, then assess students later without the tool. Software designers can offer hints before solutions, request evidence, show uncertainty, and prompt users to identify what would change their mind.

When using LLMs, write a rough answer, ask the model to criticize it, inspect the cited sources, then close the chat and think about the results. Make the LLM challenge you and make you think.

References

• Sesame Street research - street cred Wikipedia

Created: Jul 22, 2026