How LLMs reach beyond the conversation

01 · In lecture: Example: File Reading

File Reading

An uploaded file is turned into content the model can read, and that content goes straight into the context window.

Watch for

1. notes.pdf is dragged in and uploads.
2. Clawd reads it line by line. Every line becomes tokens in the context bar, up to +1,440.
3. Asked for the key points, he answers from the file and cites page 2.

The whole file costs context. What actually reaches the model depends on the file type and how the app processed it.

02 · In lecture: Example: Browsing

Browsing

Training data has a cutoff. For current facts the model calls a search tool, reads the results, and cites them.

Watch for

1. Today’s weather isn’t in the training data, so Clawd writes web_search("Iowa City weather").
2. He scans the results and pulls two snippets into context as web [1] and web [2].
3. The answer carries citations. Each one pulses with the source it came from.

Search does three jobs: finding what’s new since training, confirming what the model thinks it knows, and adding detail to topics it knows only a little about.

03 · In lecture: Example: File Search

File Search (RAG)

When the knowledge base is too big for the context window, retrieve only the pieces that match the question.

Watch for

1. Six lecture-note documents (480k tokens) are split into chunks and embedded into a vector index ahead of time.
2. The question is embedded too. A similarity ring picks the three nearest chunks.
3. Only those chunks enter context (+640 tokens), and the answer names the notes they came from.

Compare with File Reading: there the whole file went in; here only three small chunks do.

04 · In lecture: Example: Code Execution

Code Execution

Some answers need computation, not prediction. The model writes code, a sandbox runs it, and the output comes back as context.

Watch for

1. The model sees str · aw · berry as tokens, not letters, so counting letters by eye is unreliable.
2. The first run fails with a NameError, and the error goes back into context.
3. Clawd fixes the missing quotes, runs it again, and answers from the output: 3.

Errors are context too. Seeing the traceback is what lets the model fix its own code.

05 · In lecture: How?

How Tool Use Works

The model acts as an intelligent API caller. It knows which tools exist, decides when one would help, and writes the arguments. Software runs the tool, and the result returns as context.

Watch for

1. Roll call: a magnifier (browse), a paintbrush (image generation) and a terminal (run code).
2. Each call is written out with its arguments, such as browse("uiowa.edu").
3. Every result lands in the context bar before Clawd reaches for the next tool.

Same loop as the slide: tool descriptions → model requests a tool with arguments → software executes → result returns as context → model continues.

06 · In lecture: Example: Image & Video Understanding

Image Understanding

A vision-language model cuts an image into patches, turns each patch into tokens, and reads them alongside your text.

Watch for

1. The photo is split into an 8×5 grid of patches.
2. Row by row, the patches become image tokens in context.
3. The answer draws on the patches that matter, including the text typed on the screen.

The 8×5 grid is a toy version. Real models use far more patches, so one photo often costs over a thousand tokens.

07 · In lecture: Example: Image & Video Understanding

Video Understanding

Video usually reaches the model as a handful of sampled frames, each one treated like an image.

Watch for

1. The clip plays and one frame is captured each second.
2. Each frame is split into patches and becomes image tokens (f1 to f4).
3. The answer points to the moment it came from: 0:02.

Anything that happens between sampled frames is invisible to the model.

08 · In lecture: Expanded Capabilities and Tools

Audio Understanding

Native audio models turn short slices of sound into tokens, so they can pick up tone as well as words.

Watch for

1. The voice memo plays. Each second of audio becomes audio tokens (32 per second here).
2. The words surface as they are spoken.
3. The rising pitch at the end reads as excitement, and the reply responds to both.

Some systems transcribe speech to text first. That works for the words but loses the tone.

09 · In lecture: Be My Eyes Accessibility with GPT-4o

Realtime Voice and Camera

Realtime models stream camera frames and audio continuously, answer quickly, and keep listening while they talk.

Watch for

1. Camera frames and mic audio stream into context the whole time, even during silence.
2. Clawd starts answering about 0.3 s after you stop talking.
3. You cut in mid-sentence (barge-in). He stops, listens, and answers again.

Look for the overlap on the timeline: both voices at once is what full-duplex means.