Videos
All the course videos in one place. The narration and captions are in English, and each video has a full transcript below it. Selecting a time in the transcript moves the video to that part.
Before session 3: setting up the workspace
Practice starts in session 3, and before it you need a ready workspace. The two videos show how to set it up and how to work with Git and GitHub in the course.
Welcome to the Agentic AI course. Your lecturer is Dr. Eran Beeri-Bamani, founder and director of the Physical Human Intelligence Lab, and your TA is Alex Scherbakov. Before the first lesson, you'll set up your workspace on your own laptop, and you only do this once. It takes about forty-five minutes to an hour, mostly waiting for downloads. It costs nothing: no subscriptions, and no API credits. You'll need about four gigabytes of disk space, and ideally eight gigabytes of memory, for the real AI model. Here's the plan. Nine steps set up your laptop: accounts, tools, your own course repository, and a local AI model. A smoke test at the end checks that everything works. The last three steps are habits for the whole semester: saving your work, submitting your homework, and getting new exercises and their solutions. The course has two kinds of exercises. Lessons one, two and three are built live in class, with the lecturer. You follow along in your own copy, and you don't submit them. Each lesson builds on your code from the lesson before. Homework four and five you do on your own, and submit. Each homework stands alone: it brings its own copy of the lesson code, so it works even if your lessons aren't finished. And whenever a word is new, open the glossary, in the workspace folder. It explains every course term in plain language. Step one: create a GitHub account. GitHub stores your code, and your homework is submitted as a link to it. Go to github.com/signup, and use your university email. Choose your username carefully: it appears in every submission link. Verify your email with the code GitHub sends, and turn on two-factor authentication when GitHub asks. The Student Developer Pack is a free extra, but you don't need it for this course. Step two: install Git. Git saves versions of your code, and sends them to GitHub. On Windows, run the installer, and keep every default option. On a Mac, run xcode-select --install. On Linux, use your package manager. Then close and reopen your terminal, and check the version. Tell Git your name and email, using the same email as your GitHub account. Step three: install the GitHub command line tool, called gh, and sign in. It connects your laptop to your GitHub account, so pushing code never asks for a password. After installing, close and reopen the terminal. Run gh auth login, and answer the questions: GitHub.com, HTTP S, yes, and log in with a web browser. Copy the one-time code, and paste it into the browser. Finally, gh auth status should say that you're logged in. Step four: install Python, version 3.11 or newer, from python.org. On Windows, there's one checkbox you must not miss. Tick “Add python.exe to PATH” before you click Install Now. On a Mac, just run the installer. On Linux, install python three with your package manager. Then reopen the terminal, and check the version. It must say 3.11, or higher. Step five is recommended: install VS Code, a free editor that makes Python easier. Download it, then open the Extensions panel on the left. Search for Python, and install the extension published by Microsoft. Step six: create your own private copy of the course code. Open the course template on GitHub, click Use this template, and create a new repository. Name it agentic-ai-course, and make it private. A public repository would let other students copy your work. Then open Settings, Collaborators, and add the instructor: alexsherba29. Without this step, your submissions can't be graded. Finally, clone the repository to your laptop. From now on, you'll run every course command inside this folder. Step seven: create a virtual environment. It keeps the course's Python packages separate from everything else on your laptop. Create it once, from your course folder. Then activate it, every time you open a new terminal. The command depends on your system, and you'll see .venv at the start of the prompt. On Windows, if running scripts is disabled, run the Set-ExecutionPolicy command once. Now install the one package the course needs: pytest. Once the environment is active, python means the right Python, on every system. Step eight: install Ollama. It runs a real, free AI model on your own laptop, with no account, and it works offline once the model is downloaded. Install it from ollama.com. On Windows and Mac, it then runs in the background. Next, pull the course model: llama3.2:3b. It's about two gigabytes, and you only download it once. Try it with a short question. The first answer can take up to a minute. If your laptop is slow, pull the smaller Qwen model instead, and set the OLLAMA_MODEL variable. And if Ollama can't run at all, don't worry: every exercise, and every test, also works with the course's fake model. Step nine: run the smoke test. With your virtual environment active, run python workspace/smoke_test.py. It checks Python, pytest, the course code, your git repository, Ollama, and the model. A green tick means it works. A warning is only a hint. A red cross tells you exactly what to fix. No Ollama? Add --fake-only. When you see READY, your workspace is done. Step ten: save your work. Your code is only safe once it's on GitHub. At the end of every session: git add, git commit with a short message, and git push. Lessons two and three build on your own code from the lesson before, so keep everything in this one repository. Step eleven: submit your homework. Only homework four and five are submitted. The lessons are not. First, make sure the public tests pass. Save and push your work. Then create a tag, a bookmark on this exact version, called ex04-submit, and push it. Your submission is a link to the homework folder at that tag. Paste it into the course form. If you change something before the deadline, move the tag to your newest commit. And check that the link points to the right homework. Step twelve: get a new exercise, or a solution. Exercises are released one at a time, in the course template. The first time only, add the template as a remote called course. After that, fetch it, and check out the new exercise and the shared common code. Then commit, and push. Solutions come later, in each exercise's solution folder: the day after a lesson, and two to three weeks after a homework is released. To get one, check out only the solution folder. Your own code stays exactly as you left it. If something goes wrong, check the troubleshooting table at the end of the setup guide. Most problems are fixed by reopening the terminal, activating your virtual environment, or starting Ollama. If git push asks for a password, sign in again with gh auth login. And if a submission link shows 404, push the tag, and check the collaborator. Still stuck, or have any question? Email the course at eran.beeri-bamani@biu.ac.il, and include the full error message. That's it: your workspace is ready. In the first lesson, you'll build your first agent, together with the lecturer. If you have any questions, email eran.beeri-bamani@biu.ac.il. Good luck!Transcript and chapters
Introduction
The plan
How the course works
Step 1: a GitHub account
Step 2: install Git
Step 3: the GitHub CLI (gh)
Step 4: install Python
Step 5: VS Code (recommended)
Step 6: your course repository
Step 7: a virtual environment
Step 8: install Ollama
Step 9: the smoke test
Step 10: save your work
Step 11: submit your homework
Step 12: get exercises and solutions
If something goes wrong
Wrap-up
Git and GitHub: the basics. In this course, you save, share, and submit all your code with two tools: Git, and GitHub. This short video explains the few ideas and commands you need. Git is a program on your laptop. It records snapshots of your project, called commits, so you can always go back to an earlier version. GitHub is a website that keeps a copy of your project online. Your laptop has the copy you work on. GitHub has the backup, which the course staff can see and grade. A repository, or repo, is a project folder that Git tracks. Your course repo is called agentic-ai-course. It holds every exercise, plus a hidden folder called .git, where Git keeps the history. Never delete or move the .git folder. Every change passes through three places. First, your working folder, where you edit files. Second, the staging area. git add chooses which changes go into the next snapshot. Third, the history. git commit saves that snapshot, with a short message. Run git status at any time, to see where your changes are. Here's what that looks like in the terminal. git status shows the file you changed. git add -A stages all your changes, and git commit -m saves them, with a message that says what you did. git log shows your history: what changed, and when. Commit often. Small commits are easy to understand, and easy to undo. A commit lives only on your laptop, until you push it. git push uploads your new commits to GitHub. git pull downloads commits from GitHub, for example after you worked on another computer. Git calls the GitHub copy a remote. Your remote is named origin. To get a repo onto a laptop for the first time, you clone it. gh repo clone, then your username and the repo name, downloads the whole project and its history. You do this once, in setup step six. After that, just push and pull. A tag is a name for one exact commit, like a bookmark. You submit each homework with a tag: ex04-submit for homework four, and ex05-submit for homework five. Create it with git tag, and push it with git push origin, and the tag name. The course staff grade the commit the tag points to, so tag your finished work. Your repo has a second remote, called course: the course template. When a new exercise, or a solution, is released, git fetch course downloads it, and git checkout copies just that folder into your repo, without touching your own code. Then commit, and push, as usual. Four rules keep your work safe. One: push at the end of every session. Two: keep your repository private, with the instructor added as a collaborator. Three: never commit passwords, or tokens. Four: when Git shows an error, read it. It usually tells you exactly what TODO. Here's every command from this video, on one screen. Pause the video, and take a screenshot. That's all the Git you need for this course. If you have any questions, email eran.beeri-bamani@biu.ac.il.Transcript and chapters
Introduction
Git and GitHub
Repositories
Three places for a change
Status, add and commit
Push and pull
Clone
Tags: how you submit homework
New exercises and solutions
Four rules
All commands on one screen
Wrap-up
Session 3: build an agent in VS Code
The video walks the whole way in VS Code: a working folder, the Python and Foundry Toolkit extensions, and the Python packages; a free model running on your own computer (qwen3:4b through Ollama); developing the agent with Microsoft Agent Framework; and an evaluation workflow that measures, improves and measures again. At the end: ideas for what to build next, and how to share the agent's tools over MCP (Model Context Protocol), so VS Code's chat and Foundry Toolkit can use them too. All without a key and without paying, and it also estimates what a cloud model would cost. The code from the video is in code/vscode-agent/, and the lesson itself is session 3.
Build an AI agent in VS Code. In this video, we go from an empty folder to a tested IT help desk agent, without leaving VS Code. It runs on a free model on your own laptop, so it costs zero dollars. Everything you see was really run, and every number is real. Here's the plan, in four parts. One: set up the folder, the extensions, and the Python packages. Two: set up a free, local model, and see what a cloud model would cost. Three: develop the agent: its tools, its instructions, and the agent itself. Four: evaluate it: test questions, a score, one improvement, and a new score. It's the same help desk as lesson one. There, you wrote the loop yourself. Here, Microsoft Agent Framework runs it for you. Part one: set up. Step one: a working folder. Create a folder called vscode-agent, and open it in VS Code, with File, Open Folder. Or, in a terminal inside the folder, type code ., and press Enter. One folder per project keeps its packages, its settings, and its tests together. By the end, it holds the agent, a requirements file, five test cases, a scorer, and the tests. Step two: two extensions. Open the Extensions view, with Ctrl+Shift+X, or ⌘⇧X on a Mac. Search for Python, by Microsoft, and install it. It runs Python, picks your environment, and shows your tests. Then search for Foundry Toolkit, also by Microsoft, and install it. It used to be called AI Toolkit. It needs the .NET runtime. If VS Code asks for it, install it, and restart. A Foundry Toolkit icon appears on the left. We'll use My Resources, for the model, and Developer Tools, for the evaluation. Step three: a Python environment, just for this project. Open the terminal, and create it: python -m venv .venv. Then activate it. On Windows: .venv\Scripts\activate. On a Mac or Linux: source .venv/bin/activate. Tell VS Code to use it: Command Palette, Python: Select Interpreter, and pick .venv. Now the packages. Our requirements file has just two lines. agent-framework-ollama connects Microsoft Agent Framework to Ollama. It's still a preview, so the line asks for a preview of this one package only. Don't use the --pre flag instead. It takes previews of every dependency, and today, that installs a broken HTTP library. The second line is pytest, for our tests. Install both with pip install -r requirements.txt. It's about sixty megabytes. Part two: the model. Step four: a free model, with Ollama. You installed Ollama in the setup video. It runs open models on your own laptop: offline, with no account, no key, and no bill. Download qwen3:4b: ollama pull qwen3:4b. It's two and a half gigabytes. Why this one? An agent needs a model that calls tools well, and many small models don't. We tried llama3.2:3b first. It invented tool arguments, and answered from steps that no tool had returned. qwen3:4b called the right tools, with the right arguments. Microsoft's documentation recommends it for tools, too. With a graphics card, an answer takes about ten to fifty seconds. Without one, it's slower. Step five: the model in Foundry Toolkit. Open My Resources, Models, and the Catalog tab. Filter, Hosted by: Ollama. Find qwen3:4b, and select Add. Ollama must be running. The model now appears under My Resources, Local Resources, Models. Try it in the Model Playground: ask a short question, and check that it answers. It runs on your laptop, so nothing leaves it. Step six: what does it cost? Locally: nothing. Zero dollars, and no limit on how often you run it. A cloud model is paid per token, roughly a piece of a word. We measured our agent: one question used about 2,200 tokens in, and 2,300 out. With gpt-5-nano, at five cents per million tokens in, and forty cents per million out, that's about a tenth of a cent per question, or one dollar for a thousand. gpt-5-mini costs about half a cent per question, or five dollars for a thousand. Our whole evaluation, fifteen runs, would cost about two cents with nano. These are November 2026 prices, so check the provider's page. Part three: develop. Step seven: the tools. A tool is an ordinary Python function. The model never runs it. It asks for it by name, and the framework runs it. So the model decides from what it can read: the function's name, its docstring, and its parameters. So we write those for the model. Literal lists the only allowed services: VPN, email, and wifi. The model can't invent a fourth one. And the guide search is forgiving. Asked for VPN connection issues, it still finds the VPN guide. Our first version only matched exact words, and that's exactly where the model failed. Step eight: the agent. make_agent puts three things together: the model client, the instructions, and the list of tools. OllamaChatClient talks to Ollama on your laptop, with the model qwen3:4b. agent.run sends the question, and runs the loop from lesson one for us: ask the model, run the tool it chose, add the result, and ask again, until the model answers. Run python agent.py. The agent checks the VPN, finds the guide, and answers from both. Tip: Foundry Toolkit's Agent Builder, under Developer Tools, Build, Create Agent, lets you try instructions and tools without code, and View Code turns them into a project. Step nine: look inside one run. The result holds every message. For each tool call, we print the tool's name and arguments, and what it returned. Here: check_service_status with VPN returned operational. search_guides with VPN returned the guide. Then the answer, built from both. This is how you debug an agent: not by reading the answer, but by reading the steps. Part four: evaluate. Step ten: the test questions. One answer that looks right proves little. The same model can answer differently every time. So we write five cases, one per line, in eval/cases.jsonl. Each case has the question, the tools it should call, phrases the answer must include, phrases it must not include, and a ground truth answer. The printer case has no tool at all. The agent should say there's no guide, and send you to the help desk, instead of inventing one. Step eleven: score it. eval/score.py runs every case three times, and checks two things: did the agent call the right tools, and is its answer grounded in what they returned? Our first instructions scored eight out of fifteen for the right tools, and five out of fifteen for grounded answers. The pattern was clear. When a service was down, the model stopped after the status check, and never searched the guide. And for a forgotten password, it checked the email anyway, and mentioned the outage. Step twelve: improve, and measure again. We rewrote the instructions as four short rules. Check the status only for VPN, email, or wifi. Always search the guides, even when a service is down. Use only what the tools returned. And say so, when there's no guide. Same cases, same three runs: fifteen out of fifteen for the right tools, and thirteen for grounded answers. Two answers out of fifteen still weren't grounded. That's the next thing to look at. Change one thing at a time, and always compare on the same cases. Step thirteen: tests, and Foundry Toolkit's evaluation. tests/test_agent.py turns every case into a pytest test. Open the Testing view, and run them all. When we first ran them, the email test failed. But the answer was right: the service is down, use web mail. Our check wanted the word outage. So a failing test can be a bad test. We fixed the check, not the agent. And because the model varies, a good test can still fail now and then. That's why the score counts many runs. eval/score.py also saves every answer, with its ground truth, to a results file. In Foundry Toolkit, open Developer Tools, Monitor, Evaluation, and select Add Evaluation. Import that file, choose F1 Score, and select Run Evaluation. F1 compares each answer with the ground truth, word by word. It needs no judge model, so it's free. Evaluators like relevance, or tool call accuracy, need a judge model. A cloud judge costs about as much as one more question, per row. Step fourteen: what next. You have an agent, two tools, and a way to measure it. Here are five ways to grow it. Memory. Create a session, and pass it to every run, so a follow-up question knows what came before. Real guides, from documents instead of a dictionary, are session eight. Ask before acting. A tool that changes something, like resetting a password or opening a ticket, should wait for a person. Give it approval_mode="always_require". Permissions are session two. More agents. A triage agent can hand each question to a specialist, in a workflow. Planning and graphs are sessions four and six. Watch it. Send traces to Foundry Toolkit's Tracing view, and turn every bug you find into a new test case. That's sessions seven and ten. And share your tools, with MCP. Step fifteen: MCP, the Model Context Protocol. Right now, our tools live inside agent.py. Only this agent can use them. MCP is an open standard for offering tools to any AI app. A server offers the tools. A client, inside the app, calls them. mcp_server.py is the whole server: import the two tools, add them, and run. It talks over standard input and output. It reuses the same functions, so the names, the docstrings, and the vpn, email, wifi limit all reach the client unchanged. One server, many clients: our agent, VS Code's chat, and Foundry Toolkit's Agent Builder. It works the other way, too. Your agent can use MCP servers that other people wrote, for GitHub, files, or databases. But a server runs code on your laptop. Only add servers you trust, and give them only the access they need. Step sixteen: MCP in practice. Add two packages to the requirements file: MCP, and tenacity. The framework's MCP client needs both. agent_mcp.py starts the server, and gets its tools from it. The rest of the agent doesn't change. Run it. The same two tool calls, the same answer. But now the tools came over MCP. For VS Code's chat, the file .vscode/mcp.json registers the server. Run MCP: List Servers, and start helpdesk. VS Code's chat uses GitHub Copilot's model, not your local one. Copilot has a free plan, with a monthly limit. In Foundry Toolkit, under Local Resources, Tools, select plus, to add an MCP server. Then right-click it, and choose Connect to Agent. And a quick test checks the server on its own, without the model, in about two seconds. That's it: an agent built and tested in VS Code, for zero dollars. Set up the folder and packages. Pick a model that calls tools well. Write tools the model can read. And measure, before you trust it. Next, build your own in session three. Questions? Email eran.beeri-bamani@biu.ac.il.Transcript and chapters
Introduction
The plan: four parts
Step 1: a working folder
Step 2: two extensions (Python, Foundry Toolkit)
Step 3: a Python environment and packages
Step 4: a free model with Ollama
Step 5: the model in Foundry Toolkit
Step 6: what it costs
Step 7: the tools
Step 8: the agent
Step 9: look inside one run
Step 10: test questions
Step 11: score it
Step 12: improve, then measure again
Step 13: tests and Foundry evaluation
Step 14: what next
Step 15: MCP: share the tools
Step 16: MCP in practice
Wrap-up
In-class workshop: an IT support agent
A three-part workshop that builds an IT support agent together. Each part continues from the previous part's code: first the ideas, then every TODO with its finished code.
Lesson one: my first agent. In this lesson, we build a small IT support agent, step by step, together in class. It can check whether a service, like the VPN, is working, and look up a support guide. And we write the part that makes it an agent, and not just a chatbot: the loop. First, the four pieces our agent is made of. A tool is an ordinary Python function, like check_service_status. The model can't run code. It can only ask for a tool, by name, and our code runs it. The messages list is the agent's memory. It holds the question, every reply from the model, and every tool result, in order. The model is reached through one file in the common folder, the shared course code. In class we use the fake model: a pretend model that follows a short script, so everyone sees the same thing. And the tools talk to mocks: a pretend status page and pretend guides. They run on your own laptop, and send nothing anywhere. How does the model know which tools exist? We describe them. Each tool spec gives a name, a short description of when to use it, and its parameters. The model reads these descriptions, so we write them for the model. When the model wants a tool, it doesn't run anything. It replies with a tool call: just a name, and the arguments it chose. Our code looks up that name in a dictionary of real Python functions, and calls the function with those arguments. So the model chooses, and our code acts. That split is the whole idea of an agent. Now, what does the model actually see? A model has no memory between calls. Every time we call it, we send the whole messages list again. The list starts with two messages: the system prompt, which holds our rules, and the user's question. Then it grows. The model's first reply asks for check_service_status. We add that reply, and then the tool's result: the VPN is operational. On the next call, the model sees that result, and asks for a guide. We add the guide. On the third call, it answers. That's why adding the result to the messages matters: it's the only way new information can change the model's next choice. Here's the loop, in three moves. One: ask the model, with all the messages. If it asks for a tool, two: our code runs that tool. Three: add the result to the messages. Then back to one: ask the model again. When the model replies without any tool call, that reply is the final answer, and we stop. And if it takes too many steps, we stop anyway, with a polite fallback answer. That's the step limit. A word about the fake model we use in class. It has the same chat method as a real model. So our agent code can't tell the difference, and switching models changes no agent code. Inside, it's a few if statements. Nothing checked yet? Ask for check_service_status. The service works? Ask for the guide. Got the guide? Answer. So every run is the same, on every laptop, even without the real model. The tests use fake models too. At home, you can try a real model, through Ollama, with --backend ollama. We start from the starter folder. The tools and the demo are already written. The agent file has four TODO blocks: that's what we write together. Run the demo: it stops with an error at TODO three. Run the tests: all five fail. That's expected. Each TODO we finish makes another test pass. By the end of the lesson, all five pass. TODO one: stop when the model answers. After each call, we look at the reply. If it has no tool calls, the model is done: its text is the final answer. So we return it, with stopped_reason “answered”, the steps we ran, and how many model calls we made. Run the tests with -k final_answer. -k runs only the tests whose name contains those words. One passed. TODO two: run the tool the model chose. The tools are in a dictionary, from name to function. So we look the function up by the name the model gave, and call it with the model's arguments. What if the tool fails, say, because of a wrong argument? We don't let the agent crash. We catch the error, and return it as a result. The model reads that error on its next turn, and can try something else. A failing tool is information, not the end. Run the run_tool test: one passed. TODO three is the heart of the agent. The model may ask for one tool, or several. For each tool call: run it, and add the result to the messages, so the model sees it next time. We also record a step, so the tests can see what happened, and print it, so you can follow along. Then delete the raise line. Without it, the while loop goes round again, and asks the model with the new messages. Now the demo runs to a final answer: step one checks the status, step two looks up the guide, step three answers. TODO four: the step limit. Why do we need one? A model can get stuck, asking for the same tool again and again. Every call takes time, and with a paid model, money. So before every model call, we check: have we already made max_steps calls? If so, we stop, and return a fallback answer that sends the user to the help desk. Run the demo with --max-steps 1: it stops after a single call, without an answer. Run all the tests: five passed. Our agent is complete. Finally, the most important thing to see: new information changes the next action. Same question, two runs. With the VPN working, step one says operational. So step two looks up a guide, and step three answers with its steps. During an outage, step one says the VPN is down. So step two answers right away: no fix is needed, the IT team is on it. We didn't change any code between the two runs. The only difference was the tool result, which the model saw in the messages. That's lesson one: an agent is a loop. The model chooses. Our code acts, and decides when to stop. After the lesson, think about the two questions in your handout, and get the finished code from the solution folder, the next day. Questions? Email eran.beeri-bamani@biu.ac.il.Transcript and chapters
Introduction
The four pieces of an agent
Tool specs
What the model sees
The loop
The fake model
The starter folder
TODO 1: stop when the model answers
TODO 2: run the tool the model chose
TODO 3: run every tool call
TODO 4: the step limit
New information changes the next action
Wrap-up
Lesson two: an agent that uses my documents. Our agent from lesson one can now search the IT support guides, and answer from them. Every answer must name its sources. And when the guides don't cover a question, the agent asks a follow-up question, instead of guessing. The key idea: our code, not the model, decides whether an answer is backed by evidence. Why not just let the model answer from its own memory? First, it doesn't know your company: your procedures, your portal, your help desk. Second, what it learned may be out of date. And third, it can make up an answer that sounds sure, but is wrong. That's often called a hallucination. Documents fix all three. They're your own, you keep them up to date, and anyone can check them. So the agent answers from documents, and says which ones. Retrieval means looking things up before answering. The agent gets two tools. search_guides takes a short query, and returns the best matching guides, each with its doc_id, its title, and a short snippet. The doc_id is the guide's file name, like password-reset. read_guide takes a doc_id, and returns the guide's full text. The guides live in a mock document store: six pretend company guides, in the common folder, on your laptop. The model reads the guide, writes the answer, and ends it with a sources line, like Sources: password-reset. How does search decide what matches? With a score. The score is the share of the question's important words that a guide contains. Small words like how, do, and my are ignored. For: how do I set up a second monitor, only set, second, and monitor count. No guide has enough of them. For: how do I reset my password, the password reset guide has both important words: a score of one. Two other guides have only one of the two: a score of 0.5. Below a score of 0.6, a guide doesn't count as evidence. A weak match is not proof. Our system prompt asks the model for a fixed format, so our code can read the answer. The last line must be: sources, then the doc ids it used. And if the guides don't cover the question, the model replies with NEED_INFO, and a question for the user. A provided function, parse_sources, splits the answer in two: the text for the user, and the list of cited ids. Small models don't always follow the format. They forget the line, or cite a title instead of an id. The evidence check catches that too. But a model can invent a source, or answer from memory and add a real-looking id. So our code checks every answer, before the user sees it. Does it cite a source? And did the agent really retrieve that guide, in this run? Yes to both: the user gets the answer, with its sources. Otherwise, the agent asks a follow-up question, and says which guides it did look at. Not enough evidence is a normal result, not an error. A good support agent says: I couldn't find this. Can you tell me more? How does our code know what was really retrieved? From the steps. Remember lesson one: the agent records every tool it ran, with its result, as a step. retrieved_guides goes through those steps, and collects every guide a tool returned: from the search results, and from read guide. A tool call that failed returns an error, not a guide. So it doesn't count. The result maps each doc_id to its title. That's our evidence, for this run only. This lesson builds on your own lesson one code: the loop you wrote. If yours isn't finished, the course's reference copy is used instead, and one grey line says so. Right now, the search tool isn't written. The step log shows an error, and the tests fail. That's where we start. TODO one: search the guides. We call the store's search, with k=3, which means at most three results, and with the minimum score. If nothing is found, we say so clearly: found is false, with a message. The model needs to know there's no evidence. Otherwise, found is true, and for each guide we return just its id, title, and snippet. The full text comes later, from read guide. What should a question about chocolate cake return? Nothing: no guide matches well enough. Run the search_guides test: one passed. TODO two: check the sources. First, retrieved_guides tells us which guides this run really found. Then every cited id goes into valid, if it was retrieved, or invalid, if it wasn't. We keep the order they were cited in. For example, the model cites password-reset, and made-up-guide. password-reset was retrieved, so it's valid. made-up-guide wasn't, so it's invalid. Run the check_sources test: one passed. TODO three: only accept answers with evidence. This goes in judge_answer, after two checks that are already written: the step limit, and NEED_INFO. We split the answer with parse_sources, and check the cited ids with check_sources. At least one valid source, and no invalid one: it's an answer, and the user sees its sources. Anything else, no source at all, or an invented one, becomes a follow-up question. If a source was invented, we also log it. The careless demo uses a model that invents a source, called it-handbook-2019. Our check rejects it, and asks the user instead. Last, watch knowledge come from documents. Ask how to set up a second monitor. No guide covers it, so the agent asks a follow-up question. Now write a short guide, second-monitor.md, in the guides folder: a title, one sentence on when to use it, and a few numbered steps. Use the words people would search for. Ask again. Now the agent finds your guide, and answers, citing second-monitor. We didn't change any code. The agent's knowledge comes from the documents we give it. That's lesson two: answer from documents, name the sources, and let code check the evidence. After the lesson, think about the questions in your handout, and get the finished code from the solution folder, the next day. Questions? Email eran.beeri-bamani@biu.ac.il.Transcript and chapters
Introduction
Why answer from documents
Retrieval: two tools
How search scores a guide
The answer format
Checking the evidence
What was really retrieved
Starting from your lesson 1 code
TODO 1: search the guides
TODO 2: check the sources
TODO 3: accept only answers with evidence
Adding a guide
Wrap-up
Lesson three: an agent that acts with my approval. Our agent can now prepare a support ticket, but it may not create one on its own. A person has to approve it first. And while it waits, the task is saved, even if the program stops. Some actions are risky: creating a ticket, sending an email, deleting data. They change something in the real world, and can't always be undone. A model can misunderstand the user, or be tricked by text it reads. So the agent may only draft the action. A person decides. Only then does our code act. How does a request turn into a ticket? Our agent is the lesson two agent, with one new tool: draft_ticket. Its rules say: if the guide didn't help, or the user asks for a ticket, draft one. A normal question, like how do I reset my password, is answered from the guides, with sources, just like in lesson two. But if the user says the VPN guide didn't help, please open a ticket, the model calls draft_ticket. handle_request looks through the steps. If a ticket was drafted, it replies: pending_approval, with the task id. Nothing has been created yet. The draft is saved as a task, in a file called tasks.json. You can open it, and read it. Each task has an id, the draft, a status, and a history: a line for everything that happened to it. A new draft is pending_approval. A person approves it, or rejects it, with a note. Resume creates the ticket only for an approved task, records the ticket id, and marks the task done. Which status changes are allowed? One table, in the common folder, decides. pending_approval can become approved, or rejected. Approved can become done. Rejected and done are final: nothing comes after them. So a rejected task can never be approved later. Task.move_to checks this table. A change that isn't allowed raises a ValueError, and nothing changes. You'll also see failed in the table. It's for homework four, when the ticket system can't be reached. Why save the task in a file? Each command is a separate run of the program. Request runs, and stops. Later, approve runs, and stops. Later still, resume. When a program stops, everything in its memory is gone. Only the file is still there, so it carries the task from one run to the next. The mock ticket system forgets too: each run starts with an empty one. That's why the task records its ticket id. This is what keeping the task in progress means. It can wait for hours, and nothing is lost. Three separate locks stop a ticket that nobody approved. One: the model has a draft_ticket tool, but no tool that creates a ticket. Two: resume checks the saved status before it does anything. Three: the ticket system itself refuses, unless the task is approved. Any one lock alone is enough. We start from the starter folder. approve.py and the demo are written. We write three TODO blocks: draft_ticket, decide, and resume. Right now, asking for a ticket fails. draft_ticket isn't written yet, so the agent can't prepare one. All the tests fail. That's expected. By the end, all five pass. TODO one: draft, don't create. Make a new task: a fresh id from the store, like T-001, the question, the draft with its title, description, and priority, the status pending_approval, and one history line. Save it to the file. Then tell the model what happened: the task id, the status, and a message that says the ticket is not created yet. Otherwise, a model might tell the user it was. Run the draft_ticket test: one passed. TODO two: the person's decision. approve.py calls this function. Only a waiting task can be decided. Anything else raises a ValueError. So a rejected task can't be approved later, and a done task can't be approved twice. Then pick the new status, approved or rejected, and move the task to it, with a history line that includes the note. Keep the note, so resume can show it to the user, and save. Run the decide test: one passed. TODO three: block the action until it's approved. Resume reads the task from the file, and looks at its status. Still waiting? Say so, and create nothing. Rejected? Tell the user, with the reviewer's note. Approved? Only now do we call the ticket system. We pass the status, so the system can check it too, and the task id, as the idempotency key. Then record the ticket id, move the task to done, and save. Run the resume tests: two passed. What's an idempotency key? A label that makes repeating a request safe. We create a ticket with the key T-001. The system makes ticket TCK-3201, and remembers the key. Ask again with the same key, and the system doesn't make a new ticket. It returns the same one. Without a key, asking twice makes two tickets: a duplicate. In this lesson, that's good practice. In homework four, where a reply can get lost, it becomes essential. Now walk through both flows, one command at a time, and watch tasks.json change. Request a ticket: T-001 is pending. Resume it straight away: waiting. Nothing was created. Approve it, and resume again. Now the ticket is created. A second request, about the printer, is rejected with a note. Resume gives the user the note. And trying to approve it afterwards is refused: a rejected task stays rejected. That's lesson three: the agent proposes, a person decides, and three locks keep it that way. Next come the two homework exercises: handling failures, and two agents working together. Get the finished code from the solution folder, the day after the lesson. Questions? Email eran.beeri-bamani@biu.ac.il.Transcript and chapters
Introduction
Why risky actions need approval
From a request to a ticket
Tasks and their status
Allowed status changes
Why the task is saved in a file
Three locks
The starter folder
TODO 1: draft, don't create
TODO 2: the person's decision
TODO 3: act only after approval
The idempotency key
Both flows, one command at a time
Wrap-up
Found a mistake on this page? Email the lecturer