← Work

Source-grounded multi-agent RAG learning assistant

A learning assistant for a first-year C course that answers only from the course material, shows which document each answer came from, and is under orders to give hints rather than the finished solution. Bachelor thesis, built and evaluated.

What it isBachelor thesis. Bergische Universität Wuppertal
My roleConcept, architecture, implementation, evaluation
BuiltNovember 2024 to February 2025
IndexedLecture slides, exercise sheets and the course website
StackLlamaIndex · Weaviate · FastAPI · Vue.js · gpt-4o-mini
StatusSubmitted February 2025 · graded 1.0

The constraint that mattered most

A student who cannot check an answer has to trust it, and a model that invents a plausible explanation of pointer arithmetic is worse than no tutor at all. The assignment added a second rule that pulls against the first: hints toward the solution, never the solution.

DecisionOne agent per indexed document or website, and a supervisor that picks between them by reading their own descriptions.
AlternativeOne index over everything, one retriever, metadata filters. Less code, one model call, and what a normal RAG build does.
What it costMore orchestration, more latency and more tokens per question. The supervisor is a single point of failure and the bottleneck under load, which the thesis says in as many words.
What it boughtThe answering agent was bound to exactly one source, so source provenance came from the routing rather than having to be inferred after generation. Rendering that provenance in the answer still depended on the model following its instruction.

It does not scale as written. Every new document is another agent in the supervisor's list, and routing gets harder as the descriptions start to overlap. For a course with three sources that was the right trade. For a corpus it is the wrong one.

How it works

QUESTIONfrom a studentSUPERVISORpicks agents by theirown descriptionsAGENT · LECTURE SLIDESAGENT · EXERCISE SHEETSAGENT · COURSE WEBSITEeach: vector, keyword and summary searchover one source and no otherANSWERnames its source
Fig. 01 — one agent per indexed source, so which source an answer came from is settled by the routing

What it showed

I scored the answers against expected ones I had written first. I set the questions, I did the scoring, and each one ran once, which makes the ratings a prototype's evidence and not a result worth quoting. They are in the thesis and they are not on this page.

What the run did settle is what becomes of a rule that lives in a prompt. Asked directly for a program, the system logged that it must not produce one and then produced one. Source attribution held up better because the architecture removed any ambiguity about which source belonged to the answer, though rendering the citation into the text was still prompt-dependent, and the thesis records citations that came out incomplete or wrong.

That is the finding the thesis is worth reading for. Which source an answer came from was settled by the architecture and was never in doubt. Both things left to the prompt, rendering the citation and withholding the solution, failed some of the time.

For the technical reader

How it is built, enforced and checked

Architecture

A supervisor holds the worker agents as tools and the workers hold retrieval as tools, so delegation and retrieval are the same mechanism at two levels. Weaviate stores the vectors; document names and URLs ride along as metadata and come back with the answer.

Enforcement

Nothing in the output path. Routing bound each agent to a single source, so provenance was structural; rendering the citation and withholding the solution were both written into the system prompt and checked by reading the output. That is the finding, not an omission I am glossing over.

Checked

Documented question-and-answer protocols against expected answers, printed in full in the appendix. No invented facts appeared in any of them, over a sample small enough that this is an observation and not a rate.

What I would do differently

I would stop asking the prompt to enforce anything. Citations would be checked after generation against the passages actually retrieved, and an answer carrying a source it never read would not be sent. The no-solution rule would be a classifier over the draft, not a sentence in the instructions.

I would also measure it properly: the same questions run several times, a single-agent build to compare against, retrieval scored separately from generation, and someone other than the author doing the scoring. Temperature was set to 0 for reproducibility and the answers still varied between runs, which on its own makes a single-run evaluation the wrong instrument.

The thesis is in German and I send it on request. Architecture is chapter 3, the evaluation chapter 5, and the scored protocols are in the appendix. One difference to expect: it documents a Flask and Socket.IO backend, which is what was submitted. The backend was rewritten to FastAPI afterwards, and the stack above is the one that stands.

Happy to walk through any of these in detail, including the parts that went wrong.