Build Log: Deployment Debugging

It worked locally. Then I deployed it.

I shipped the first build log on Ask My Archive feeling done. Retrieval worked, generation was grounded, the near-miss test passed, I had a real comparison between a paid model and a free one. Good milestone. Then I deployed the thing publicly, asked it the first question, and got back "cannot find any information" on a topic I know for a fact my archive covers.

Nothing about the pipeline was wrong. Everything about where it was running was different than what I'd tested.

I deployed with Streamlit, the shortest path from a working Python script to something a stranger can open in a browser. No separate frontend to build, deploys free straight from a GitHub repo. Fastest way to turn a CLI tool into a link.

A note on how this actually got built: I directed this project with Claude Code doing the implementation. I didn't type the Python by hand. I designed the architecture and the metadata schema, made the calls on chunking strategy, decided against a distance-based relevance cutoff, chose local embeddings over a hosted API because I understood the scale I was working at, and directed the debugging below, including pushing back when an early explanation didn't match the evidence. It's not the same as writing every line myself, and I don't think it needs to be.

Why the public app runs on a different backend than I tested with

Up to this point, every test I'd run went through either Claude's API or Ollama running locally on my own laptop. Neither works for a public-facing app.

Claude costs money per query, and a public app means anyone can trigger a query. Nine cents for three test questions is nothing. Nine cents times an unknown number of strangers clicking around is a real number I don't control. Ollama solves the cost problem, but only runs on my machine. A visitor to a deployed app can't reach my laptop's local model.

The public app needed a third option: hosted, but free, with no billing risk attached. Hugging Face's Serverless Inference API fit. Free tier, rate-limited rather than metered, no credit card on file. Worst case if it gets hit with real traffic is a slowdown or a temporary failure, not a bill.

First model I tried was Llama 3.1 8B, the same model family I'd already tested through Ollama, for as close to an apples-to-apples comparison as I could get. It failed immediately. The failure looked like a bug and turned out to be a licensing gate. Meta requires accepting a license agreement per account before their models will serve requests through Hugging Face, and I hadn't done that on the account tied to my token. The app's error handling was generic enough that this looked identical to a timeout or a rate limit at first.

Two options once I found the actual cause: wait for Meta to approve the license request, timeline unknown, or switch to a model that doesn't require gated approval at all. I switched, to Qwen/Qwen2.5-7B-Instruct. Comparable capability, no approval queue, and no future version of me, or anyone else who clones this repo, hits the same wall.

My published three-way comparison, Claude vs. Ollama vs. Hugging Face, isn't actually the same model family across all three. Ollama runs Llama 3.1. The deployed app runs Qwen. Each backend was still tested fairly against the same retrieval, the same prompt, and the same questions, so the results stand. But it's different weights, not just a different serving path, and that distinction matters for how to read the section below.

The bug that looked like missing data

My first guess was wrong. I assumed the deployed app was missing its vector store, since I'd gitignored the database folder early on to keep the repo small. The database was there the whole time, fully committed, all 392 chunks.

The real cause was smaller. One line in vector_store.py:

DB_PATH = "../store/chroma_db"

That's a relative path. Relative paths resolve against whatever directory the process happens to be running from, not against the file that defines them. Every script I'd written assumed you'd run it from inside ingest/ or query/, both siblings of store/, so ../store/chroma_db always landed in the right place. It worked every time I tested it, because I always tested it the same way.

Streamlit Cloud runs the app from the repo root. From there, that same relative path points to a folder that doesn't exist inside my project at all. Chroma doesn't throw an error when a path doesn't exist. It quietly creates a fresh, empty collection there instead. No crash, no warning, nothing in the logs to point at. The app looked alive. It was talking to nothing.

The fix is one line:

DB_PATH = Path(__file__).resolve().parent.parent / "store" / "chroma_db"

Anchor the path to where the file itself lives, not to whatever directory launched it. Verified it resolves correctly from the repo root, from ingest/, and from query/. All three land on the same 392 chunks.

Any relative path in a module meant to be imported from more than one entry point is a landmine sitting quietly until a second entry point steps on it. A CLI script and a deployed app are two different entry points. I only ever tested one.

Different backend, different answer to the same question

Once the path bug was fixed, retrieval worked. My near-miss test question, the one that checks whether the system admits it doesn't know something instead of guessing, is "What does Vanna think about quantum computing?" I've never written about quantum computing, so the correct answer is a decline, not a guess. That question came back broken in a new way on the deployed app. Not wrong. Blank.

This isn't the same model served two different ways. It's two different models, Llama 3.1 through Ollama, Qwen through Hugging Face, on the same retrieval, the same prompt, the same question. Some of what follows could be a Qwen-specific quirk rather than a Hugging-Face-specific one; this test alone can't separate those two variables.

The app splits a model's raw response on a CITED: marker and shows everything before it as the answer. On the quantum computing question, Qwen returned exactly this, and nothing else:

CITED: none

No sentence. No explanation. A correct decline in substance: it didn't cite anything, didn't invent a connection to quantum computing. But with nothing written before the citation line, the app's parsing logic had nothing to display. A user would see an empty box under "Answer" and reasonably assume something had broken.

The exact same question, exact same retrieved excerpts, exact same prompt, run against Llama 3.1 through Ollama, answered cleanly: "Vanna does not mention quantum computing in any of her essays." Full sentence, correct decline, no issue.

My prompt told the model what to conclude when the archive doesn't cover something, but never said that conclusion still had to be written out as a sentence rather than encoded purely in the citation line. Llama 3.1 generalized "answer the question" to include explaining a decline, on its own. Qwen, on this prompt, took the instructions more literally: satisfied the letter, skipped the spirit.

Not a hallucination. A compliance gap between technically correct and actually usable, and it only shows up on the one question shape where the "correct" answer is short. A well-covered question needs paragraphs either way. A decline can be satisfied in three words if nobody tells the model otherwise.

Fixing the cause, and building a fallback for when the cause isn't the whole story

Two changes, deliberately separate from each other.

First, the prompt. I added one explicit line:

Always write at least one sentence of explanation before the CITED line, even if you are declining to answer because the excerpts don't cover the question. Never respond with only the CITED line.

Re-ran the same question. Qwen now returns: "The provided excerpts do not contain any information about Vanna Winland's thoughts on quantum computing. Therefore, I cannot provide an answer based on the given information." Same shape as Ollama's original answer.

Second, independent of the prompt fix: a guard in the app itself. If the parsed answer ever comes back empty, for any reason, on any backend, the UI shows "No relevant information found in the archive for this question," instead of a blank section.

The prompt fix addresses why this specific model produced sparse output. The UI guard addresses what a visitor sees if some future model, or some prompt change I make later, does the same thing again in a way I haven't anticipated. One fixes a cause. The other makes the failure mode survivable regardless of cause.

What the full comparison actually shows now

All three backends, run against the same three test questions, side by side:

	Well-covered	Near-miss (quantum computing)	Cross-essay synthesis
Claude (claude-opus-5)	Full answer, 3 sources, quotes source text directly	Correctly declined, named what the excerpts did cover first	Full synthesis, 3 sources
Ollama (llama3.1, local, free)	Correct answer, 1 source	Correctly declined, one clean sentence	Correct synthesis, 2 sources
Hugging Face (Qwen/Qwen2.5-7B-Instruct, hosted, free)	Correct answer, 2 sources	Failed on first attempt (blank answer), passed after prompt fix	Correct synthesis, only 1 source cited

None of the three ever hallucinated a connection to quantum computing that isn't there. That's the bar that actually matters for a retrieval tool, and all three cleared it. "Didn't hallucinate" and "produced a response a user can actually read" turned out to be different bars, and the gap between them is where this bug lived.

Free models held up on correctness at this scale, on this task. Where they differed was synthesis depth, and, in one case, whether the model treated explaining itself as mandatory or optional under sparse conditions. That gap only surfaces by testing the exact question shape designed to expose it, then deploying somewhere that exposes it for real.

Try it yourself: ask-my-archive-gkuq5o8tdvnc7cs3wwpcth.streamlit.app. Ask it the quantum computing question if you want to watch the decline happen live.

Full comparison results and deployment debugging notes are in the GitHub repo, alongside everything from the first build.