Build Log 3: Scaling the Archive, Then Trying to Break It

The first two build logs got the core working and deployed. This one covers what came next: expanding the archive as far as I could find material, then finding out whether it was actually safe to leave running in public.

A note on process: Claude Code did the implementation, but I directed it — architecture, the metadata schema, chunking strategy, local embeddings over a hosted API, and the debugging calls throughout, including overriding a mislabeled taxonomy tag because I'd written the piece myself.

Expanding the corpus

The archive started with 16 blog posts. I'd also published 27 pieces on IBM Think — 14 explainers, 12 tutorials, and one more I only found by digging through my author page. Indexing them meant two new loaders solving problems the blog loader never had to.

My portfolio site lists every IBM Think piece, but the pages are a client-rendered React SPA with no server-side HTML to scrape. Instead of standing up a headless browser, the loader parses the plain arrays straight out of my portfolio's source files, which it already clones for the blog corpus. Always-current, one less moving part.

Metadata — content type, published date, byline — gets pulled from the live page rather than assumed from where I'd filed it. That distinction mattered once: the loader was told to trust a page's own content-type label over my categorization, a reasonable default that was wrong exactly once. One piece I wrote as an explainer carried an external tag marking it a tutorial. The loader followed its instruction correctly and got it wrong anyway, because the instruction's premise didn't hold on that page. Claude flagged the mismatch while walking me through the output. If I'd skimmed past it, a wrong tag would have gone straight into the metadata. I overrode it explicitly, documented inline — and it's a reminder that reviewing model output as it comes, not just the final diff, is where catches like this actually happen.

Co-authorship was more common than I remembered. I'd flagged two pieces going in as possibly co-authored; actually checking every byline turned up ten, including one credited to three of us.

The 12 tutorials double as write-ups of code in IBM's public tutorials repo, and my tutorial cards link straight to the exact file in my fork. So instead of fuzzy-matching by title, the loader verifies the path exists upstream and pulls body text from there. Some source files are Jupyter notebooks mixing prose, code, and run output; the loader keeps the writing and code, drops the output — a stack trace is a log, not content. For tutorials whose link points at a folder instead of a file, it picks the real write-up out by matching filename to folder name.

Result: 43 documents, 1,749 chunks. I re-ran my three standing test questions against the expanded corpus (a well-covered question, a quantum-computing near-miss to check the system admits what it doesn't know, and a two-essay question), then added two new ones for the IBM Think content. All five passed. One is worth flagging: the LlamaIndex question retrieved exactly the right chunks, but the model reading them misread a footnote citation as an authorship credit. Retrieval and generation are two different failure points — worth being precise about which one actually broke.

Triaging an automated security review

GitHub Copilot's security review came back with one "critical" finding, four "high priority," and a nine-step remediation plan. I checked each against the actual repo before touching anything.

The "critical" finding didn't hold up. Copilot flagged my local .env file as a compromised secret. I checked the full commit history — never committed, on any branch, ever. The scanner flagged a plaintext file on disk without checking whether it had ever left the machine. (A scanner reading the file to analyze it is its own small version of that question, so I rotated the keys anyway — cheap insurance.)

One finding was accurate but aimed at the wrong deployment. The devcontainer config disables CORS and XSRF — true, but that only applies inside a GitHub Codespaces session, not the live app, which runs on Streamlit's own hosting. Codespaces commonly needs those checks off because its networking breaks otherwise. I left it and added a comment explaining the scope, so nobody mistakes a dev convenience for a production setting.

Two findings were real but assumed a different kind of system. No auth, no rate limiting — true, and an open endpoint can genuinely be flooded, scraped, or hit with expensive queries. But it's a public form by design, on a free backend chosen so usage spikes can't cause billing risk, and there's nothing sensitive behind it. The realistic failure mode is a slow or unavailable demo, not a data breach — a tradeoff I made deliberately rather than a risk I missed. Retrieved text going straight into the prompt as "untrusted input" — a legitimate concern for a RAG system ingesting outside content. Every document in this corpus is first-party, written by me, which narrows that threat model considerably. It doesn't erase it outright, though: the pipeline still pulls from live external pages and a GitHub repo, and trusts that content hasn't changed underneath it since I wrote it.

One finding was correct and cheap to fix. Dependencies weren't pinned to exact versions. Fixed — locked all eight direct dependencies to the versions already tested.

I fed my corrected assessment back to Copilot; it agreed and downgraded its own recommendation to a shorter checklist. Two items were worth doing. Pinning was already done. I ran pip-audit against the pinned versions and it found one real, named vulnerability in the vector database library — a code-injection bug, but only when the library runs as a network server with trust_remote_code enabled. This project runs it embedded in-process, and that setting never appears in the code anywhere. A real vulnerability in a mode this app never turns on.

Net result: two real fixes (pinned dependencies, a documentation comment), everything else confirmed either fine or built for a threat this app doesn't face. Every finding in the original report was technically accurate; almost none of it was urgent. Severity depends on context a scanner doesn't have — what's actually deployed, who could realistically attack it, what the data contains. Same treatment as everything else in this project: claims to check, not a checklist to execute on faith.