Case study 06 / AI Engineering · Software Engineering
Retrieval with an authorization boundary.
An internal knowledge assistant with department-scoped retrieval, hybrid search, streaming responses, and evaluation.
The problem
A shared knowledge assistant must answer useful questions without retrieving documents the current user is not allowed to read.
What I built
Built a RAG application with JWT authentication, Redis token revocation and conversation memory, department-scoped retrieval, BM25/vector search, cross-encoder reranking, and streaming responses.
How it works
Requests pass through authentication, per-user rate limits, and input checks. Retrieval filters accessible namespaces before reranking. Redis stores conversation state and revoked tokens. Output handling includes PII redaction; optional Llama Guard adds another safety check.
- Authenticated request
- Scoped hybrid retrieval
- Rerank & generate
- Output handling
Engineering decisions
Authorize before generation
Department access is enforced at retrieval rather than relying on instructions to the model or a hidden UI control.
Measure the full question set
The 25-question evaluation scored 0.9116 faithfulness. The 50-question run scored 0.7423 with a different judge model. Both belong in the results.
Separate evaluation from tests
Automated tests cover retrieval, Redis, re-ingestion, and guardrails. They are distinct from the question sets used to evaluate answer quality.
Results & evidence
50-question faithfulness
0.7423
Full recorded evaluation using llama-3.1-8b-instant for scoring; answer relevancy 0.6495.
25-question faithfulness
0.9116
Subset evaluated using llama-3.3-70b-versatile; answer relevancy 0.7346. Different samples and judges prevent a direct comparison.
Results are documented in the linked project artifacts. They have not been independently reproduced for this portfolio.
Limits & lessons
Model-judged scores depend on the question set and judge. Pattern-based guardrails and optional model checks do not establish comprehensive security or production readiness.
The authorization boundary and evaluation conditions matter more than a single headline score.