AskUmer

A founder or recruiter gives a portfolio under a minute. A chatbot that answers from a vague memory of the site, or makes things up, does more harm than no chatbot. Plain RAG also breaks on multi-hop questions and ships with no safety gate and no way to tell when a change made it worse.

• Hybrid retrieval: Qdrant dense search (BAAI/bge-base-en-v1.5, 768-dim) and BM25 sparse search, fused with Reciprocal Rank Fusion and reranked by a FlashRank cross-encoder down to the top five documents.
• A LangGraph ReAct agent decides on every turn whether to search at all, so a follow-up answerable from the conversation costs zero database calls.
• NeMo Guardrails runs as an input gate: jailbreaks, prompt injection and off-topic questions are blocked before any retrieval or model call.
• A Portkey gateway handles failover from gpt-oss-120b to gpt-oss-20b, with three retries on 429 and 503 and response caching in front.

• 898 indexed chunks.
• A two-phase DeepEval suite generates live answers, then scores Faithfulness and Answer Relevancy with an LLM judge. CI exits non-zero if any sample falls below 0.7.
• Every request is traced in Logfire, LangSmith and Portkey.
• It is the assistant answering questions on this site.

• Decide what worse means before you build: pick the metrics and the threshold, and make CI enforce them.
• Put guardrails before retrieval, not after generation. It is cheaper and stops injection earlier.
• Let the agent skip retrieval when the answer is already in the conversation.