AuditAI:Agentic RAG Compliance Engine
A production agentic compliance engine using LangGraph and Corrective RAG (CRAG) to audit organizational policies against 4 major cybersecurity frameworks simultaneously: NIST CSF 2.0, NIST SP 800-53 Rev 5, ISO 27001:2022, and SOC 2 Trust Services Criteria.
Core Impact
“Audits policies against 4 frameworks (NIST CSF 2.0, SP 800-53, ISO 27001, SOC 2) simultaneously, achieving 96.7% RAGAS Faithfulness and 100% Context Recall via Corrective RAG with LLM-as-judge grading, cross-validated by an independent Claude Sonnet 5 judge within ~0.25 points on every dimension.”
4
Frameworks audited at the same time
96.7%
RAGAS faithfulness, 100% context recall
0
Phantom citations on low-confidence answers
±0.25
Max score gap between Gemini and Claude judges
The Problem
Compliance audits are slow, manual, and easy to get wrong. A single policy can satisfy NIST and quietly violate ISO, so someone has to read every policy against four separate frameworks by hand and hope they didn't miss anything. I wanted a system that checks a policy against all four at once, points to the exact page it based each answer on, and refuses to make things up when it isn't sure.
The Approach
Why corrective RAG
Plain RAG retrieves some chunks and hopes they're relevant. I added an LLM judge that grades every chunk first, and when the answer comes back weak, it rewrites the query in proper compliance language and tries again, up to three times before it gives up. That grading loop is what keeps the answers honest.
Why query all four frameworks in parallel
Each framework has its own index, and I query all of them at the same time through a ThreadPoolExecutor with grading batched through asyncio.gather. Worst-case latency is one LLM call no matter how many chunks come back, and every framework gets equal representation instead of whichever one happened to match first.
Why a semantic cache
A second Qdrant collection stores past questions by their embedding. Anything within 0.93 cosine similarity of an earlier question skips the entire graph and answers instantly at zero LLM cost, which matters when several people ask the same compliance question different ways.
Why page-level citations
Every claim links back to the framework name and the exact PDF page it came from. If the model isn't confident, six refusal-phrase patterns catch the low-confidence generation and it declines to answer instead of inventing a citation that looks real but isn't.
Why a second, cross-family judge
RAGAS grades answers with a Gemini judge, the same family that generates them, which risks the model flattering its own outputs. I added a separate Claude Sonnet 5 judge running the identical rubric. The two converged within 0.25 points on every dimension, and the Claude judge is what originally caught a verbosity defect the Gemini judge scored a perfect 5 on.
Technical Deep Dive
Engineered Corrective RAG pipeline using LangGraph state machine with LLM-as-judge document grading and autonomous query reformulation loop (up to 3 retries before fallback), auditing policies simultaneously against NIST CSF 2.0, NIST SP 800-53, ISO 27001:2022, and SOC 2.
Parallelized retrieval across all 4 framework indexes via ThreadPoolExecutor and document grading via asyncio.gather. Worst-case grading latency equals 1 LLM call regardless of chunk count; equal framework representation guaranteed.
Built semantic cache in second Qdrant Cloud collection at cosine similarity threshold 0.93. Near-duplicate queries skip the full graph, delivering near-instant response at zero LLM cost.
Engineered page-level citation system linking every answer to exact source document and page number from official PDFs. Zero phantom citations via 6-pattern refusal-phrase detection on low-confidence generations.
Built automated RAGAS evaluation harness across 33 adversarial questions spanning all 4 frameworks: Faithfulness 96.7%, Context Recall 100%, Context Precision 79.4%, Answer Relevancy 77.6%.
Added cross-family Claude Sonnet 5 LLM-as-judge alongside Gemini judge to rule out self-preference bias, scoring Correctness, Groundedness, Completeness, and Relevance; independent judges converged within ~0.25 across all 4 dimensions, confirming the RAGAS signal wasn't the generator flattering itself.
Systems Analysis Concluded