Last reviewed 2026-07-27
ThynkQ is a small engineering studio, not a research lab. Alongside client work we build open-source tooling for AI evaluation and agent security, and we publish what we learn using it. This page separates what we have shipped from what we are still working on from what we have not started. Where prior work exists, we cite it.
Tools in production use, with public source.
Open-source security scanner for Model Context Protocol servers.
Complete and tested. Not published, so treat it as a claim you cannot verify yet.
Python toolkit for evaluating RAG pipelines using LLM-as-judge.
Every judge prompt is plain text and readable in the source. Existing tools score through prompts you cannot inspect. Ours are in the repo.
Runs locally via Ollama, so evaluation does not require sending data to a third-party API.
Not yet published. No public link to verify against.
Questions we have committed to answering. Each carries a dated status and a named deliverable, so you can check whether we shipped it.
Measure test-retest stability, agreement with human labels, and calibration error across judge models and prompt variants, including our own unreleased rag-eval-toolkit as one of the instruments under test. Compare locally-run open-weight judges against frontier API judges on cost per reliable verdict.
A repeated-measures design needs many thousands of judge calls, which is the main thing that decides how wide this study can go. We will publish the exact run counts and the cost alongside the results.
Score the same labeled examples through Ragas, DeepEval, and rag-eval-toolkit. Quantify disagreement, then trace specific disagreements back to prompt wording.
rag-eval-toolkit is unreleased. It runs in this comparison as an internal instrument, not as a tool the reader can install today.
Questions we think matter and are not working on yet. Listed so you know what we know we do not know.
Does hallucination detection degrade in long contexts?
Detector accuracy reportedly drops as retrieval context grows. Production RAG routinely exceeds the context lengths most detectors were validated on.
Do faithfulness metrics transfer to regulated domains?
A metric tuned on general QA is unvalidated on clinical notes or pharma documentation. The data that would settle this is mostly not public.
How does the MCP ecosystem drift over time?
Published surveys of MCP server security are single snapshots. A snapshot cannot see a server that ships a malicious update after you installed it.
Which models resist tool poisoning, and does hardening help?
Existing work measures agent frameworks. Whether resistance varies by base model and system prompt is less well covered.