Applied research record · reviewed 2026-09-05

Research alongside delivery.

ThynkQ is a small engineering studio, not a research lab. Alongside client work we build open-source tooling for AI evaluation and agent security, and we publish what we learn using it. This page separates what we have shipped from what we are still working on from what we have not started. Where prior work exists, we cite it.

Use the tools See the published work

Tools

Use it before you read about it.

The public tools behind this research. Free, no signup, no waitlist.

Start here

Sprint Fit Calculator

A five-question browser tool that turns a messy software problem into the right next step: product, audit, sprint, build, or CTO lane.

Run calculator

New

MCP Risk Calculator

A five-question browser tool that routes an MCP rollout into the safest next step: free scan, hosted Pro preview, paid 48-hour review, or broader Build Readiness Audit.

Run MCP calculator

01

Published and usable

Public source and a specific boundary on what each tool can prove.

Published · MIT

mcp-scan

npm, with live download stats Source

Open-source security scanner for Model Context Protocol servers.

Checks

  • tool poisoning
  • prompt injection surfaces
  • over-broad permissions
  • unpinned server versions

What this does not do

  • Static analysis flags patterns, not confirmed exploits. Some findings will be false positives. We have not yet published a precision and recall measurement for this scanner, including our own false-positive rate. That is listed below.

Published · MIT

rag-eval-toolkit

PyPI

Python toolkit for evaluating RAG pipelines using LLM-as-judge.

Checks

  • faithfulness
  • hallucination
  • groundedness
  • relevancy
  • context precision
  • context recall

What this does not do

  • We have not yet measured how well these judges agree with human raters. That study is listed below under In progress. Until it is published, treat the scores as a consistent signal, not a validated ground truth.
  • LLM judges are known to show position, verbosity, and self-preference bias. We have not yet characterized how much of each affects these specific metrics.

How it works

Every judge prompt is plain text and readable in the source.

Runs locally via Ollama, so evaluation does not require sending data to a third-party API.

02

Studies in design

Questions we have committed to answering. Neither study has started collecting data.

Experiment design, July 2026. No data collected yet.

Are LLM judges reliable enough to be measurement instruments for RAG evaluation?

Measure test-retest stability, agreement with human labels, and calibration error across judge models and prompt variants, including our own rag-eval-toolkit as one of the instruments under test. Compare locally-run open-weight judges against frontier API judges on cost per reliable verdict.

Datasets
RAGTruth, HaluEval, LLM-AggreFact, ExpertQA (all public)
Deliverable
Judge scorecard with published methodology and raw results

A repeated-measures design needs many thousands of judge calls, which is the main thing that decides how wide this study can go. We will publish the exact run counts and the cost alongside the results.

Experiment design, July 2026. No data collected yet.

Do RAG evaluation frameworks disagree when scoring identical outputs, and can readable prompts explain why?

Score the same labeled examples through Ragas, DeepEval, and rag-eval-toolkit. Quantify disagreement, then trace specific disagreements back to prompt wording.

Datasets
RAGTruth, WikiEval
Deliverable
Framework agreement matrix with prompt-level root causes

rag-eval-toolkit is published on PyPI. It runs in this comparison alongside Ragas and DeepEval, not as a claim that it beats them.

03

Open questions

Areas we think matter but are not yet studying.

  1. 01

    Does hallucination detection degrade in long contexts?

    Detector accuracy reportedly drops as retrieval context grows. Production RAG routinely exceeds the context lengths most detectors were validated on.

  2. 02

    Do faithfulness metrics transfer to regulated domains?

    A metric tuned on general QA is unvalidated on clinical notes or pharma documentation. The data that would settle this is mostly not public.

  3. 03

    How does the MCP ecosystem drift over time?

    Published surveys of MCP server security are single snapshots. A snapshot cannot see a server that ships a malicious update after you installed it.

  4. 04

    Which models resist tool poisoning, and does hardening help?

    Existing work measures agent frameworks. Whether resistance varies by base model and system prompt is less well covered.

Operating standard

A useful result includes its limits.

  • We publish methodology before results.
  • We cite prior work by name. Where a question has already been answered, we say so and build on it rather than around it.
  • We publish numbers that do not flatter us, including our own tools' failure rates.
  • We do not claim "first" or "novel" without a dated literature check.
  • Client work under NDA stays under NDA. It informs what we find interesting. It is not evidence we offer you.