Open source · MIT
Applied research record · reviewed 2026-10-09
Applied research, on the record.
ThynkQ is a small engineering studio, not a research lab. Alongside client work we build open-source tooling for AI evaluation and agent security, and we publish what we learn using it. This page separates what we have shipped from what we are still working on from what we have not started. Where prior work exists, we cite it.
TQ-R-01mcp-scan
TQ-R-02rag-eval-toolkit
01 / Index
Every entry, with its state.
Reviewed 2026-10-09. A row is dated only when the date can name what it was measured against, and an entry with nothing you can open says so rather than waiting until it does.
| Kind | Record | Status | As of | Artifacts |
|---|---|---|---|---|
| PublicationTQ-R-01Agent security | mcp-scan A static analysis scanner for MCP server configurations that reports what a config exposes before you run it. | Published | 2026-10-10latest capture, CLI v2.1.0 against a committed synthetic config | |
| PublicationTQ-R-02AI evaluation | rag-eval-toolkit A Python toolkit for evaluating RAG pipelines with LLM judges whose prompts are readable in the source. | Published | 2026-10-09the PyPI JSON API returned 200 for the project on the last record review | |
| StudyTQ-S-01AI evaluation | Are LLM judges reliable enough to be measurement instruments? Test-retest stability, agreement with human labels, and calibration error across judge models and prompt variants. | In design | 2026-07scoped July 2026; no data collected, so no run counts exist to print | none yet |
| StudyTQ-S-02AI evaluation | Do evaluation frameworks disagree on identical outputs? The same labeled examples scored through Ragas, DeepEval and rag-eval-toolkit, with specific disagreements traced back to prompt wording. | In design | 2026-07scoped July 2026; no data collected, so no disagreement rate exists to print | none yet |
| QuestionTQ-Q-01 | Does hallucination detection degrade in long contexts? | Not started | n/ano start date committed; it is not scheduled | none yet |
| QuestionTQ-Q-02 | Do faithfulness metrics transfer to regulated domains? | Not started | n/ano start date committed; it is not scheduled | none yet |
| QuestionTQ-Q-03 | How does the MCP ecosystem drift over time? | Not started | n/ano start date committed; it is not scheduled | none yet |
| QuestionTQ-Q-04 | Which models resist tool poisoning, and does hardening help? | Not started | n/ano start date committed; it is not scheduled | none yet |
02 / Status
Research status
Counts of the rows in the record above.
- 2published
- 2in design
- 4not started
- 6with no artifact yet
03 / Tools
Use it before you read about it.
The public tools behind this research. Free, no signup, no waitlist.
Start here
Sprint Fit Calculator
A five-question browser tool that turns a messy software problem into the right next step: product, audit, sprint, build, or Engineering Retainer.
New
MCP Risk Calculator
A five-question browser tool that routes an MCP rollout into the safest next step: free scan, hosted Pro preview, paid MCP Risk Review, or broader Build Readiness Audit.
04 / Published
Published and usable
Public source and a specific boundary on what each tool can prove.
Published · MIT
mcp-scan
Open-source security scanner for Model Context Protocol servers.
Checks
- tool poisoning
- prompt injection surfaces
- over-broad permissions
- unpinned server versions
What this does not do
- Static analysis flags patterns, not confirmed exploits. Some findings will be false positives. We have not yet published a precision and recall measurement for this scanner, including our own false-positive rate. That is listed below.
Published · MIT
rag-eval-toolkit
Python toolkit for evaluating RAG pipelines using LLM-as-judge.
Checks
- faithfulness
- hallucination
- groundedness
- relevancy
- context precision
- context recall
What this does not do
- We have not yet measured how well these judges agree with human raters. That study is listed below under In progress. Until it is published, treat the scores as a consistent signal, not a validated ground truth.
- LLM judges are known to show position, verbosity, and self-preference bias. We have not yet characterized how much of each affects these specific metrics.
How it works
Every judge prompt is plain text and readable in the source.
Runs locally via Ollama, so evaluation does not require sending data to a third-party API.
05 / Rule surface
What the scanner can raise.
66 rule ids, each with the false positives it can raise and the steps to clear it. One row per severity, one cell per id, and the ramp is the severity a rule fires at.
Share of the set, then every rule as one cell, then the noise each rung is known to produce
- Critical1015.2%34 documented false-positive cases
- High2436.4%85 documented false-positive cases
- Medium1725.8%63 documented false-positive cases
- Low913.6%35 documented false-positive cases
- Informational69.1%19 documented false-positive cases
data-flow-source-sink. Every id, with its false positives and fix steps.06 / Instrumentation
Every number names its source.
A figure on this page that cannot say how it was counted does not get printed.
Scanner modules, in the CLI's own order
- secret-scanner
- typosquat-scanner
- registry-scanner
- permission-scanner
- transport-scanner
- config-scanner
- prompt-injection-scanner
- env-leak-scanner
- ast-scanner
- package-scanner
- tool-poisoning-scanner
- supply-chain-scanner
- license-scanner
- network-egress-scanner
- data-flow-scanner
- data-controls-scanner
Published counts, with their basis
16scanner modules
mcp-scan scanners command output
17named finding types
mcp-scan README What It Detects table, 17 rows
17supported clients
client config paths wired in mcp-scan source
2.1.0current npm release
npm dist-tags.latest for mcp-scan, read at request time and revalidated hourly
07 / Capture
The output this is written against.
Not an illustration. This is the CLI's real output, on one synthetic config, in the page rather than a screenshot of it.
CRITICAL exposed-secret
Exposed GitHub Token in environment variable 'WEATHER_TOKEN'.
↳ Move the secret to a secure environment variable and reference it instead (e.g., ${WEATHER_TOKEN}).
MEDIUM unverified-source
Server 'weather-demo-server' is an unscoped package from an unverified source (typosquatting risk).
↳ Verify the publisher of 'weather-demo-server' on npmjs.com before trusting this server, or ask the maintainer to publish with npm provenance (npm publish --provenance from GitHub Actions or GitLab CI) so mcp-scan can confirm it automatically.
Scanned 1 server across 1 client in 494ms
1 critical 0 high 1 medium 0 low
# our read: 1 finding needs a human decision.
08 / In design
Studies in design
Questions we have committed to answering. Neither study has started collecting data.
Are LLM judges reliable enough to be measurement instruments for RAG evaluation?
Measure test-retest stability, agreement with human labels, and calibration error across judge models and prompt variants, including our own rag-eval-toolkit as one of the instruments under test. Compare locally-run open-weight judges against frontier API judges on cost per reliable verdict.
A repeated-measures design needs many thousands of judge calls, which is the main thing that decides how wide this study can go. We will publish the exact run counts and the cost alongside the results.
Do RAG evaluation frameworks disagree when scoring identical outputs, and can readable prompts explain why?
Score the same labeled examples through Ragas, DeepEval, and rag-eval-toolkit. Quantify disagreement, then trace specific disagreements back to prompt wording.
rag-eval-toolkit is published on PyPI. It runs in this comparison alongside Ragas and DeepEval, not as a claim that it beats them.
09 / Not started
Open questions
Not studied and not scheduled. No date is printed here because there is no date.
Does hallucination detection degrade in long contexts?
Detector accuracy reportedly drops as retrieval context grows. Production RAG routinely exceeds the context lengths most detectors were validated on.
Do faithfulness metrics transfer to regulated domains?
A metric tuned on general QA is unvalidated on clinical notes or pharma documentation. The data that would settle this is mostly not public.
How does the MCP ecosystem drift over time?
Published surveys of MCP server security are single snapshots. A snapshot cannot see a server that ships a malicious update after you installed it.
Which models resist tool poisoning, and does hardening help?
Existing work measures agent frameworks. Whether resistance varies by base model and system prompt is less well covered.
10 / Standard
A useful result includes its limits.
- We publish methodology before results.
- We cite prior work by name. Where a question has already been answered, we say so and build on it rather than around it.
- We publish numbers that do not flatter us, including our own tools' failure rates.
- We do not claim "first" or "novel" without a dated literature check.
- Client work under NDA stays under NDA. It informs what we find interesting. It is not evidence we offer you.
Cite this record
ThynkQ. (2026). Applied research record (reviewed 2026-10-09). https://thynkq.com/research
@misc{thynkq_research_20261009,
author = {{ThynkQ}},
title = {Applied research record},
howpublished = {\url{https://thynkq.com/research}},
note = {Reviewed 2026-10-09},
year = {2026}
}