Distribution

Current channels

GitHub Releases

Pre-built binaries via GoReleaser on every v* tag. Cross-compiled for linux/darwin/windows, amd64/arm64.

go install github.com/blackwell-systems/mcp-assert/cmd/mcp-assert@latest

GitHub Action

blackwell-systems/mcp-assert-action@v1: one line in any workflow. Downloads binary, runs assertions, uploads JUnit XML + badge. Listed on GitHub Marketplace.

- uses: blackwell-systems/mcp-assert-action@v1
  with:
    suite: evals/

How it stays in sync: The action is a composite action in a separate repo. It downloads the mcp-assert binary from GitHub Releases at runtime (default: latest). New mcp-assert releases are picked up automatically without touching the action repo.

When to update the action repo: Only when the action's own interface changes (new inputs, new outputs, different install logic). Not for new mcp-assert features or bug fixes.

Versioning: Users pin to @v1 (floating major tag). When updating the action repo, tag the specific release (v1.0.1) and force-update the v1 tag:

cd mcp-assert-action
git tag v1.0.1 && git push origin v1.0.1
git tag -fa v1 -m "Update v1 to v1.0.1" && git push origin v1 --force

Marketplace listing: Updates automatically when a new tag is pushed to the action repo. GitHub reads action.yml from the tagged commit.

Homebrew

brew install blackwell-systems/tap/mcp-assert

Formula auto-generated by GoReleaser on each release and pushed to blackwell-systems/homebrew-tap.

curl | sh (macOS / Linux)

curl -fsSL https://raw.githubusercontent.com/blackwell-systems/mcp-assert/main/install.sh | sh

Detects OS and architecture, downloads the matching binary from GitHub Releases, installs to /usr/local/bin. Set INSTALL_DIR to override.

npm

npx @blackwell-systems/mcp-assert

Uses the platform-specific optional dependency pattern (same as esbuild, turbo). npm resolves the correct binary for the user's OS/arch automatically. No Go toolchain required. Published via scripts/npm-publish.sh on each release tag.

PyPI

pip install mcp-assert

Platform-specific wheels containing the Go binary. Each wheel is tagged with the correct platform (e.g. macosx_11_0_arm64, manylinux2014_x86_64), so pip resolves the right one automatically. Built via scripts/pypi-build-wheels.sh and published with twine on each release tag.

Scoop (Windows)

scoop bucket add blackwell-systems https://github.com/blackwell-systems/scoop-bucket
scoop install mcp-assert

Manifest auto-generated by GoReleaser on each release and pushed to blackwell-systems/scoop-bucket.

Docker

docker run blackwellsystems/mcp-assert audit --server "npx my-server"

Multi-arch images (linux/amd64, linux/arm64) published to both Docker Hub and GHCR on each release:

• blackwellsystems/mcp-assert:latest / blackwellsystems/mcp-assert:<version>
• ghcr.io/blackwell-systems/mcp-assert:latest / ghcr.io/blackwell-systems/mcp-assert:<version>

Built via docker/build-push-action@v6 in the docker job of release.yml. Uses a two-stage Dockerfile: Go build on golang:1.25-alpine, final image on alpine:3.21 with just the binary and CA certificates.

Winget (Windows)

winget install BlackwellSystems.mcp-assert

Manifest submitted to microsoft/winget-pkgs. Auto-updated on each release by the winget job in release.yml using winget-releaser. Uses InstallerType: zip with NestedInstallerType: portable (same pattern as agent-lsp).

Maintenance: the winget-releaser action auto-submits a PR to winget-pkgs on every v* tag. Uses installers-regex: 'windows.*\.zip$' to match the GoReleaser Windows zip assets. The HOMEBREW_TAP_TOKEN secret is reused for the winget fork (it needs public_repo scope).

pytest Plugin

pip install pytest-mcp-assert
pytest --mcp-suite evals/

Each YAML assertion becomes a pytest test item with pass/fail/skip semantics. Configurable via pyproject.toml (mcp_suite, mcp_fixture, mcp_timeout). The Go binary handles all MCP logic; the plugin is a 170-line bridge.

Source: pytest-plugin/ directory in this repo. Published to PyPI automatically by the pytest-plugin-publish job in release.yml on each v* tag.

Maintenance surface: The plugin is a thin bridge (~170 lines in plugin.py). It calls the mcp-assert Go binary with --json and maps the JSON output to pytest Items. The version is stamped from the git tag at publish time (sed in the release workflow). Breakage scenarios: (1) the --json output format changes in the Go binary, (2) the Go binary isn't on PATH when pytest collects, (3) PyPI token expires. The plugin has no dependencies beyond pytest itself.

Vitest Plugin

npm install -D vitest-mcp-assert @blackwell-systems/mcp-assert

Two usage modes. Auto-discovery:

// mcp.test.ts
import { describeMcpSuite } from 'vitest-mcp-assert'
describeMcpSuite('mcp server', 'evals/')

Or per-test control:

import { test } from 'vitest'
import { runMcpAssert } from 'vitest-mcp-assert'
test('echo tool', () => runMcpAssert('evals/echo.yaml'))

Each YAML assertion becomes a Vitest test with the same pass/fail semantics. Same YAML files work across all five test framework plugins and the CLI.

Source: vitest-plugin/ directory in this repo. Published to npm as vitest-mcp-assert.

Maintenance surface: The plugin is a thin bridge (~150 lines across 5 TypeScript files). It calls the mcp-assert Go binary with --json and maps the JSON output to Vitest test outcomes. Same architecture as the pytest plugin. Breakage scenarios: (1) the --json output format changes in the Go binary, (2) the Go binary isn't found via the 3-tier resolution (explicit, PATH, npm package). The plugin has no runtime dependencies beyond vitest as a peer dependency.

Go Test Plugin

go get github.com/blackwell-systems/mcp-assert/go-plugin

func TestMCPServer(t *testing.T) {
    mcpassert.Suite(t, "evals/")
}

Each YAML file becomes a t.Run subtest. Same bridge architecture: shells out to the mcp-assert binary, parses JSON, maps to t.Error/t.Skip/pass.

Source: go-plugin/ directory in this repo. Published as a Go module at github.com/blackwell-systems/mcp-assert/go-plugin.

Jest Plugin

npm install -D jest-mcp-assert @blackwell-systems/mcp-assert

Two usage modes. Auto-discovery:

// mcp.test.ts
import { describeMcpSuite } from 'jest-mcp-assert'
describeMcpSuite('mcp server', 'evals/')

Or per-test control:

import { runMcpAssert } from 'jest-mcp-assert'
test('echo tool', () => { runMcpAssert('evals/echo.yaml') })

Same YAML files work across all five test framework plugins and the CLI.

Source: jest-plugin/ directory in this repo. Published to npm as jest-mcp-assert.

Maintenance surface: Same thin bridge architecture as the vitest and pytest plugins (~100 lines). Calls the mcp-assert Go binary with --json and maps the JSON output to Jest test outcomes. Uses CommonJS module format (Jest's default).

Bun Plugin

bun add -d bun-mcp-assert @blackwell-systems/mcp-assert

// mcp.test.ts
import { describeMcpSuite } from "bun-mcp-assert"
describeMcpSuite("mcp server", "evals/")

Same bridge architecture. Uses native Bun APIs (Bun.spawnSync, Bun.which). Ships as TypeScript source (no build step; Bun runs it directly).

Source: bun-plugin/ directory in this repo. Published to npm as bun-mcp-assert.

PHPUnit Plugin

composer require --dev blackwell-systems/phpunit-mcp-assert

use BlackwellSystems\McpAssert\McpAssertRunner;

public function testEchoTool(): void
{
    McpAssertRunner::assertYaml('evals/echo.yaml');
}

Same bridge architecture. Published to Packagist as blackwell-systems/phpunit-mcp-assert. Separate repo required (Packagist reads composer.json from repo root).

"Works with mcp-assert" Badge

Static and dynamic (CI-verified) badge for MCP server READMEs. Every badge is a backlink. See the Badge guide.

Awesome Lists

List	Stars	Status	Section
punkpeye/awesome-mcp-devtools	448	PR #144 open	Testing Tools
appcypher/awesome-mcp-servers	5.5K	Branch pushed, needs PR	Development Tools
wong2/awesome-mcp-servers	4K	Not submitted	Community Servers
TheJambo/awesome-testing	2.2K	Not submitted	AI & LLM Testing
ai-boost/awesome-harness-engineering	705	Not submitted	Verification & CI
ai-for-developers/awesome-ai-coding-tools	1.7K	Not submitted	Testing and QA
rohitg00/awesome-devops-mcp-servers	979	Not submitted	Testing & Chaos Engineering
avelino/awesome-go	172K	Blocked until Sep 2026 (5-month history req)	Testing Frameworks

Snap (Linux)

sudo snap install mcp-assert --classic

Published to the Snap Store on each release. Uses classic confinement since mcp-assert needs to launch arbitrary server processes. Built via snapcore/action-build and published via snapcore/action-publish in the snap job of release.yml.

Planned

Channel	Priority	Description
Nix flake / nixpkgs	Medium	Nix users are quality-focused and vocal. High signal in a niche community.
AUR	Low	Arch Linux users. PKGBUILD file.
Chocolatey	Low	Windows, tracks download counts.
MCP registry integration	Low	Test badge on Glama and Smithery listings.

How to release

A single git tag triggers the entire pipeline. Every channel is automated.

Step-by-step

1. Update CHANGELOG.md: move items from [Unreleased] to a new version header (e.g., ## [0.5.0] - 2026-05-01).

2. Tag and push: bash git tag v0.5.0 git push origin v0.5.0

3. The release.yml workflow does the rest (triggered by the v* tag):

Job	What it does	Channel updated
release (GoReleaser)	Builds binaries for 6 platforms, creates GitHub Release, pushes Homebrew formula to homebrew-tap, pushes Scoop manifest to scoop-bucket	GitHub Releases, Homebrew, Scoop
docker	Builds multi-arch Docker images (amd64, arm64), pushes to Docker Hub and GHCR	Docker Hub, GHCR
npm-publish	Runs scripts/npm-publish.sh, downloads binaries from the Release, publishes platform packages to npm	npm
pypi-publish	Runs scripts/pypi-publish.sh, builds platform wheels with Go binary inside, publishes to PyPI with twine	PyPI
pytest-plugin-publish	Builds and publishes the pytest-mcp-assert plugin to PyPI with twine	PyPI (pytest plugin)
snap	Builds snap package with snapcore/action-build, publishes to Snap Store	Snap Store
jest-plugin-publish	Builds and publishes the jest-mcp-assert plugin to npm	npm (jest plugin)
bun-plugin-publish	Publishes the bun-mcp-assert plugin to npm	npm (bun plugin)
vitest-plugin-publish	Builds and publishes the vitest-mcp-assert plugin to npm	npm (vitest plugin)
winget	Submits manifest PR to microsoft/winget-pkgs via winget-releaser	Winget

1. Verify (after CI completes, ~5-10 minutes): ```bash # GitHub Release exists gh release view v0.5.0

# Homebrew updated brew update && brew info blackwell-systems/tap/mcp-assert

# npm packages updated npm view @blackwell-systems/mcp-assert version npm view vitest-mcp-assert version npm view jest-mcp-assert version npm view bun-mcp-assert version

# PyPI updated pip index versions mcp-assert pip index versions pytest-mcp-assert

# Docker images docker pull blackwellsystems/mcp-assert:latest --quiet && echo "Docker Hub OK" docker pull ghcr.io/blackwell-systems/mcp-assert:latest --quiet && echo "GHCR OK" ```

What is NOT automatic

Channel	How to update
GitHub Action	Only update when the action's interface changes. See the GitHub Action section above.
install.sh / install.ps1	These fetch latest from GitHub Releases. No update needed unless the script logic changes.
Badges	Static SVGs in assets/. Update manually if the badge design changes.
Awesome MCP DevTools listing	Manual PR to the upstream repo.
Go plugin (mcpassert)	Go modules are published automatically when the tag is pushed. No workflow job needed.
Download stats badge	Auto-updated hourly by the stats.yml workflow. No manual action.

Secrets required

Secret	Where	Purpose
GITHUB_TOKEN	Automatic (GitHub provides)	GoReleaser, GitHub Release creation, GHCR push
NPM_TOKEN	Repository secret	npm publish
PYPI_TOKEN	Repository secret	twine upload to PyPI
DOCKERHUB_USERNAME	Repository secret	Docker Hub push
DOCKERHUB_TOKEN	Repository secret	Docker Hub push
HOMEBREW_TAP_TOKEN	Repository secret	Homebrew formula push, winget fork
SNAP_TOKEN	Repository secret	Snap Store publish

Rollback

If a release has a critical bug: delete the GitHub Release and tag, then publish a patch.

gh release delete v0.5.0 --yes
git push origin :refs/tags/v0.5.0
# Fix the bug, then tag v0.5.1

npm, PyPI, and Docker Hub don't support deletion (npm has a 72-hour unpublish window). Publish a patch version instead. The Snap Store token has not been configured, so the snap job will fail silently until SNAP_TOKEN is set.

The scan-and-contribute flywheel

mcp-assert spreads through the bugs it finds, not through marketing. Every issue filed on a popular MCP server is passive promotion with built-in credibility.

Scan server -> Find bugs -> File issue -> Link mcp-assert -> Maintainers discover tool -> Adopt

Server suites shipped

Server	Language	Assertions	Coverage	Notes
agent-lsp + gopls	Go	63	100% (50/50 tools)	Internal dogfooding server
agent-lsp skill protocols	N/A	20	20/20 skills	Trajectory assertions
@modelcontextprotocol/server-filesystem	TypeScript	14	92%	Read, list, search, info, write, edit, create dir, move, directory tree, path traversal rejection
@modelcontextprotocol/server-memory	TypeScript	9	100% (9/9 tools)	Clean scan
mcp-server-time	Python	5	100% (2/2 tools)	Clean. UTC, named timezone, conversion, invalid timezone rejection.
mcp-server-fetch	Python	3	100% (1/1 tool)	Clean. URL fetch, invalid URL, unreachable host.
mcp-server-git	Python	11	92% (11/12 tools)	Clean. Status, log, branch, diff, show, invalid repo/ref rejection.
mcp-server-sqlite	Python	9	100% (6/6 tools)	Clean scan
mark3labs/mcp-go everything	Go	9	100%	stdio
mark3labs/mcp-go everything	Go	5	100%	HTTP transport conformance
mark3labs/mcp-go everything (prompts)	Go	4	100%	prompts/list, prompts/get
mark3labs/mcp-go everything (resources)	Go	4	100%	resources/list, read, subscribe
mark3labs/mcp-go everything (completion)	Go	3	--	completion/complete
mark3labs/mcp-go everything (logging)	Go	2	--	logging/setLevel, message capture
mark3labs/mcp-go typed_tools	Go	3	100%	Clean
mark3labs/mcp-go structured	Go	6	100%	Clean
mark3labs/mcp-go roots_server	Go	1	100%	Bidirectional roots/list
mark3labs/mcp-go sampling_server	Go	3	100%	Bidirectional sampling
mark3labs/mcp-go elicitation	Go	4	100%	Accept, decline, cancel, validation
PrefectHQ/fastmcp testing_demo	Python	16	100%	All 3 MCP feature categories
PrefectHQ/fastmcp testing_demo (SSE)	Python	11	100%	SSE transport verification
github/github-mcp-server	Go	20	--	17 read-only tools across 7 toolsets: context, repos, git, issues, pull requests, users, gists
4t145/rmcp counter	Rust	14	100% (6/6 tools + resources + prompts)	First Rust server. Bug found: get_value mutates state.
rust-mcp-stack/rust-mcp-filesystem	Rust	23	92% (22/24 tools)	Clean scan. Read, list, search, write, edit, zip/unzip, path traversal.
haris-musa/excel-mcp-server	Python	15	52% (13/25 tools)	Clean. Workbook, sheets, data, formulas, charts, pivots, formatting.
antvis/mcp-server-chart	TypeScript	25	93% (25/27 tools)	9 bugs found (#291). Unhandled exceptions on default input.
@modelcontextprotocol/server-everything	TypeScript	13	92% (12/13 tools)	Clean. Official Anthropic reference server.
hashicorp/terraform-mcp-server	Go	5	56% (5/9 tools)	Clean. Provider, module, policy search.
makenotion/notion-mcp-server	TypeScript	22	100% (22/22 tools)	Clean. Official Notion server.
jamesward/hello-spring-mcp-server	Kotlin	3	100% (2/2 tools)	Clean. First JVM server. Spring AI MCP, HTTP transport.
mongodb/mongodb-mcp-server	TypeScript	4	--	Clean. Knowledge search, error handling.
microsoft/playwright-mcp	TypeScript	14	67% (14/21 tools)	Clean. Navigate, snapshot, screenshot, JS evaluate, console, network, resize, close, tabs, navigate back, press key, wait for element.
openai/sample-deep-research-mcp	Python	4	100% (2/2 tools)	Clean. Search and fetch against static JSON.
@google-cloud/storage-mcp	TypeScript	6	35% (6/17 tools)	Clean. Bucket metadata, object listing, IAM policy.
grafana/mcp-grafana	Go	54	100% (50/50 tools)	1 bug (fixed). 10 live-backend assertions use skip_unless_env.
blazickjp/arxiv-mcp-server	Python	5	50% (5/10 tools)	1 bug: isError flag not set on error content.
awslabs/aws-documentation-mcp-server	Python	4	100% (4/4 tools)	Clean. Search, recommend, no-results handling.
exa-labs/exa-mcp-server	JavaScript	2	100% (2/2 tools)	Clean. Proper 401 with API key guidance.
onmyway133/git-mcp	TypeScript	14	39% (14/36 tools)	Clean. Status, log, branches, diff, show, reflog.
nicobailey/codegraph-context-mcp	TypeScript	16	76% (16/21 tools)	Clean. Code graph indexer.
perplexityai/mcp-server	TypeScript	4	100% (4/4 tools)	Clean. All tools return isError with 401.
Gentleman-Programming/engram	Go	16	100% (16/16 tools)	Clean. Full coverage including writes.
steipete/Peekaboo	Swift	6	27% (6/22 tools)	1 bug. First Swift server.
@modelcontextprotocol/server-puppeteer	TypeScript	7	100% (7/7 tools)	1 bug: puppeteer_navigate crashes on invalid URL.
getsentry/XcodeBuildMCP	TypeScript	10	37% (10/27 tools)	Clean. Sentry-backed.
chrome-devtools-mcp	TypeScript	7	24% (7/29 tools)	Clean.
mozilla/firefox-devtools-mcp	TypeScript	7	24% (7/29 tools)	Clean. Mozilla-backed.
@upstash/context7-mcp	TypeScript	2	100% (2/2 tools)	Clean. Upstash-backed.
modelcontextprotocol/csharp-sdk	C#	2	100% (2/2 tools)	Clean. 7th language.
duckduckgo-mcp-server	Python	2	100% (2/2 tools)	Clean. Zero auth.
excalidraw-architect-mcp	Python	4	100% (4/4 tools)	Clean. Zero auth.
mcp-server-kubernetes	Python	2	40% (2/5 tools)	Clean.
lighthouse-mcp-server	TypeScript	2	4% (2/57 tools)	Clean. Tencent Cloud.
markitdown-mcp	Python	1	100% (1/1 tool)	Clean. Microsoft MarkItDown.
mcp-server-math	Python	4	25% (4/16 tools)	Clean. Zero auth.
mobile-next/mobile-mcp	TypeScript	6	29% (6/21 tools)	Clean.
sec-edgar-mcp	Python	5	24% (5/21 tools)	Clean. Uses skip_unless_env.
Pimzino/spec-workflow-mcp	TypeScript	1	100% (1/1 tool)	Clean.
narumiruna/yfinance-mcp	Python	4	22% (4/18 tools)	Clean. Zero auth.
Total	7 languages	570		61 suites

Target servers (next)

Server	Language	Stars	Why target
Rust MCP servers (rmcp SDK)	Rust	--	Underserved community, no existing testing tools
Java MCP servers (Spring, Quarkus)	Java	--	Enterprise community, high signal

Content (planned)

Channel	Status	Description
r/MCP	Posted 2026-04-24	Launch post: feature overview, scorecard, install methods
r/ClaudeCode	Posted 2026-04-24	Adapted for Claude Code/Desktop users
LinkedIn	Posted 2026-04-24	Launch announcement to professional network
Blog post: dogfooding	Planned	"We tested the most popular MCP server and found 4 DX issues in our own tool"
MCP community Discord/forums	Not started	Post when awesome-mcp-devtools listing is merged
Hacker News	Not started	Ready to post

Non-goals

• Paid tier. mcp-assert is free and open source. The value is ecosystem positioning, not revenue.
• SaaS dashboard. No hosted version. CI-native, single binary.
• LLM-as-judge features. Stay in our lane. Deterministic assertions only.