Scorecard AI helps developers evaluate and improve AI agents and LLM applications by creating test datasets, defining evaluation metrics, running automated evaluations, analyzing scores, managing system versions, and reviewing annotations.
Hermes is Nous Research's command-line agent. MCP servers are declared in its YAML config and reloaded without a restart.
This guide connects Scorecard MCP to Hermes through MewCP, so Hermes can use Scorecard MCP's 2 tools, like execute and search_docs, without your Scorecard MCP credentials ever reaching the app.
Scorecard MCP signs in through managed OAuth. MewCP stores the token encrypted, refreshes it before it expires and injects it only at call time. Your agent never holds it, so it can never leak it.
More apps
Manage repositories, pull requests, issues, branches and development workflows on GitHub. Automate software delivery and collaborate across engineering projects.
TypeSafe's Jev model for typed decisions — ask yes/no, multiple-choice or rubric-score questions about a block of state and get calibrated probabilities back instead of prose. Built for classification, routing and scoring steps inside a workflow
Connect to external APIs, webhooks and internet services through standard HTTP requests. Build integrations with virtually any platform that exposes a web-based interface.
Turn your AI agent into a creative studio The world's best image, video, audio and 3D models, inside the AI assistant you already use.
Other clients
Scorecard MCP works the same way in every client below. Pick yours for its own guide.
Free to start.
Scope access to the exact Scorecard MCP tools and accounts you allow, work and personal side by side. Anything outside that scope is refused at call time, even when the model asks for it.
Each call is logged: what ran, on which account, and whether it worked. Failures come back structured, so the agent knows whether to retry instead of guessing.
Every app sits behind one secure MCP URL, and there is no server for you to deploy or keep running. Add an integration in the dashboard and it is live on the next call, with no config to change and no new credentials to wire up. Tools load only when a task needs them, so your agent stays fast and focused however many apps you connect.
Edit config.yaml in your Hermes config directory (~/.hermes).
Put this under the top-level "mcp_servers:" key.
Run /reload-mcp in Hermes, or start a fresh session. Sign in when the browser window opens.
Start a new chat and ask: "List the tools you can use from MewCP." You should see your connected apps.
Runs JavaScript code to interact with the Scorecard API. You are a skilled TypeScript programmer writing code to interface with the service. Define an async function named "run" that takes a single parameter of an initialized SDK client and it will be run. For example: ``` async function run(client) { const run = await client.runs.create('314', { metricIds: ['789', '101'], testsetId: '246' }); console.log(run.id); } ``` You will be returned anything that your function returns, plus the results of any console.log statements. Do not add try-catch blocks for single API calls. The tool will handle errors for you. Do not add comments unless necessary for generating better code. Code will run in a container, and cannot interact with the network outside of the given SDK client. Variables will not persist between calls, so make sure to return or log any data you might need later. Remember that you are writing TypeScript code, so you need to be careful with your types. Always type dynamic key-value stores explicitly as Record<string, YourValueType> instead of {}.
Search SDK documentation to find methods, parameters, and usage examples for interacting with the API. Use this before writing code when you need to discover the right approach.
