Scorecard AI helps developers evaluate and improve AI agents and LLM applications by creating test datasets, defining evaluation metrics, running automated evaluations, analyzing scores, managing system versions, and reviewing annotations.
VS Code
Encrypted at rest, isolated from the model
Resolved from an AES-256-GCM vault at the moment of the call and attached to the request — the model never sees the secrets.
Try asking
Runs JavaScript code to interact with the Scorecard API. You are a skilled TypeScript programmer writing code to interface with the service. Define an async function named "run" that takes a single parameter of an initialized SDK client and it will be run. For example: ``` async function run(client) { const run = await client.runs.create('314', { metricIds: ['789', '101'], testsetId: '246' }); console.log(run.id); } ``` You will be returned anything that your function returns, plus the results of any console.log statements. Do not add try-catch blocks for single API calls. The tool will handle errors for you. Do not add comments unless necessary for generating better code. Code will run in a container, and cannot interact with the network outside of the given SDK client. Variables will not persist between calls, so make sure to return or log any data you might need later. Remember that you are writing TypeScript code, so you need to be careful with your types. Always type dynamic key-value stores explicitly as Record<string, YourValueType> instead of {}.
Search SDK documentation to find methods, parameters, and usage examples for interacting with the API. Use this before writing code when you need to discover the right approach.
One endpoint, the same key, whichever client you use.
~/Library/Application Support/Claude/claude_desktop_config.json (Mac) · %APPDATA%\Claude\claude_desktop_config.json (Windows)
Replace API_KEY with your own key.
Already have an "mcpServers" section in your config? Just add the server entry inside it.
Discovery, routing, credentials, tool scoping and execution logs all happen at the gateway→connections stay ACTIVE with no work from you
Scorecard MCP runs through a gateway that holds the credentials, scopes the access and records every call.
Managed auth, hosted MCP servers, and every Gmail tool your agent needs.
Free to start.
