Runway enables AI agents to generate and edit videos and images using text prompts and visual references. Create video clips, transform existing footage, generate images, and apply AI-powered visual effects for creative production.
The official MCP SDK for TypeScript lets Node and edge services call MCP servers directly, with an API key for headless auth.
This guide connects Runway MCP to TypeScript through MewCP, so TypeScript can use Runway MCP's 39 tools, like whoami, list_workspaces and list_brand_kits, without your Runway MCP credentials ever reaching the app.
Runway MCP signs in through managed OAuth. MewCP stores the token encrypted, refreshes it before it expires and injects it only at call time. Your agent never holds it, so it can never leak it.
More apps
TypeSafe's Jev model for typed decisions — ask yes/no, multiple-choice or rubric-score questions about a block of state and get calibrated probabilities back instead of prose. Built for classification, routing and scoring steps inside a workflow
Turn your AI agent into a creative studio The world's best image, video, audio and 3D models, inside the AI assistant you already use.
Access Google's Gemini models for text generation, reasoning, coding, multimodal analysis and AI-powered workflows. Process text, images and structured data through advanced AI capabilities.
Give AI coding agents access to a curated library of real-world website designs, screenshots, design systems, and reusable UI components. Search visual references, find similar designs, analyze layouts, extract colors and typography, compare websites, and retrieve copy-ready JSX components for building better interfaces
Other clients
Runway MCP works the same way in every client below. Pick yours for its own guide.
Free to start.
Scope access to the exact Runway MCP tools and accounts you allow, work and personal side by side. Anything outside that scope is refused at call time, even when the model asks for it.
Each call is logged: what ran, on which account, and whether it worked. Failures come back structured, so the agent knows whether to retry instead of guessing.
Every app sits behind one secure MCP URL, and there is no server for you to deploy or keep running. Add an integration in the dashboard and it is live on the next call, with no config to change and no new credentials to wire up. Tools load only when a task needs them, so your agent stays fast and focused however many apps you connect.
SDK clients sign in with an API key. Create one in your dashboard and replace API_KEY in the code below.
SDK clients run without a browser, so they sign in with your API key. Needs Node 18+ as an ES module (for example npx tsx script.ts).
Confirms the Runway connection is authenticated and returns the workspace this MCP connection is pinned to (chosen at sign-in) plus the list of image/video models available in that workspace. It does not return personal account details. Call this first to confirm auth and to pick a model the user can actually run. Check `modelDeprecations`: briefly warn about an upcoming sunset when using that model; it remains usable until that date. Never offer a retired or sunset task variant. Ask before switching models. If `deprecationsKnown` is false, deprecation status could not be checked; do not claim a model is current or retired from that result. Also call it after a paid_plan_required result — `availableImageModels` and `availableVideoModels` are the only models you may offer as a fallback. Do not guess from a generate tool schema. Runway account availability is checked against the currently connected workspace. If `upgradeUrl` is present, the connected workspace is on a free plan. When the user asked for something that plan cannot run (every video tool, or an image model missing from `availableImageModels`), call `show_plans_and_credits` with `gatedCapability` set to the blocked action (e.g. "Generating video") to render the inline upgrade card — do not stop at describing the limitation. Paste `upgradeUrl` only if that tool is unavailable. If `multipleWorkspacesAvailable` is true, the user belongs to other Runway workspaces — call `list_workspaces` to see them. To switch the active workspace, the user must disconnect and reconnect the MCP from their client.
Lists every Runway workspace the authenticated user belongs to, with role and a compact disabled-model summary per workspace. The MCP connection is pinned to ONE workspace (chosen at sign-in); use this tool to confirm which workspace you are currently posting to and to compare options before suggesting a switch. By default this omits full disabled-model arrays to keep the response compact; pass includeDisabledModels=true only when you need the exact policy list. To switch workspaces, the user must disconnect and reconnect the Runway MCP from their client (this is intentional — it keeps account context and audit logs aligned to a single workspace per session).
Lists the connected workspace's visible Brand Kits with their category names and descriptions, without loading assets. Use this for cheap discovery when the user asks for branded work and no Brand Kit is already in context. Call `get_brand_kit` only after a kit is selected or when there is a single clearly applicable kit. An empty list means this workspace has no visible Brand Kits.
Gets one visible Brand Kit from the connected workspace, including its curated image, video, and audio assets grouped by category. Call this when a kit has been selected, when there is one clearly applicable kit, or when the user asks what is inside a kit. Image and video asset URLs can be passed unchanged into the existing generation tools as references. Reading Brand Kits is free and non-destructive.
Lists Runway editor workflows in the authenticated workspace (name, id, updated time). Does not include the graph — use `get_workflow` with `include_graph: true` to inspect. If a similar workflow already exists, reuse it (`get_workflow` / `save_workflow_version` / `run_workflow`) instead of creating a duplicate. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.
Gets a Runway workflow by id (name, description, updated time). Set `include_graph: true` to also load the saved graph. Optional `version` loads a specific version instead of latest. The graph JSON is for your next tool call — summarize the pipeline for the user; do not dump node IDs or ASCII diagrams unless they ask how it is wired. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.
Runs a saved Runway workflow (latest version) in the authenticated workspace. Consumes credits. Returns immediately with a task id — the inline MCP App viewer polls progress when available. Only call `get_task` to poll when no inline viewer is present or the user asks for status. Optional `node_ids` runs only those workflow nodes (upstream must already have succeeded). Optional `node_outputs` overrides constant/asset values for this run without saving a new version. Use `get_workflow` with `include_graph: true` first if you need node ids. Runway account availability is checked against the currently connected workspace. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.
Creates a new Runway workflow with an initial graph (version 1). Do not invent graph JSON — `list_workflows` then `get_workflow` include_graph: true and copy nodes/edges. If a similar workflow already exists in the workspace, update that one instead of creating a duplicate. Use the fewest necessary nodes and set non-overlapping `nodeProps.position` coordinates: inputs left, processing (including system prompt + LLM + JSON Parse) center, outputs right. Never tell the user to implement the graph themselves. Call `validate_workflow_graph` first. Response includes an editor URL and validation errors/warnings even when saved. To run it, call `run_workflow` next. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.
Saves a new version of an existing workflow graph. Graph must be API JSON copied from `get_workflow` — not `{ type, config }`. Keep the graph minimal and its `nodeProps.position` layout readable: inputs left, processing center, outputs right, with no overlaps. Optional `expected_version` guards against stale edits (must be within 1 of latest). Response includes an editor URL. Validate first with `validate_workflow_graph`. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.
Validates a workflow graph without saving. Use before `create_workflow` or `save_workflow_version`. Returns task deprecations for model steps. Tell the user about upcoming sunsets and ask before switching a retired model. Graph must be API JSON (UUID node ids, `nodeType`, `nodeOutputs`) — copy from `get_workflow` include_graph, do not invent `{ type, config }`. Set `estimate_cost: true` to also fetch a credit estimate for the graph. Do not invent graph JSON. `list_workflows` → `get_workflow` include_graph: true and copy nodes/edges. Read `runway://docs/workflows/authoring` before composing or changing a graph, `runway://docs/workflows/examples` for the API shape, and `runway://docs/workflows/models` before image/video chains.
Generate a polished creative product ad video from a product URL or product image plus a campaign idea. CREDIT MODE: Runway account availability is checked against the currently connected workspace. Use this for product marketing / creative ad requests like: - "Make a jewelry ad featuring a chameleon in a jewelry store" - "Turn this product page into a cinematic ad" - "Use this product image and this character reference to make a funny product commercial" v1 supports only format: `creative_ad_video`. Workflow: 1. Get the exact product image from `productImages[0]` or extract the best high-resolution product image from `productUrl`. 2. Use optional `referenceImages` as character, scene, mood, or world references. 3. Internally generate a 3x3 storyboard image, preferring GPT Image 2 for paid workspaces and falling back to Nano Banana Pro otherwise. 4. Use that storyboard as sequential shot guidance for Seedance 2, along with the product image and references, to create the final video. PRODUCT URL FALLBACK: - `productUrl` extraction is best-effort. Some ecommerce sites block automated access or hide product images behind scripts. - If extraction fails, do not retry the same URL or claim Runway cannot make the ad. Ask the user for a screenshot of the product page or a direct product image upload, then call this tool again with `productImages` / `productImageFile`. - Do not call `generate_image` with a text description of the product to create the product reference. That produces a lookalike, not the real SKU. - For branded/luxury products, the exact product image is required for an accurate ad. Only invent or generate a stand-in product if the user explicitly agrees that an approximate substitute is acceptable. Inputs: - productUrl: Product landing page URL. The tool will scrape metadata and rank image candidates, preferring high-resolution product/gallery images. - productImages: Product image URLs from uploads, recent assets, or previous generations. Preferred over productUrl extraction. - productImageFile: ChatGPT-only uploaded product image file. Use this when ChatGPT provides a user-uploaded product image. - referenceImages: Optional character/scene/mood/world references. - promptText: Required creative campaign idea. If the user requests a named artist/director/photographer/studio style, describe the visual qualities instead and do not include the name. - duration: 10 by default; use 15 only when the user asks for a longer ad. STYLE SAFETY: Do not include names of artists, directors, photographers, studios, or other creators anywhere in prompts submitted to Runway tasks. When style direction is needed, translate it into camera language, palette, lighting, genre, era, composition, and material texture. USER-FACING REPLIES: DO NOT mention internal model names unless asked. Do not mention the intermediate storyboard unless useful. Tell the user the creative ad video is generating and the viewer will update when ready.
Generate OR edit an image using a Runway-hosted image model. This is the only image tool — there is no separate "edit_image" tool. Pass the source image as `referenceImages[0]` to edit it. MODEL STATUS: Call `whoami` before choosing a model. Check `modelDeprecations` for upcoming sunsets and unavailable task variants; warn briefly about a relevant upcoming sunset, and ask before switching models. Model enums describe supported inputs, not current availability. CREDIT MODE: Runway account availability is checked against the currently connected workspace. For creative product ad videos from a product URL or product image, use `generate_product_marketing_video` instead. It handles product-image extraction, storyboard generation, and final video creation. If product URL extraction fails, do NOT use this tool to synthesize the product from a text description unless the user explicitly agrees to a stand-in; ask for the real product image or screenshot so the ad uses the actual SKU as reference. For video generation/editing, use `generate_video` (single shot) or `generate_multishot_video` (3-5 connected scenes). To translate an existing ad's on-screen copy into another language while keeping the same creative, use `localize_ad` — it translates the text and preserves the layout, which prompting an edit here does not do reliably. MODES: 1. Text-to-image — pass `promptText` only. The model creates a new image from scratch. Example: "a corgi astronaut floating in a nebula, cinematic lighting". 2. Image edit / transform — pass the source image as `referenceImages[0]` (give it a tag like "input" or "cat") and describe the change in `promptText`. Example: `referenceImages: [{url: ..., tag: "cat"}], promptText: "@cat on a beach at sunset"`. Use for: background swaps, style transfer, object additions/removals, restyling, photo retouching. 3. Composite from multiple references — pass 2+ `referenceImages` each with a distinct tag, then reference them by tag in the prompt. Example: `referenceImages: [{url: ..., tag: "person"}, {url: ..., tag: "room"}], promptText: "@person standing in @room"`. HANDLING USER ATTACHMENTS: ChatGPT: if it supplies an uploaded image as `referenceImageFile`, pass it directly. This tool will download the temporary ChatGPT file server-side, upload it to Runway, and use it as `referenceImages[0].url`. Claude/Cursor/local agents: if the user attached an image, run `init_upload` -> curl -> `complete_upload` first to get a Runway-hosted asset URL, then pass it as `referenceImages[].url`. REUSING ASSETS: URLs returned by other Runway tools (image outputs from prior `generate_image` calls, `complete_upload` outputs) are stable, hosted asset URLs. Pass them directly into `referenceImages[].url` — never re-upload. COPY THESE URLS VERBATIM. They end in a long `_jwt=` authorization token that is meaningless if even one character changes. Copy the whole string exactly as it appeared in the earlier tool result; never retype it, truncate it, shorten it for readability, or reconstruct it from the parts you remember. If you cannot reproduce the URL exactly, call `init_upload` and use the URL it returns instead. EDIT BUTTON (viewer → chat): when a user message says "Edit this image" and names a Runway task ID, the user clicked Edit on a finished image in the inline viewer. Follow this two-step flow: 1. Do NOT generate yet. Reply in one or two sentences: confirm you can edit that image (describe it briefly if you know its prompt) and ask what they want changed — background, style, objects, colors, text. 2. Once they answer, call this tool with `referenceImages: [{ taskId: "<that task ID>", tag: "input" }]` (never retype the image URL), `promptText` describing the change (e.g. "@input at sunset"), and the same ratio as the source image when you know it. Leave `model` unset (the default is the best editing model). MULTIPLE IMAGES / COUNT: When the user asks for multiple images, options, variations, or directions, use `count` for the number of separate images. Rewrite `promptText` so it describes ONE standalone image only. Do NOT include phrases like "3 images", "three options", "multiple versions", "grid", "collage", "contact sheet", "side-by-side", or "set of images" in `promptText` unless the user explicitly wants a single image containing a grid/collage. For product variations, prompt for one clean standalone product composition and let `count` create separate tasks, e.g. "single standalone product image of @product in a premium studio hero composition, not a collage or grid". If the user wants deliberately different concepts (for example studio hero, ecommerce packshot, and lifestyle scene), make separate `generate_image` calls with `count: 1` and a tailored one-image prompt for each concept. STYLE SAFETY: Do not include names of artists, directors, photographers, studios, or other living creators as style anchors. If the user asks for a named style, translate it into neutral visual descriptors such as camera language, palette, lighting, genre, era, composition, and material texture before calling the tool. USER-FACING REPLIES: Pick the right model internally, but DO NOT mention the model name to the user (e.g. "using Nano Banana Pro", "with GPT Image 2") unless they explicitly ask which model was used. Talk about the image content — subject, style, composition — not the engine. MODEL NAMES (a user may name a model by its product name; pass the id on the left): - `nano-banana-pro` = Nano Banana Pro - `nano-banana-2.1` = Nano Banana 2.1 - `nano-banana-2` = Nano Banana 2 - `nano-banana-2-lite` = Nano Banana 2 Lite - `gpt-image-2` = GPT Image 2 - `gpt-image-2.5-flare` = GPT Image 2.5 Flare - `gpt-image-2.5-sunburst` = GPT Image 2.5 Sunburst - `seedream-5` = Seedream 5.0 - `seedream-5-pro` = Seedream 5.0 Pro - `seedream-5-flash` = Seedream 5.0 Flash - `ideogram-4` = Ideogram 4.0 - `ideogram-4.5` = Ideogram 4.5 - `ideogram-4.5-precise-edit` = Ideogram 4.5 Precise Edit - `p-image-ideogram` = Ideogram P-Image - `grok-imagine-image-2` = Grok Imagine Image 2 - `muse-image` = Muse Image - `fibo-1.5` = FIBO 1.5 - `fibo-edit-1.5` = FIBO Edit 1.5 - `recraft-v4.1-vector` = Recraft V4.1 Vector - `recraft-v4.1-flash` = Recraft V4.1 Flash - `gen-4` = Gen-4 - `gen-4-image-turbo` = Gen-4 Turbo Image MODELS: | model | tier | best for | |--------------------|--------|-----------------------------------------------------------------------------------------| | nano-banana-pro | any | DEFAULT except ChatGPT. Photo-real images, character consistency, edits/composites with references. | | nano-banana-2.1 | any | Newest flash Nano Banana. Close prompt adherence, precise edits, up to 14 references, 1K-4K. | | nano-banana-2 | any | Faster, lower-cost Gemini flash model for photo-real images and edits. | | nano-banana-2-lite | any | Cheapest/fastest Gemini model. Great for drafts, iterations, and high-volume 1K images. | | gpt-image-2 | paid | Readable text in images, charts/infographics, strict adherence to complex instructions. | | gpt-image-2.5-flare | paid | DEFAULT on ChatGPT. Faster GPT Image 2.5 for everyday iteration; in-image text, 1K-4K, up to 16 references. | | gpt-image-2.5-sunburst | paid | Highest-fidelity GPT Image 2.5 for precision edits; slower, same controls as Flare. | | seedream-5 | any | Many-reference composites (up to 14 reference images), low-cost 2K/3K output. | | seedream-5-pro | any | Higher-quality Seedream for reference-guided product/style work. Max 10 references. | | seedream-5-flash | paid | Fastest, lowest-cost Seedream (2 credits/image). 1K/2K, up to 10 references. | | ideogram-4 | paid | Ideogram 4.0. Text-to-image, or remix from a single reference image (max 1). | | ideogram-4.5 | paid | Ideogram 4.5. Text-to-image or reference-guided generation from up to 5 references, 1K/2K. | | ideogram-4.5-precise-edit | paid | Targeted edits that keep the rest of the image intact. REQUIRES the source as referenceImages[0]. | | p-image-ideogram | any | Ideogram P-Image. Low-cost text-to-image (2-3 credits/image), 1K/2K. No references. | | grok-imagine-image-2 | paid | Grok Imagine Image 2. Text-to-image or edits/composites from up to 3 references. | | muse-image | any | Meta Muse Image. Very-low-cost generation or edits/composites with up to 10 references. | | fibo-1.5 | paid | Bria FIBO 1.5, trained on licensed data. Text-to-image or reinterpret 1 reference, 1MP/4MP. | | fibo-edit-1.5 | paid | Bria FIBO Edit 1.5. Instruction edits. REQUIRES 1-4 reference images. | | recraft-v4.1-vector | paid | Recraft V4.1 Vector. Text-to-vector SVG (logos, icons, flat illustration). No references. | | recraft-v4.1-flash | paid | Recraft V4.1 Flash. Very fast, cheapest raster text-to-image (1 credit/image). No references. | | gen-4 | any | Fast, low-cost general-purpose Runway model. | | gen-4-image-turbo | any | Cheapest tier for rapid iteration on edits/composites. REQUIRES 1-3 reference images. | Picking heuristic: - nano-banana-pro (DEFAULT except ChatGPT) — photo-realistic images, edits, multi-image composites; highest quality of the Gemini image models. - nano-banana-2.1 — pick when speed/cost matter more than the extra polish of Pro and the brief must be followed closely: reference-guided edits, complex multi-part prompts, or subject consistency across up to 14 references. Prefer it over nano-banana-2; it costs less (4/6/12 credits per image at 1K/2K/4K). Pro still leads on final-grade polish. - nano-banana-2 — the previous flash model, for photo-real generation and edits. Pick it only when the user asks for it by name. - nano-banana-2-lite — pick for the cheapest/fastest option: quick drafts, rapid iteration, or high-volume generation where 1K output is fine. - gpt-image-2 — pick when the image needs readable text, charts, infographics, or strict adherence to a long instruction list. - gpt-image-2.5-flare (DEFAULT on ChatGPT) — pick for iterating on text-heavy or instruction-heavy images: it is the fast GPT Image 2.5 variant (~40s) with 1K-4K output and up to 16 references. On ChatGPT, omit `model` (or pass this id) unless the user names another model. Availability is account-gated: on an unknown-task-type error, fall back to gpt-image-2 or nano-banana-pro. - gpt-image-2.5-sunburst — same controls as Flare, tuned for precision editing and the highest fidelity. Pick when final quality matters more than latency (~120s, same credit cost as Flare at a given tier). Same account gate and fallback as Flare. - seedream-5 — pick when a composite needs MANY references (it accepts up to 14 reference images vs 2-3 elsewhere), or for cheap high-volume 2K generation. - seedream-5-pro — pick over seedream-5 when quality matters more than cost on reference-heavy product shots, style/color/sketch references, or multilingual in-image text. Costs more than seedream-5 and caps at 10 references with no 3K tier. Less predictable on very specific fine-detail instructions; prefer gpt-image-2 when English in-image text fidelity is paramount. - seedream-5-flash — pick for fast, cheap Seedream drafts or high-volume output (2 credits/image at 1K or 2K), including 21:9 at up to 2K, with up to 10 references. It has no grounding, so use seedream-5 or seedream-5-pro for prompts about current products or events. Availability is account-gated: on an unknown-task-type error, fall back to seedream-5 or nano-banana-pro. - ideogram-4 — Ideogram 4.0. Text-to-image, or "remix" from a single reference image (max 1). Pick when the user asks for it by name. - ideogram-4.5 — Ideogram 4.5. Text-to-image, or reference-guided generation from up to 5 references, at 1K or 2K. Pick when the user asks for Ideogram by name (prefer it over ideogram-4 unless they name 4.0). Costs about 25 credits/image for text-to-image and 8 with references. Availability is account-gated: on an unknown-task-type error, fall back to ideogram-4 or nano-banana-pro. - ideogram-4.5-precise-edit — Ideogram 4.5's precise edit mode. Pick for surgical changes to an existing image where everything else must stay untouched: swap or fix in-image text, change one object's color or material, remove or add a single element, retouch a detail. Put the image being edited in referenceImages[0]; up to 4 more references can show what to add (e.g. a logo or product). Output keeps the source image's dimensions, so ratio and imageSize are ignored. Costs about 25 credits/image. Do NOT use it for restyles, new scenes, or prompt-only generation (use ideogram-4.5 or nano-banana-pro). Same account gate as ideogram-4.5: on an unknown-task-type error, fall back to nano-banana-pro. - p-image-ideogram — Ideogram P-Image. Low-cost prompt-only text-to-image at 1K (2 credits/image) or 2K (3 credits/image). Pick when the user asks for P-Image by name, or for cheap Ideogram-style drafts. It takes no reference images; use ideogram-4.5 for reference-guided work. Availability is account-gated: use it only when `whoami` lists it, and on a model_unavailable result fall back to nano-banana-2-lite or nano-banana-pro. - grok-imagine-image-2 — xAI's Grok Imagine Image 2. Text-to-image, single-image edits, or composites from up to 3 tagged references. Prompts cap at 2500 characters (shorter than every other model here). Pick when the user asks for Grok by name, or wants its extra-wide/tall ratios (1:2, 2:1, 9:20, 20:9, 9:19.5, 19.5:9). Availability is account-gated: on an unknown-task-type error, fall back to nano-banana-pro. - muse-image — Meta's very-low-cost option (1 credit/image) for drafts, batch exploration, or edits/composites using up to 10 references. It has one fixed resolution around 2.4MP and is weaker on dense photorealistic scenes and fine texture. Reference tags are not supported, so describe reference roles in the prompt by order instead of using @tags. Availability is account-gated: on an unknown-task-type error, fall back to nano-banana-pro. - fibo-1.5 — Bria FIBO 1.5, trained entirely on licensed data. Text-to-image, or a reinterpretation of a single reference image (max 1). imageSize 1K (1MP, 5 credits/image) or 4K (4MP, 10 credits/image). Pick when the user asks for FIBO or Bria by name, or wants a licensed-data model. Reference tags are ignored. Availability is account-gated: on an unknown-task-type error, fall back to nano-banana-pro. - fibo-edit-1.5 — Bria FIBO Edit 1.5. Edits 1-4 source images from an instruction in promptText (e.g. "change the bicycle to blue"); 5 credits/image. REQUIRES referenceImages: put the image being edited first. Reference tags are ignored, so refer to images by order. If the user asks for a FIBO edit without an image, ask for one instead of calling the tool. Use fibo-1.5 for prompt-only generation. Same account gate as fibo-1.5. - recraft-v4.1-vector — Recraft V4.1 Vector. Text-to-image that returns a vector SVG; 8 credits/image. Pick only when the user asks for a vector/SVG (logos, icons, flat illustrations for print or design tools). Prompt-only: it takes no reference images. The SVG can be viewed and downloaded but cannot be passed to any other Runway tool (edits, animation, upscaling); use a raster model when the image will be reused. Availability is account-gated: on an unknown-task-type error, fall back to nano-banana-pro and say the result is a raster image. - recraft-v4.1-flash — Recraft V4.1 Flash. Low-latency raster text-to-image at 1 credit/image; outputs PNG. Pick for very fast, very cheap prompt-only drafts or high-volume exploration. It takes no reference images. Availability is account-gated: on an unknown-task-type error, fall back to nano-banana-2-lite or nano-banana-pro. - gen-4 — pick for speed, or when the user explicitly wants to continue with the current account settings. - gen-4-image-turbo — pick as the lowest-cost option for RAPID iteration/exploration on reference-based edits and composites (cheapest tier). It ONLY works with 1-3 reference images — it has no text-to-image mode, so never use it for prompt-only generation (use gen-4 or nano-banana-pro for that). ASPECT RATIOS: - nano-banana-pro: auto, 21:9, 16:9, 4:3, 1:1, 3:4, 9:16 - nano-banana-2.1: auto, 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9, 9:21, 21:9, 1:4, 4:1, 1:8, 8:1 - nano-banana-2: auto, 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9, 21:9, 1:4, 4:1, 1:8, 8:1 - nano-banana-2-lite: auto, 1:1, 3:2, 2:3, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9, 21:9 - gpt-image-2: 21:9, 16:9, 3:2, 4:3, 5:4, 1:1, 4:5, 3:4, 2:3, 9:16 - gpt-image-2.5-flare, gpt-image-2.5-sunburst: 21:9, 16:9, 3:2, 4:3, 5:4, 1:1, 4:5, 3:4, 2:3, 9:16 - seedream-5: 1:1, 4:3, 3:4, 16:9, 9:16, 3:2, 2:3, 21:9 - seedream-5-pro: auto, 1:1, 4:3, 3:4, 16:9, 9:16, 3:2, 2:3 - seedream-5-flash: auto, 1:1, 4:3, 3:4, 16:9, 9:16, 3:2, 2:3, 21:9 - ideogram-4: 1:1, 16:9, 9:16, 4:3, 3:4, 3:2, 2:3, 21:9, 9:21 - ideogram-4.5: auto, 1:1, 16:9, 9:16, 4:3, 3:4, 3:2, 2:3, 5:4, 4:5, 2:1, 1:2 - ideogram-4.5-precise-edit: keeps the source image's aspect ratio; ratio is ignored. - p-image-ideogram: 1:1, 16:9, 9:16, 4:3, 3:4, 3:2, 2:3, 21:9, 9:21 (default 1:1) - grok-imagine-image-2: auto, 1:1, 3:4, 4:3, 9:16, 16:9, 2:3, 3:2, 9:19.5, 19.5:9, 9:20, 20:9, 1:2, 2:1 - muse-image: auto, 21:9, 16:9, 3:2, 4:3, 1:1, 3:4, 2:3, 9:16 - fibo-1.5: 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9 - fibo-edit-1.5: auto (default, follows the source), or with 2-4 reference images only: 1:1, 2:3, 3:2, 3:4, 4:3, 4:5, 5:4, 9:16, 16:9 - recraft-v4.1-vector: 1:1, 4:3, 3:4, 16:9, 9:16 (default 1:1) - recraft-v4.1-flash: 1:1, 2:1, 1:2, 3:2, 2:3, 4:3, 3:4, 5:4, 4:5, 6:10, 14:10, 10:14, 16:9, 9:16 (default 1:1) - gen-4: 16:9, 9:16, 1:1, 4:3, 3:4, 21:9 - gen-4-image-turbo: 16:9, 9:16, 1:1, 4:3, 3:4, 21:9 IMAGE SIZE (resolution): - nano-banana-pro, nano-banana-2.1, nano-banana-2: 1K (default), 2K, 4K. Higher tiers cost more credits and take longer. - gpt-image-2.5-flare, gpt-image-2.5-sunburst: 1K (default), 2K, 4K. 2K is the same credit cost as 1K; only 4K costs more. - seedream-5: 2K (default) or 3K. 3K is the same credit cost as 2K. - seedream-5-pro: 1K or 2K (default). 2K costs roughly twice 1K. - seedream-5-flash: 1K or 2K (default), at the same credit cost. - grok-imagine-image-2: 1K (default) or 2K. - ideogram-4.5: 1K or 2K (default). Ignored when ratio is auto. - p-image-ideogram: 1K (default, 2 credits) or 2K (3 credits). - fibo-1.5: 1K (1MP, default) or 4K (4MP). 4K costs twice 1K. - nano-banana-2-lite, gpt-image-2, muse-image, gen-4, gen-4-image-turbo, ideogram-4, ideogram-4.5-precise-edit, fibo-edit-1.5, recraft-v4.1-vector, recraft-v4.1-flash: not configurable — the imageSize parameter is ignored (nano-banana-2-lite outputs at 1K only; muse-image outputs around 2.4MP; gen-4-image-turbo outputs at 1080p; ideogram-4.5-precise-edit keeps the source size). Parameters: - model: One of `nano-banana-pro`, `nano-banana-2.1`, `nano-banana-2`, `nano-banana-2-lite`, `gpt-image-2`, `gpt-image-2.5-flare`, `gpt-image-2.5-sunburst`, `seedream-5`, `seedream-5-pro`, `seedream-5-flash`, `ideogram-4`, `ideogram-4.5`, `ideogram-4.5-precise-edit`, `p-image-ideogram`, `grok-imagine-image-2`, `muse-image`, `fibo-1.5`, `fibo-edit-1.5`, `recraft-v4.1-vector`, `recraft-v4.1-flash`, `gen-4`, `gen-4-image-turbo`. Default: `nano-banana-pro`. On ChatGPT: `gpt-image-2.5-flare`. - promptText: Required. Descriptive prompt for ONE standalone output image. For edits, describe the change ("make the background neon"). For composites, reference tagged images ("@cat sitting on @couch"). If `count` > 1, remove the requested quantity from this prompt and do not ask for a grid/collage/contact sheet. - ratio: Aspect ratio. See per-model list above. - imageSize: Output resolution tier. nano-banana-pro, nano-banana-2.1, and nano-banana-2: `1K` (default), `2K`, `4K` — higher tiers cost more credits and take longer. gpt-image-2.5-flare and gpt-image-2.5-sunburst: `1K` (default), `2K` (same cost as 1K), `4K` (costs more). seedream-5: `2K` (default) or `3K` at the same cost. seedream-5-pro: `1K` or `2K` (default), where 2K costs about twice 1K. seedream-5-flash: `1K` or `2K` (default) at the same cost. grok-imagine-image-2: `1K` (default) or `2K`. ideogram-4.5: `1K` or `2K` (default), ignored when ratio is auto. p-image-ideogram: `1K` (default) or `2K`. fibo-1.5: `1K` (1MP, default) or `4K` (4MP). Ignored by nano-banana-2-lite (1K only), gpt-image-2, muse-image (about 2.4MP), gen-4, and gen-4-image-turbo (1080p). - grounding: Optional boolean, default false. When true the model runs a live web search while generating, so it can pull in current brand, product, trend, or event details it would otherwise have to guess at (e.g. "this season's packaging", "the current team kit"). ONLY `seedream-5` and `seedream-5-pro` support it (not `seedream-5-flash`) — on any other model the flag is dropped, so switch to a Seedream model when the user asks for something current. Leave it off for ordinary prompts: it adds latency and makes output less deterministic. - transparentBackground: Optional boolean, default false. When true the output is a PNG with a transparent (alpha) background — use it for logos, stickers, icons, product cutouts, and other assets meant to be layered over something else. Describe only the subject in `promptText` (e.g. "a red fox mascot logo"), not a background. ONLY `seedream-5-pro`, `seedream-5-flash`, `gpt-image-2`, and `gpt-image-2.5-sunburst` support it. gpt-image-2.5-flare does NOT — pick gpt-image-2.5-sunburst or gpt-image-2 instead. If `model` is omitted, the server picks a supporting model (gpt-image-2 on paid plans, seedream-5-pro on free plans); explicitly naming any other model returns an error. seedream-5-pro and seedream-5-flash allow at most 1 reference image with this flag. To cut the background out of an EXISTING image, use `remove_image_background` instead. - colors: Optional array of up to 10 preferred palette colors as six-digit hex strings (e.g. `["#FF8800", "#1A2B3C"]`). ONLY `recraft-v4.1-vector` and `recraft-v4.1-flash` accept it; any other model returns an error. Use it when the user names exact brand or palette colors. - backgroundColor: Optional six-digit hex background color (e.g. `"#FFFFFF"`). Same models as `colors`. - thinkingLevel: Optional. ONLY `nano-banana-2.1` supports it: `minimal`, `medium` (default when omitted), or `high`. Higher levels reason longer before drawing, which can help complex multi-part prompts at the cost of latency. Omit it unless the user asks; on any other model it is dropped. - count: Number of separate image tasks to generate in one go. Options: 1, 2, 3, 4. Default: 1. This is NOT a request for one output image containing multiple panels. - referenceImages: Optional array of `{url | taskId, tag}` — for edits and composites. `tag` is the alias used in the prompt (e.g. `tag: "cat"` lets you write "@cat" in promptText), except muse-image, fibo-1.5, and fibo-edit-1.5 ignore tags and use reference order. Prefer `taskId` (the Runway task ID of a finished image) whenever the image came from a Runway tool; the server resolves the URL. Otherwise URLs come from `complete_upload` or any public https URL. REQUIRED (1-3) for gen-4-image-turbo, which has no text-to-image mode. REQUIRED for ideogram-4.5-precise-edit, where referenceImages[0] is the image being edited and up to 4 more are references. REQUIRED (1-4) for fibo-edit-1.5. NOT accepted by recraft-v4.1-vector, recraft-v4.1-flash, or p-image-ideogram (prompt-only). ideogram-4 and fibo-1.5 accept at most 1 reference image; grok-imagine-image-2 accepts at most 3; ideogram-4.5 at most 5; muse-image, seedream-5-pro, and seedream-5-flash accept at most 10; seedream-5 and nano-banana-2.1 at most 14; gpt-image-2.5-flare and gpt-image-2.5-sunburst accept at most 16.
Generate OR edit a video using a Runway-hosted video model. Pass the source video as `referenceVideo` to edit/restyle it. MODEL STATUS: Call `whoami` before choosing a model. Check `modelDeprecations` for upcoming sunsets and unavailable task variants; warn briefly about a relevant upcoming sunset, and ask before switching models. Model enums describe supported inputs, not current availability. CREDIT MODE: Runway account availability is checked against the currently connected workspace. PAID PLAN: If this tool returns paid_plan_required, follow the NEXT line in that result — it asks for `show_plans_and_credits` with the blocked action as `gatedCapability`, which renders the upgrade card. Do not retry with a different video model from this schema; every video model needs a paid plan. Only offer models `whoami` lists in `availableVideoModels` / `availableImageModels`, and only after calling `whoami`. CHOOSING BETWEEN THIS TOOL AND `edit_video` FOR FOOTAGE EDITS: - Use `edit_video` (Aleph 2.0 in-context editing) when the user wants a targeted, surgical change to existing footage and everything else must stay exactly as shot: "make the sneakers red", "change the outfit", "remove the object on the wall", "change the season to winter", "relight the scene", element swaps, or restyles that must preserve the original motion/framing/timing — or whenever the user mentions Aleph or Edit Studio. `edit_video` also requires a paid workspace — do not send free-plan users there as a workaround. - Use THIS tool's video-to-video mode (`referenceVideo`, seedance-2) when the source video is more of a loose reference: motion transfer, style/structure inspiration, heavy re-imagining, or when the source violates Aleph's constraints (must be 2-30s, <=1080p, <=30fps, <=10 cuts, conventional aspect ratio). - The two are mutual fallbacks: if an `edit_video` result was unsatisfying (or the source is out of Aleph's constraints), try v2v here; if a v2v edit here changed more than the user asked for, suggest `edit_video` instead. For creative product ad videos from a product URL or product image, use `generate_product_marketing_video` instead. It handles product-image extraction, cinematic storyboard generation, and the final ad video. For multi-shot videos (one prompt drives 3-5 connected scenes), use `generate_multishot_video` instead. MODES: 1. Text-to-video — pass `promptText` only. Most models support this directly. `gen-4-turbo` is the exception: for best reliability, generate a still image first with `generate_image`, then animate that image with `gen-4-turbo`. Example: "aerial drone shot, slow push forward over misty mountains at sunrise". 2. Image-to-video — pass `startFrame.url` (or ChatGPT's `startImageFile`) plus `promptText` to animate a still image. Use this when you want to bring a specific image to life. Example: a product photo + "slow 360 orbit around the bottle". 3. Video-to-video (edit/restyle) — pass `referenceVideo.url` plus a `promptText` describing the desired change. Use this to *modify* an existing video while preserving motion and composition: remove or replace backgrounds, remove unwanted objects/people, add new objects, swap backgrounds, change lighting, restyle scene, alter weather/time-of-day. Supported by the Seedance family (`seedance-2` is the DEFAULT for v2v; `seedance-2.5` handles sources longer than 15s), `kling-4`, `kling-o3-pro`, the WAN family (`wan3`, `wan3-prime`), and `gemini-omni-1.1-flash` (which can also extend a clip with `referenceVideoMode: "extend"`). Example: an existing dance video + "remove the background and place the dancer on a clean white studio backdrop". VIDEO EDITING REQUESTS: - If the user asks to remove a background, remove an object/person, clean up a scene, add something to an existing video, or otherwise edit footage, DO NOT say Runway MCP cannot do it. First consider `edit_video` (Aleph 2.0, paid workspace) per the routing guidance above; otherwise use video-to-video here: pass the source clip as `referenceVideo` / `referenceVideoFile`, default to `seedance-2`, and describe the edit in `promptText`. - For background removal, ask for or use the input video as `referenceVideo` and prompt for the desired replacement/background treatment, e.g. "remove the background and isolate the subject on a plain neutral studio backdrop" or "replace the background with a clean white seamless studio". DRAFT MODE (seedance-2.5 only): - `draft: true` renders a fast 480p preview at the 480p rate (20 credits/s). If the user likes it, call `enhance_video_draft` with its task ID to re-render the same shot at 1080p (68 credits/s) within 7 days. - Draft plus enhance costs MORE than a direct 1080p render, so only draft when the user wants to iterate on a shot (try several prompts, pick one) or explicitly asks for a draft/preview. For a single clip the user already wants in high quality, render at 1080p directly. DURATION DEFAULTS (the tool sets these automatically when you OMIT `duration`): - Text-to-video & image-to-video → 10s (or the model's largest supported value <= 10s; veo-3.1 → 8s). - Video-to-video WITH `referenceVideo.durationSeconds` → largest model-supported value <= source. (e.g. 20s source with kling-o3-pro {5,10,15} → 15s; 8s source with seedance-2 {4..15} → 8s.) - Video-to-video WITHOUT `referenceVideo.durationSeconds` → model's MAX supported duration (15s for seedance/kling) so the edit doesn't get accidentally truncated. DURATION FOR V2V — IMPORTANT: Always pass `referenceVideo.durationSeconds` when you know it (read it from the file metadata you uploaded). It's the only way the tool can know how much of the source to edit. - DO NOT pass `duration: 5` for a 20-second source video — that will silently clip the edit to the first 5 seconds. Just OMIT `duration` and pass `referenceVideo.durationSeconds` instead, and the tool will pick the right output length. - Only set `duration` explicitly when the user asks for a shorter clip ("just make me a 5-second teaser of this 20-second video"). DO NOT use this tool to edit a still **image** → use `generate_image` and pass the image as `referenceImages[0]`. ANIMATE BUTTON (viewer → chat): when a user message says "Animate this image" and names a Runway task ID, the user clicked Animate on a finished image in the inline viewer. Follow this two-step flow: 1. Do NOT generate yet. Reply in one or two sentences: confirm you can turn that image into a short video (default 10 seconds), describe the image briefly if you know its prompt, and ask what should happen in it — motion, camera, mood — and whether they want a different length. 2. Once they answer, call this tool with `startFrame.taskId` set to that task ID (never retype the image URL), `promptText` built from their answer, and `duration` only if they asked for a specific length. Keep the ratio matching the source image when you know it. HANDLING USER ATTACHMENTS: - ChatGPT: if it supplies an uploaded image as `startImageFile`, pass it directly. This tool will download the temporary ChatGPT file server-side, upload it to Runway, and use it as `startFrame.url`. - ChatGPT: if it supplies an uploaded video as `referenceVideoFile`, pass it directly for video-to-video edits. This tool will download the temporary ChatGPT file server-side, upload it to Runway, and use it as `referenceVideo.url`. - ChatGPT: NEVER pass local paths such as `/mnt/data/file.png` or `/mnt/data/file.mp4` as `startFrame.url` or `referenceVideo.url`; the remote MCP server cannot read them. Use `startImageFile` / `referenceVideoFile` instead. - Claude/Cursor/local agents: if the user attached a local image or video, run `init_upload` -> curl -> `complete_upload` first to get a Runway-hosted asset URL, then pass it as `startFrame.url`, `endFrame.url`, `referenceImages[].url`, or `referenceVideo.url`. REUSING ASSETS: URLs returned by other Runway tools (image outputs from `generate_image`, the asset URL from `complete_upload`) are stable, hosted asset URLs. Pass them directly into the relevant field — never re-upload. COPY THESE URLS VERBATIM. They end in a long `_jwt=` authorization token that is meaningless if even one character changes. Copy the whole string exactly as it appeared in the earlier tool result; never retype it, truncate it, shorten it for readability, or reconstruct it from the parts you remember. If you cannot reproduce the URL exactly, call `init_upload` and use the URL it returns instead. Note: outputs of `generate_video` itself are NOT yet usable as `referenceVideo` (re-upload via `init_upload` if needed). USER-FACING REPLIES: Pick the right model internally, but DO NOT mention the model name to the user (e.g. "using Veo 3.1", "with Seedance 2") unless they explicitly ask which model was used. Talk about the video content — subject, motion, mood — not the engine. MODEL NAMES (a user may name a model by its product name; pass the id on the left): - `seedance-2` = Seedance 2.0 - `seedance-2.5` = Seedance 2.5 - `seedance-2-fast` = Seedance 2.0 Fast - `seedance-2-mini` = Seedance 2.0 Mini - `kling-4` = Kling 4.0 - `kling-o3-pro` = Kling O3 Pro - `kling-o3-standard` = Kling 3.0 Omni Standard - `kling-3-pro` = Kling 3.0 Pro - `kling-3-standard` = Kling 3.0 Standard - `kling-3-turbo` = Kling 3.0 Turbo - `kling-3-motion-control` = Kling 3.0 Motion Control - `kling-3-4k` = Kling 3.0 4K - `kling-o3-4k` = Kling 3.0 Omni 4K - `gen-4.5` = Gen-4.5 - `veo-3.1` = Veo 3.1 - `grok-imagine-1.5` = Grok Imagine 1.5 - `gemini-omni-flash` = Gemini Omni Flash - `gemini-omni-1.1-flash` = Gemini Omni 1.1 Flash - `h3-max` = MiniMax H3 Max - `hailuo-3` = MiniMax H3 - `wan3` = WAN 3.0 - `wan3-prime` = WAN 3.0 Prime - `gen-4-turbo` = Gen-4 Turbo MODELS (`*` = image-first workflow): | model | tier | t2v | i2v | v2v | end | refs | audio | durations | max res | |-------------------|------|-----|-----|-----|-----|------|-------|------------|---------| | seedance-2 | paid | Y | Y | Y | Y | Y | Y | 4-15 | 4k | | seedance-2.5 | paid | Y | Y | Y | Y | Y | Y | 4-30 | 1080p | | seedance-2-fast | paid | Y | Y | Y | Y | Y | Y | 4-15 | 720p | | seedance-2-mini | paid | Y | Y | Y | Y | Y | Y | 4-15 | 720p | | kling-4 | paid | Y | Y | Y | Y | Y | Y | 3-30 | 1080p | | kling-o3-pro | paid | Y | Y | Y | Y | Y | Y | 5, 10, 15 | 1080p | | kling-o3-standard | paid | Y | Y | Y | Y | Y | Y | 3-15 | 720p | | kling-3-pro | paid | Y | Y | - | Y | - | Y | 5, 10, 15 | 1080p | | kling-3-standard | paid | Y | Y | - | Y | - | Y | 3-15 | 720p | | kling-3-turbo | paid | Y | Y | - | - | - | Y | 3-15 | 1080p | | kling-3-4k | paid | Y | Y | - | Y | - | Y | 3-15 | 4k | | kling-o3-4k | paid | Y | Y | - | Y | Y | Y | 3-15 | 4k | | gen-4.5 | paid | Y | Y | - | - | - | - | 2-10 | 1080p | | veo-3.1 | paid | Y | Y | - | Y | - | Y | 4, 6, 8 | 1080p | | grok-imagine-1.5 | paid | Y | Y | - | - | Y | Y | 1-15 | 1080p | | gemini-omni-flash | paid | Y | Y | - | - | Y | - | 3-10 | 720p | | gemini-omni-1.1-flash | paid | Y | Y | Y | Y | Y | - | 3-10 | 4k | | h3-max | paid | Y | Y | - | Y | - | - | 5-15 | 768p | | hailuo-3 | paid | Y | Y | Y | Y | Y | Y | 5-15 | 1080p | | wan3 | paid | Y | Y | Y | Y | Y | Y | 2-30 | 1080p | | wan3-prime | paid | Y | Y | Y | Y | Y | Y | 2-30 | 1080p | | gen-4-turbo | paid | * | Y | - | - | - | - | 5, 10 | 720p | Legend: t2v = text-to-video, i2v = image-to-video (startFrame), v2v = video-to-video (referenceVideo, edit/restyle existing video), end = end-frame target, refs = reference images. `*` for gen-4-turbo means it is not native text-to-video: it needs a starting image. For text-only requests, create a still image first, then call gen-4-turbo with that image as `startFrame`. Picking heuristic: - seedance-2 (DEFAULT for t2v/i2v/v2v) — best general-purpose, including existing-footage edits like removing/replacing backgrounds, removing objects/people, and adding new elements. Pick this unless you have a reason not to. The only Seedance tier that does 4k (reserve 4k for explicit 4K requests — it costs ~4x 1080p). - seedance-2.5 — newest Seedance generation. Same capabilities as seedance-2 plus durations up to 30s and prompts up to 15000 characters, and goes up to 1080p (no 4k). The only model with draft mode (see DRAFT MODE below). Pick when the user asks for Seedance 2.5 by name, wants a clip longer than 15s, or wants to preview cheaply before committing to 1080p. Availability is account-gated: if the API rejects it with an unknown-task-type error, fall back to seedance-2. - seedance-2-fast — same capabilities as seedance-2 but faster and cheaper; caps at 720p. Pick when latency/cost matter more than resolution and 1080p isn't needed. - seedance-2-mini — cheapest Seedance tier; identical features, 720p max. Pick for budget-sensitive or high-volume generation. - kling-4 — Kling 4.0, the newest Kling generation in one model: text-to-video, start/end-frame animation, up to 10 reference images (frames count toward the 10; tags are ignored, so refer to references in order: "image 1", "video 1"), up to 5 reference videos (`referenceVideos`, 30s combined), and v2v edits (`referenceVideo`, up to 30s; the output follows the source length). Durations 3-30s, 720p (default) or 1080p, ratios 21:9/16:9/9:16/1:1 plus `adaptive` (follow the references; the default with references). Native audio is ALWAYS ON (no toggle — do not pass generateAudio: false). Every reference video needs `durationSeconds`. Prompts up to 26000 English characters, fewer in other scripts. COST: 15.7 credits/s at 720p (157 for 10s), 23.5 credits/s at 1080p (235 for 10s); with any input video it is 9.4/14.1 credits/s billed on the output length plus the longer of the input video length and the output length. Pick when the user asks for Kling 4 by name, or wants Kling consistency with clips longer than 15s. Not enabled on every workspace yet: only pick it when `whoami` lists it in `availableVideoModels`. On an account_limitation with reason model_unavailable, fall back to kling-o3-pro or seedance-2 and do not retry kling-4. - kling-o3-pro — pick when consistency matters: character/product must look identical across shots, brand or serialized content, or when v2v needs to preserve identity. Best for ads, episodic content, and product storytelling. Optimized for 1-2 main subjects. - kling-o3-standard — lower-cost 720p Omni tier with the same identity/style references and v2v editing strengths. Pick for consistency-focused iteration when 1080p is unnecessary; supports whole-second durations from 3-15s, with video-reference edits capped at 10s. - kling-3-pro — pick when prompt fidelity matters more than reference-driven consistency: complex multi-character scenes (3+ people), crowded environments, experimental scripts, rapid ideation without reference assets. (No v2v support.) - kling-3-standard — lower-cost Kling 3.0 tier for text or first/end-frame generation at 720p, with optional native audio and whole-second durations from 3-15s. Pick it for high-volume iteration when 1080p is unnecessary. (No prompt reference images or v2v support.) - kling-3-turbo — pick for fast prompt iteration or quick first-frame animation at 720p/1080p. It accepts one start frame, has always-on native audio, and supports whole-second durations from 3-15s. (No end frame, reference images, or v2v support.) - kling-3-4k — native 4K Kling generation with optional audio and start/end-frame control. Use only when the user explicitly asks for Kling 3.0 4K or true 4K output: it costs 43 credits/second (129 credits at the 3s minimum). Availability is account-gated; on an unknown-task-type or unavailable-model error, fall back to seedance-2 at 4k. (No reference images or v2v support.) - kling-o3-4k — native 4K Kling Omni generation with optional audio, start/end frames, and up to 7 identity/style references. Use when the user explicitly asks for Kling O3/Omni 4K and consistency matters. It costs 43 credits/second (129 credits at the 3s minimum). (No v2v support.) - veo-3.1 — pick when photorealism and natively-generated audio matter (Veo invents dialogue spoken by characters in-frame, plus ambient sound matched to the scene). (No v2v support.) - gen-4.5 — pick for keyframe-driven storytelling (provide a startFrame and the model anchors at timestamp 0). (No v2v support.) - grok-imagine-1.5 — fast generation with native audio that is ALWAYS ON (no toggle — do not pass generateAudio: false) and audio-driven performances. Pick for quick turnaround, short clips (durations go down to 1s in 1s steps), or reference images addressable from the prompt (tag a reference "hero" and write "@hero" in promptText). Constraints: startFrame and referenceImages are mutually exclusive; up to 7 reference images but they cap output at 720p (1080p is t2v/i2v only); extra ratios 3:2 and 2:3. Availability is account-gated: on an unknown-task-type error, fall back to seedance-2. (No v2v support.) - gemini-omni-flash — paid, real t2v/i2v. Pick for physics-realistic motion (gravity, fluids, collisions), knowledge-grounded scenes, or fusing image + text references in one request. Fixed 720p, no audio, ratios 16:9/9:16/auto only, up to 5 reference images. (No v2v support.) - gemini-omni-1.1-flash — Gemini Omni 1.1 Flash, the newer Omni generation. Same physics/knowledge strengths as gemini-omni-flash plus 360p/720p/1080p/4k output (default 720p), first/end frames, up to 5 reference images or 3 reference videos (`referenceVideos`, each up to 10s), and video edit or extension: pass `referenceVideo` with `referenceVideoMode: "edit"` (default; restyle a clip up to 10s, output length follows the source) or `"extend"` (continue a clip up to 30s; `duration` is the added footage). No audio, ratios 16:9/9:16/auto only, prompts up to 4000 characters. Start/end frames cannot be combined with reference images or videos; an edited or extended video must be the only video, and an extension takes no reference images. Pick when the user asks for Gemini Omni 1.1 by name or wants Omni at a resolution other than 720p, or Omni edits/extensions. - h3-max — MiniMax H3 Max, fal's post-trained H3 variant. Pick only when the user asks for H3 Max by name. It supports text-to-video or first/end-frame keyframes, whole-second durations from 5-15s, and 480p/768p output (`720p` is accepted as an alias for 768p). It has no aspect-ratio control, arbitrary reference images, reference video, or native audio. Prompts up to 6000 characters. Availability is account-gated: on a 403 "This model is not available for your account." fall back to hailuo-3 or seedance-2. - hailuo-3 — MiniMax H3. Native audio is ALWAYS ON (no toggle — do not pass generateAudio: false). Pick for keyframe-driven or reference-driven clips up to 15s with sound, or when the user asks for H3/MiniMax by name; it also does v2v. Ratio note: with reference images or a reference video it defaults to `adaptive` (the reference's own framing); `adaptive` is rejected for text-only prompts. Prompts up to 6000 characters. Availability is account-gated: on a 403 "This model is not available for your account." fall back to seedance-2. - wan3 — Alibaba WAN 3.0 (standard). Pick when the user asks for WAN/Alibaba by name without saying Prime, or for long clips with sound: durations run 2-30s and native audio is on by default (pass `generateAudio: false` to turn it off). Takes up to 10 reference images, a reference video of at most 15s, and prompts up to 20000 characters; it also does v2v. Ratio note: with reference images or a reference video it defaults to `adaptive` (the reference's own framing). Start/end frames cannot be combined with reference images or a reference video. Availability is account-gated: on an unknown-task-type error, fall back to seedance-2. - wan3-prime — high-speed WAN 3.0. Same controls as wan3 (t2v/i2v/v2v, 2-30s, audio, refs, prompt limit) but faster and ~40% more credits (14/s at 720p vs 10/s). Pick when the user asks for WAN Prime / WAN 3.0 Prime / wan3_prime by name, or wants faster WAN generations. Same flag gating as wan3: on an unknown-task-type error, fall back to wan3 then seedance-2. - gen-4-turbo — image-to-video model. If the user supplied an image, pass it as `startFrame`. For text-only requests, first call `generate_image`, then call `generate_video` with `model: "gen-4-turbo"` and `startFrame.url` set to the generated image URL. Do not keep retrying if the starter-frame step is still pending or fails. (No v2v support.) Parameters: - model: One of `seedance-2`, `seedance-2.5`, `seedance-2-fast`, `seedance-2-mini`, `kling-4`, `kling-o3-pro`, `kling-o3-standard`, `kling-3-pro`, `kling-3-standard`, `kling-3-turbo`, `kling-3-motion-control`, `kling-3-4k`, `kling-o3-4k`, `gen-4.5`, `veo-3.1`, `grok-imagine-1.5`, `gemini-omni-flash`, `gemini-omni-1.1-flash`, `h3-max`, `hailuo-3`, `wan3`, `wan3-prime`, `gen-4-turbo`. Default for t2v/i2v: `seedance-2`. Default for v2v (when `referenceVideo` is set): `seedance-2`. - promptText: Required. For t2v/i2v, describe the motion/style. For v2v, describe the edit ("remove the background", "remove the person in the left background", "add a floating logo above the table", "make it nighttime", "snow falling", "cyberpunk neon"). Prompts longer than 3500 characters are truncated. Model-specific limits: `kling-3-turbo`, `kling-3-4k`, and `kling-o3-4k` 2500; `seedance-2.5` 15000; `kling-4` 26000 English characters (fewer in other scripts); `h3-max` 6000; `hailuo-3` 6000; `gemini-omni-1.1-flash` 4000; WAN family (`wan3`, `wan3-prime`) 20000. - ratio: Aspect ratio (16:9, 9:16, 1:1, plus 4:3, 3:4, 21:9, 3:2, 2:3 on some models). Ignored for v2v — the model preserves the source video's ratio. - duration: Seconds (model-dependent — see table). USUALLY OMIT — defaults to 10s for t2v/i2v, and to the largest model-supported value <= source for v2v. See DURATION DEFAULTS above. Only set explicitly when the user asks for a non-default length. - startFrame: `{taskId}` or `{url}` — i2v only. Animate this image. Prefer `taskId` (the Runway task ID of a finished image) whenever the image came from a Runway tool; the server resolves the URL. Otherwise `url` must be public HTTPS or a Runway-hosted asset URL. Do NOT use ChatGPT local paths like `/mnt/data/...`; use `startImageFile` for ChatGPT uploads. - startImageFile: ChatGPT-only uploaded image file param `{download_url, file_id, mime_type, file_name}`. Use this instead of `startFrame` only when ChatGPT provides a user-uploaded file. - endFrame: `{url}` — i2v only. Final-frame target. Requires startFrame. (Seedance family, kling-4, kling-o3-pro, kling-o3-4k, kling-3-pro, kling-3-standard, veo-3.1, gemini-omni-1.1-flash, h3-max, hailuo-3, wan3, wan3-prime.) - referenceImages: Array of `{url, tag}` — i2v style/subject anchors. (Seedance family, kling-4, kling-o3-pro, kling-o3-4k, kling-o3-standard, grok-imagine-1.5, gemini-omni-1.1-flash, hailuo-3, wan3, wan3-prime. kling-4 takes up to 10 images counting startFrame/endFrame and ignores tags; Kling O3 and grok take up to 7; gemini-omni-1.1-flash takes up to 5 and cannot combine them with startFrame/endFrame; hailuo-3 takes up to 9 and WAN takes up to 10. MiniMax H3 and WAN ignore tags and cannot combine references with startFrame/endFrame. For grok-imagine-1.5 the tag is prompt-addressable: tag "hero" -> write "@hero" in promptText.) - referenceVideo: `{url, durationSeconds?}` — v2v source video for editing/restyling existing footage, including removing/replacing backgrounds, removing objects/people, and adding elements. URL must be a Runway-hosted `/datasets/<UUID>.<ext>` URL from `complete_upload`. Do NOT use ChatGPT local paths like `/mnt/data/...`; use `referenceVideoFile` for ChatGPT uploads. Pass `durationSeconds` from your file metadata when known so the model picks an appropriate output length; kling-4 requires it. - referenceVideoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. Use this instead of `referenceVideo` only when ChatGPT provides a user-uploaded video. - referenceVideoMode: gemini-omni-1.1-flash only. `edit` (default) restyles `referenceVideo` per the prompt; `extend` continues it, with `duration` as the added footage. - referenceVideos: Array of `{url, durationSeconds?}` — extra reference clips to borrow from (a dance's choreography, a camera move, a character, a look), separate from `referenceVideo`, the footage to edit. Seedance family, kling-4 and gemini-omni-1.1-flash: seedance-2.5 takes up to 10 videos per request and 30s combined; seedance-2, -fast and -mini take 3 and 15s; kling-4 takes 5 and 30s, each clip 3-30s, and requires `durationSeconds` on every clip. A `referenceVideo` source counts toward both caps. gemini-omni-1.1-flash takes up to 3 clips of at most 10s each and no `referenceVideo` alongside them. Same URL rules as `referenceVideo`; pass `durationSeconds` when known. Not combinable with startFrame/endFrame (except on kling-4). Say in promptText what each clip is for, in order (e.g. "the dancer from video 1 does the choreography from video 2"). - generateAudio: Whether to generate native audio (model-dependent; default true for audio-capable models). For v2v with kling-o3-pro, controls whether to keep the original audio. kling-4, kling-3-turbo, grok-imagine-1.5, and hailuo-3 have no toggle (audio is always on) — omit this parameter for them. h3-max is silent and rejects `generateAudio: true`. - resolution: Override default resolution (kling-4: 720p/1080p, default 720p; kling-3-4k and kling-o3-4k: fixed 4k; kling-3-standard and kling-o3-standard: fixed 720p; kling-3-motion-control: 720p/1080p, default 720p; kling-3-turbo: 720p/1080p, default 1080p; seedance-2: 480p/720p/1080p/4k; seedance-2.5: 480p/720p/1080p; seedance-2-fast & seedance-2-mini: 480p/720p; veo: 720p/1080p — 1080p requires 8s; grok-imagine-1.5: 480p/720p/1080p — 1080p not with referenceImages; h3-max: 480p/768p (720p alias accepted); hailuo-3: 720p/1080p; wan3 & wan3-prime: 480p/720p/1080p; gemini-omni-flash: fixed 720p; gemini-omni-1.1-flash: 360p/720p/1080p/4k, default 720p). COST GUARDRAIL: 4K is expensive (Kling 4K costs 43 credits/second; seedance-2 `4k` costs roughly 4x `1080p`). ONLY use a 4K model or set `4k` when the user explicitly asks for 4K (or "2160p"/"four K"). Never infer `4k` from vague quality requests like "high quality", "best", "high-res", or "crisp" — use the default or `1080p` for those. - characterOrientation: Kling Motion Control only. `image` (default) follows the character-image framing and accepts a 3-10s driving video; `video` follows the driving-video framing and accepts 3-30s.
Re-render a Seedance 2.5 draft video at 1080p. A draft is a 480p preview made with `generate_video` using `model: "seedance-2.5"` and `draft: true`. The enhanced video keeps the draft's prompt, references, aspect ratio, duration and audio. CREDIT MODE: Runway account availability is checked against the currently connected workspace. COST: billed as a full 1080p Seedance 2.5 render (68 credits/s, plus reference-video seconds). The draft's cost is not credited back. WHEN TO USE: - The user liked a draft and wants the full-quality version ("make this one in HD", "enhance it", "render that at 1080p"). WHEN NOT TO USE: - To change anything about the shot (prompt, framing, length): make a new draft with `generate_video` instead. Enhancement cannot change the content. - For videos that were not generated as drafts, including normal seedance-2.5 renders: use `upscale_video` to sharpen an existing video. RULES: - Pass the draft's task ID as `taskId`. The draft must have finished and be less than 7 days old. - Enhance each draft once. Calling this again for the same draft creates, and bills, a second 1080p render. USER-FACING REPLIES: Talk about the result ("rendering your draft in full quality"), not the underlying engine.
Edit an existing video with Runway's Aleph 2.0 in-context video editor: it changes ONLY what you ask for and preserves everything else — subjects, motion, framing, timing, background, lighting. CREDIT MODE: Runway account availability is checked against the currently connected workspace. PAID PLAN REQUIRED: Aleph 2.0 is not available on free Runway workspaces. On a free workspace this tool returns paid_plan_required without creating a task — follow the NEXT line in that result (`show_plans_and_credits` with `gatedCapability`) and do not retry. Tell the user that editing video needs a paid plan rather than naming the model. Do not guess a fallback from this schema. WHEN TO USE THIS vs generate_video: - USE edit_video (this tool) for targeted, surgical edits of existing footage where preservation matters: "make the sneakers red", "change the outfit to a yellow fur coat", "remove the items on the wall behind her", "add graffiti on the wall", "change the season to winter", "relight the scene", "make it dark anime style" — or whenever the user mentions Aleph or Edit Studio. - USE generate_video with referenceVideo (seedance-2) when the source video is more of a loose reference: motion transfer, using the clip as style/structure inspiration, or heavy re-imagining where preserving the original isn't the point. Also fall back to it when the source violates Aleph's constraints (see below), or when an Aleph result was unsatisfying — and suggest edit_video when a seedance v2v edit changed more than the user wanted. - USE expand_video to change the video's ASPECT RATIO by outpainting beyond the frame edges ("make it vertical", "uncrop", "convert to 21:9") — this tool edits content within the frame, expand_video grows the frame. SOURCE VIDEO CONSTRAINTS (Aleph 2.0): 2-30 seconds, up to 1080p, up to 30fps, at most 10 cuts/shot changes, conventional aspect ratio. Cost: 28 credits/second of source (56 credit minimum), so a wasted submission is expensive — that's why the keyframe preview checkpoint below is the default. HOW THE WORKFLOW WORKS (a keyframe-driven edit is much higher quality than a text-only one): 1. CANDIDATES — call with `promptText` + `video` only (video.durationSeconds required for this stage). The tool extracts ~4 evenly spaced frames from the video and returns them inline with their timestamps. LOOK at the frames and pick the one that most clearly shows the edit target (e.g. the widest shot for a background change; a close-up of the subject for a detail change). You may also pick your own timestamp if none fit. 2. KEYFRAME PREVIEW — call again with the SAME arguments plus `keyframeTimestampSeconds`. The tool extracts that exact frame, edits it with an image model to apply your change, and returns the edited keyframe inline. SHOW it to the user and ask if they're happy before continuing (this is the default checkpoint). If they want a different look, re-call with a refined `promptText` or a different timestamp. 3. SUBMIT — once approved, call again with `keyframeImage.url` set to the edited keyframe URL and the SAME `keyframeTimestampSeconds`. The tool submits the Aleph 2.0 video edit task; the result arrives via the inline viewer / `get_task`. SHORTCUTS: - If the user has said they want to skip the preview/approval step, pass `skipPreview: true` at step 2 — the tool generates the keyframe AND submits the video edit in one call. - If the user supplies their own already-edited frame, pass it directly as `keyframeImage.url` with the timestamp it came from. - `textOnly: true` submits a text-prompt-only edit with no keyframe — lower quality, but a valid fallback when frame extraction or the keyframe edit fails. PROMPTING (Aleph 2.0 guide): simple, targeted prompts work best — an action verb (add, remove, change, replace, re-light, re-style) plus the desired transformation, e.g. "change the outfit and bag to soft yellow". Because Aleph only changes what you ask for, precise language beats long descriptions. Use `extraMotionPrompt` ONLY for motion that isn't implied by the keyframe or the original video (e.g. "fire begins to spread up the trees") — most edits don't need it. REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it. HANDLING USER ATTACHMENTS: - ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `video.url`. - Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`. - ALWAYS pass `video.durationSeconds` when you know it (from file metadata) — the candidates stage requires it, and it makes credit accounting accurate. USER-FACING REPLIES: Talk about the edit (what's changing in the video), not the model or task machinery. Do not mention model names unless the user asks. Parameters: - promptText: Required. The edit instruction — what to change; everything else is preserved. - video: `{url, durationSeconds}` — source video. URL must be a Runway-hosted /datasets/ asset URL (from complete_upload) or a public HTTPS video URL. - videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. - keyframeTimestampSeconds: Timestamp (seconds) of the frame that anchors the edit. Must be within the video duration. Setting this triggers the keyframe edit + preview stage. - keyframeImage: `{url}` — an ALREADY-EDITED keyframe image (from step 2's preview, or user-supplied). Requires `keyframeTimestampSeconds`. Setting this submits the video edit. - keyframeModel: Image model for the keyframe edit. One of `nano-banana-pro`, `nano-banana-2.1`, `nano-banana-2`, `nano-banana-2-lite`, `gpt-image-2`, `gpt-image-2.5-flare`, `gpt-image-2.5-sunburst`, `seedream-5`, `seedream-5-pro`, `seedream-5-flash`, `ideogram-4`, `ideogram-4.5`, `ideogram-4.5-precise-edit`, `grok-imagine-image-2`, `muse-image`, `fibo-1.5`, `fibo-edit-1.5`, `gen-4`, `gen-4-image-turbo`. Default: `nano-banana-pro`. - extraMotionPrompt: Optional motion instruction for motion that isn't implied by the keyframe or original video. - skipPreview: When true at the keyframe stage, submit the video edit immediately after generating the keyframe instead of pausing for approval. Only use when the user has explicitly opted to skip the check. - textOnly: Submit a text-prompt-only edit with no keyframe (lower quality fallback).
Expand a video to a new aspect ratio by GENERATING new content beyond the original frame edges (video outpainting / uncrop / reframe) with Runway's Aleph 2.0. The original pixels stay at their scale and position; the model fills the new regions so lighting, motion, and subjects continue seamlessly past the old edges. CREDIT MODE: Runway account availability is checked against the currently connected workspace. PAID PLAN REQUIRED: Aleph 2.0 is not available on free Runway workspaces. On a free workspace this tool returns paid_plan_required without creating a task — follow the NEXT line in that result (`show_plans_and_credits` with `gatedCapability`) and do not retry. Tell the user that expanding video needs a paid plan rather than naming the model. Do not guess a fallback from this schema. WHEN TO USE THIS: - The user wants to change a video's aspect ratio WITHOUT cropping: "make this vertical for TikTok/Reels", "convert 16:9 to 9:16", "make it cinematic 21:9", "uncrop this video", "expand the frame". - NOT for content edits (use `edit_video`) and NOT for generating new footage (use `generate_video`). This tool only grows the canvas. SOURCE VIDEO CONSTRAINTS (Aleph 2.0): 2-30 seconds, up to 1080p, up to 30fps, at most 10 cuts/shot changes. Cost: 28 credits/second of source (56 credit minimum). REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it. HANDLING USER ATTACHMENTS: - ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `video.url`. - Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`. - ALWAYS pass `video.durationSeconds` when you know it (from file metadata) — it makes credit accounting accurate. USER-FACING REPLIES: Talk about the reframe (what ratio the video is becoming and what will fill the new space), not the model or task machinery. Do not mention model names unless the user asks. Parameters: - targetRatio: Required. The aspect ratio to expand to. One of `16:9`, `9:16`, `1:1`, `4:3`, `3:4`, `21:9`, `2:3`, `3:2`, `4:5`, `5:4`, `5:3`, `3:5`. Pick a ratio DIFFERENT from the source video's — expanding to the same ratio is a no-op. - video: `{url, durationSeconds}` — source video. URL must be a Runway-hosted /datasets/ asset URL (from complete_upload) or a public HTTPS video URL. - videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. - promptText: Optional. What should appear in the newly generated regions (e.g. "a sandy beach extends to both sides"). Omit to let the model continue the existing scene — that is the right default for most requests.
Generate a multi-shot video — 3 to 5 connected scenes from a single story or per-shot prompts. Powered by Kling 3.0 (standard at 720p, pro at 1080p). No `model` parameter — resolution selects the engine. CREDIT MODE: Runway account availability is checked against the currently connected workspace. Use this when the user wants a short narrative or sequence: - "a 10-second mini-story about a cat finding a friend" - "a multi-scene product ad: closeup, lifestyle, hero shot" - "transition through 3 environments: forest -> desert -> beach" For a single continuous shot, use `generate_video` instead. For editing/restyling an existing video, also use `generate_video` (with `referenceVideo`). To join clips that already exist into one video, use `stitch_videos`. REQUIRES A PAID RUNWAY WORKSPACE. On free workspaces this tool returns an account-limitation error without creating a task; do not retry. MODES: 1. `auto` (DEFAULT) — Pass a single `storyPrompt` (or `promptText`, accepted as an alias) and the workflow plans the shots for you. Best when the user describes the *story* not the individual scenes. Example: `mode: "auto", storyPrompt: "a barista discovers their cafe has been frozen in time"`. 2. `custom` — Pass per-shot prompts as `shots: [{prompt}, ...]` (3-5 shots). Total `duration` is divided evenly across them. Best when the user has specific ideas for each scene. Example: `mode: "custom", shots: [{prompt: "wide shot of cafe"}, {prompt: "barista looks up, surprised"}, {prompt: "close on a frozen droplet of coffee"}]`. Optional `firstSceneImage` anchors the opening frame for either mode. Availability is checked against the current Runway workspace. Multi-shot tasks routinely take 5-10 minutes. USER-FACING REPLIES: DO NOT mention the underlying engine (Kling 3.0 / Pro) to the user unless they explicitly ask which model was used. Talk about the multi-shot story — the scenes, transitions, and overall narrative — not the engine. Parameters: - mode: `'auto'` (default) or `'custom'`. - storyPrompt: Required for `mode='auto'`. Full story / overall scene description. - shots: Required for `mode='custom'`. Array of `{prompt}`, 3-5 entries. - duration: Total seconds — `5` | `10` (default) | `15`. - aspectRatio: `'16:9'` (default) | `'1:1'` | `'9:16'`. `ratio` is accepted as an alias. - resolution: `'720p'` (default, Kling 3.0 Standard) | `'1080p'` (Kling 3.0 Pro). - sound: Boolean, generate audio (default true). - firstSceneImage: Optional `{url, assetId?}` to anchor the first frame. If `assetId` is omitted, the URL is imported to a Runway dataset asset before submission because the multi-shot workflow requires a resolvable asset reference.
Generate music from a text description — songs, instrumentals, scores, loops, stings — and return an MP3. Describe genre, mood, instrumentation, tempo, structure, and (for songs) lyrics or vocal style in `promptText`. CREDIT MODE: Runway account availability is checked against the currently connected workspace. MODELS (pick with `model`; default `lyria-3-pro`): - `lyria-3-pro` (8 credits): Full-length music — songs, scores, instrumentals. Use for a complete track or soundtrack. - `lyria-3-clip` (4 credits): Short music clips — stings, loops, background beds, jingles. Use for intros, transitions, or a bed under a short video. - `eleven-music-v2.5` (10 credits for the default 30s, 20 per minute, 196 for the 10-minute maximum): Music with an exact length (`durationSeconds`, 3-600 seconds, default 30) and an optional `forceInstrumental` switch. Use when the user needs a specific length or a track up to 10 minutes. Not enabled on every workspace yet. Lyria output length is fixed per model: use `lyria-3-clip` for short beds/stings and `lyria-3-pro` for a full track. For an exact length, use `eleven-music-v2.5` with `durationSeconds`; its cost grows with the length (ceil(seconds × 196 / 600) credits, minimum 1). WHEN TO USE: - "make me a song about…", "write a lo-fi beat", "background music for my video", "an orchestral score that builds", "a 10-second jingle". - "a 3-minute track", "exactly 45 seconds of music", "a 10-minute ambient bed" → `eleven-music-v2.5` with `durationSeconds`, when `whoami` lists it. - NOT for speech/voiceover, NOT for sound effects, NOT for adding a soundtrack to an existing video (generate the music here, then combine externally). PAID PLAN REQUIRED: music generation is not available on free Runway workspaces. On a free workspace this tool returns paid_plan_required without creating a task — follow the NEXT line in that result (`show_plans_and_credits` with `gatedCapability`) and do not retry. Tell the user that generating music needs a paid plan rather than naming the model. AVAILABILITY: `whoami` lists the usable values in `availableMusicModels`. `eleven-music-v2.5` is not enabled on every workspace yet; if the result is an account_limitation with reason model_unavailable, tell the user and offer a Lyria model instead — do not retry `eleven-music-v2.5`. Some workspaces have a model disabled by an admin. If the result is an account_limitation with reason admin_disabled_model, tell the user and stop — do not retry with another model. PROMPTING TIPS: lead with genre and mood, then instrumentation and tempo, then structure ("intro, verse, chorus, outro"). For songs with vocals, include the lyrics or the theme and vocal style. For instrumental beds say "instrumental, no vocals" (with `eleven-music-v2.5`, also set `forceInstrumental: true`). Max 5000 characters on Lyria, 4100 on `eleven-music-v2.5`. USER-FACING REPLIES: Talk about the music (genre, mood, what it's for), not the model or task machinery. Do not mention Lyria, Google, or ElevenLabs unless the user asks. The output is an MP3 the viewer plays inline; headless agents get the URL from `get_task`. Parameters: - promptText: Required. What the music should sound like. Max 5000 characters on Lyria (longer prompts are truncated), 4100 on `eleven-music-v2.5` (longer prompts are rejected). - model: Optional. `lyria-3-pro` (default, full track), `lyria-3-clip` (short clip), or `eleven-music-v2.5` (exact length). - durationSeconds: Optional, `eleven-music-v2.5` only. Whole seconds, 3-600; default 30. - forceInstrumental: Optional, `eleven-music-v2.5` only. true guarantees no vocals. - name: Optional. Display name for the track in the user's Runway library. Defaults to the prompt.
Generate a sound effect from a text description — foley, ambience, impacts, UI sounds, creature noises, whooshes, loops — and return an audio file. Describe the sound source, texture, environment, and how it evolves in `promptText`. With `model: "mirelo-sfx-2"` (Mirelo SFX 2.0) it can also score an existing video: pass `video` and it creates a full sound-effect track that matches the clip. CREDIT MODE: Runway account availability is checked against the currently connected workspace. MODELS: - `sound-effects` (default): text only, MP3, 0.5–30 seconds, optional `loop`. COST: 1 credit per second of requested `duration` (rounded up); 2 credits when `duration` is omitted and the model picks the length. Works on every plan. - `mirelo-sfx-2`: text to sound effect (0.1–600 seconds, default 5) or video to sound effect (pass `video`; 1 second up to the video length, default the whole video, max 600). Prompts up to 2000 characters, plus `negativePrompt`, `format` (mp3, wav, flac), `seed` and, with a video, `preserveSpeech`. No `loop`. COST: 2 credits per second of `duration` (rounded up, minimum 1), e.g. 10 seconds = 20 credits. Paid plans only, and not yet enabled on every workspace. WHEN TO USE: - "a door creaking open", "rain on a tin roof, 10 seconds", "sci-fi laser blast", "a looping crackling campfire", "footsteps on gravel", "a notification chime". - "add sound effects to this video", "replace the audio of this clip with foley" → `model: "mirelo-sfx-2"` with `video`. - NOT for music (use `generate_music`), NOT for speech or voiceover (use `generate_speech`). AVAILABILITY: `mirelo-sfx-2` needs a paid plan and is not enabled on every workspace yet — `whoami` lists the usable values in `availableSoundEffectModels`. If a model is rejected as not enabled, tell the user and stop — do not retry that model. PROMPTING TIPS: name the source first ("heavy wooden door"), then the action ("slams shut"), then the space ("in a stone hallway, long reverb tail"). Mention pacing for longer effects ("starts quiet, builds to a roar"). Set `loop: true` for ambience or beds meant to repeat seamlessly (default model only). For a video, describe which sounds matter ("footsteps and wind, no music"). Max 450 characters on the default model, 2000 on `mirelo-sfx-2`. REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it. HANDLING USER ATTACHMENTS (video, `mirelo-sfx-2` only): - Claude/Cursor/local agents: upload local files first via `init_upload` -> PUT the bytes -> `complete_upload`, then pass the returned asset URL as `video.url`. - ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths such as `/mnt/data/file.mp4` as `video.url`. Pass the duration alongside it as `video: {durationSeconds}`. USER-FACING REPLIES: Talk about the sound (what it is, how long, what it's for), not the model or task machinery. Do not mention ElevenLabs or Mirelo unless the user asks. The output is an audio file (an audio track, not a new video) the viewer plays inline; headless agents get the URL from `get_task`. Parameters: - model: Optional. `sound-effects` (default) or `mirelo-sfx-2`. - promptText: Required. What the sound effect should be. - duration: Optional. Length in seconds. Default model: 0.5–30, omit to let the model choose. `mirelo-sfx-2`: see MODELS. Required with a video whose length is unknown. - loop: Optional, default model only. true to design the effect to loop seamlessly (ambience, beds). Default false. - negativePrompt, format, seed: Optional, `mirelo-sfx-2` only. - video: Optional, `mirelo-sfx-2` only. `{url?, durationSeconds?, taskId?}` — the video to score. URL must be a Runway-hosted `/datasets` URL (from `complete_upload`) or a public HTTPS video URL. - videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. - preserveSpeech: Optional, with a video only. false to drop the clip's dialogue; it is kept by default. - name: Optional. Display name for the asset in the user's Runway library. Defaults to the prompt.
Turn text into spoken audio (text-to-speech) with one of Runway's preset voices and return an MP3. Pass the exact words to speak in `text` and pick a `voice` by name. CREDIT MODE: Runway account availability is checked against the currently connected workspace. COST: `eleven_v3` and `eleven_multilingual_v2`: 1 credit per 50 characters of `text`, rounded up (minimum 1), max 5000 characters per call. `eleven_v4`: 2.2 credits per 1,000 characters until October 12, 2026 PT, then 5 credits per 1,000 characters, rounded up (minimum 1), max 2500 characters per call. `fish-drama-3`: 1 credit per 250 characters, rounded up (minimum 1) (provisional), max 5000 characters per call. `seed-audio-1.0`: 5 credits upfront, then 0.25 credits per second of output in total, rounded up (5–30 credits; output is capped at 120 seconds), max 2048 characters per call. Split longer scripts into several calls. AVAILABILITY: `eleven_v4` and `fish-drama-3` need a paid plan and are not enabled on every workspace yet; `seed-audio-1.0` needs a paid plan. If one is not enabled, tell the user and offer `eleven_v3` instead — do not retry that model. MODELS (pick with `model`; default `eleven_v3`): - `eleven_v3`: Most expressive. Understands inline delivery tags in the text such as [whispers], [laughs], [sighs], [excited]. Best for dialogue, characters, and emotive narration. - `eleven_multilingual_v2`: Steadier and more predictable; ignores delivery tags. Best for long neutral narration, product voiceover, and non-English text. - `eleven_v4`: Newest expressive model with inline delivery tags ([laughs], [whispers], [pause]) and a much lower per-character price. Max 2500 characters per call. The only model with multi-speaker `dialogue` and library voices from `search_voices`. Paid plans only, and not yet enabled on every workspace. - `fish-drama-3`: Fish Audio's expressive drama model. Direct delivery with bracketed cues such as [barely holding it together] or [small laugh], or wrap words in <whisper>...</whisper>. Uses its own seven English voices (see FISH AUDIO VOICES); no voice controls, speed, languageCode, voiceId or dialogue. Max 5000 characters per call. Paid plans only, and not yet enabled on every workspace. - `seed-audio-1.0`: ByteDance's omni audio model. `text` can be the exact words to speak in its default voice, or a scene description that mixes voice direction, dialogue, music, ambience and sound effects in one clip (up to 2 minutes). No preset voices, voice controls, speed, languageCode, voiceId or dialogue. Max 2048 characters per call. Paid plans only. VOICES for the Eleven models (pick with `voice`; default `Leslie`). Choose by the user's description ("deep male narrator", "cute cartoon voice", "British woman, calm"); do not ask the user to pick from the full list unless they want to browse: See the `voice` input description for the full voice profiles. FISH AUDIO VOICES: `fish-drama-3` only accepts these English voices (default `Sarah`); the voices above do not work with it: See the `voice` input description for the Fish voice profiles. WHEN TO USE: - "read this aloud", "voiceover for my video: …", "narrate this script in a British accent", "say 'welcome back' in a robotic voice". - "a conversation between a detective and a suspect", "a two-host podcast intro": `eleven_v4` with `dialogue`. - "a radio host talking over upbeat music and crowd noise": `seed-audio-1.0` with the scene described in `text`. - NOT for music (use `generate_music`), NOT for sound effects (use `generate_sound_effect`), NOT for cloning a specific person's voice, NOT for translating or re-voicing an existing audio/video file. PROMPTING TIPS: `text` is spoken verbatim — do not include stage directions as plain prose. With `eleven_v3` or `eleven_v4` you may add bracketed delivery tags inline, e.g. "[whispers] Don't move. [pause] Did you hear that?"; `eleven_multilingual_v2` ignores tags. `eleven_v4` rejects SSML `<break>` tags — use [pause]. Use punctuation and line breaks to control pacing. Set `languageCode` when the text is not English and pronunciation matters. Use a 2-letter ISO 639-1 code (`es`, `de`, `pt`) or a 3-letter code for eleven_v3 (`eng`, `fil`) — never a region tag like `en-US` or `pt-BR`. USER-FACING REPLIES: Talk about the voiceover (which voice, what it says, how long), not the model or task machinery. Do not mention ElevenLabs unless the user asks. The output is an MP3 the viewer plays inline; headless agents get the URL from `get_task`. Parameters: - text: Required unless `dialogue` is given. The words to speak, verbatim (or, for `seed-audio-1.0`, a scene description). Max 5000 characters (2500 for `eleven_v4`, 2048 for `seed-audio-1.0`). - voice: Optional. Preset voice name from the profiles in the `voice` input description. Default `Leslie`, or `Sarah` for `fish-drama-3`. - voiceId: Optional, `eleven_v4` only. A voice from `search_voices` when no preset fits the user's description. - dialogue: Optional, `eleven_v4` only. Ordered lines `[{ voice | voiceId, text }]` for a conversation between up to 10 voices in one MP3 (max 2,000 characters in total). Use instead of `text`; lines with the same voice are the same speaker. - model: Optional. `eleven_v3` (default, expressive, supports tags), `eleven_multilingual_v2` (steady), `eleven_v4` (expressive, cheaper, paid plans, max 2500 characters), `fish-drama-3` (Fish Audio drama voices, paid plans, max 5000 characters), or `seed-audio-1.0` (speech or mixed audio scenes in a default voice, paid plans, max 2048 characters, no `voice`). - speed: Optional. 0.7–1.2; 1.0 is normal. - languageCode: Optional. 2-letter ISO 639-1 code (e.g. `es`, `de`, `pt`) or 3-letter code for eleven_v3 (e.g. `eng`, `fil`). Do not include a region. - stability, similarityBoost, style, useSpeakerBoost, seed, previousText, nextText: Optional voice controls (see each parameter); not for `fish-drama-3` or `seed-audio-1.0`. Leave them unset unless the user asks for the effect. - name: Optional. Display name for the asset in the user's Runway library. Defaults to the text.
Search the Eleven v4 voice library (thousands of voices beyond the preset list in `generate_speech`). Returns voices with a `voiceId` to pass to `generate_speech` with `model: "eleven_v4"`, either as `voiceId` or as a dialogue line's `voiceId`. Free: does not spend credits. WHEN TO USE: the user wants a voice the preset list does not cover ("an old Scottish fisherman", "a Brazilian Portuguese news anchor"), or asks to browse voices. Otherwise use the preset `voice` names. Filters match the library's lowercase labels exactly; if a filtered search returns nothing, retry with fewer filters or with `query` alone. AVAILABILITY: not enabled on every workspace yet. If it is not enabled, use preset voices instead.
Localize an existing ad image for another market: the on-screen copy is translated into the target language and the ad is regenerated with the same layout, typography feel, product, and visual creative. Use this instead of re-generating an ad from scratch when the user wants the SAME creative in another language. CREDIT MODE: Runway account availability is checked against the currently connected workspace. WHEN TO USE THIS: - "localize this ad for Japan", "translate this banner into Spanish", "give me German/French/Korean versions of this creative", "adapt this ad for the LATAM market". - One call = one language. For several markets, call this once per language (each call is a separate task and a separate charge). - NOT for translating a *document* or plain text (no image involved), NOT for editing anything other than the ad's language (use `generate_image` with referenceImages), NOT for upscaling (`upscale_image`). IMAGES ONLY: there is no video ad localization. If the user asks to localize a video ad, say that isn't supported yet rather than trying `edit_video` — Aleph rewrites frames and will not reliably retype on-screen copy. SOURCE AD REQUIREMENTS: the image must contain readable on-screen text. If it has none, the task fails with "No translatable text found in the reference image" — when that happens, tell the user the ad has no detectable copy to translate instead of retrying. High-resolution sources with clear typography localize far more reliably than heavily stylized layouts. COST: flat 21 credits per call. OUTPUT: a single localized image. HANDLING USER ATTACHMENTS: - ChatGPT: pass an uploaded image as `imageFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `image.url`. - Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `image.url`. Outputs of `generate_image` can be passed directly as `image.url`. USER-FACING REPLIES: Talk about the localized ad and the language, not the model or task machinery. Parameters: - image: `{url}` — the source ad to localize. Must be a Runway-hosted asset URL (from `complete_upload` or `generate_image`) or a public HTTPS image URL. - imageFile: ChatGPT-only uploaded image file param `{download_url, file_id, mime_type, file_name}`. - targetLanguage: Required. ISO-style code or English name. Supported: ar (Arabic), zh (Chinese (Simplified)), zh-Hant (Chinese (Traditional)), nl (Dutch), en (English), fr (French), de (German), el (Greek), hi (Hindi), id (Indonesian), it (Italian), ja (Japanese), ko (Korean), pl (Polish), pt (Portuguese), ru (Russian), es (Spanish), sv (Swedish), th (Thai), tr (Turkish), uk (Ukrainian), vi (Vietnamese). Regional variants collapse to the base language (e.g. `pt-BR` -> Portuguese); `zh` is Simplified and `zh-Hant` is Traditional Chinese.
Remove the background from an image and output a PNG with a TRANSPARENT background (alpha channel) — the subject is kept, everything else is removed. The default model (SAM3) isolates the subject you describe; `model: "birefnet-v2"` (BiRefNet 2.0) removes the background automatically with no subject prompt and gives soft edges on hair and fur with `variant: "matting"`. CREDIT MODE: Runway account availability is checked against the currently connected workspace. WHEN TO USE THIS: - "remove the background", "cut out the person/product", "make the background transparent", "isolate the subject", "give me a sticker/cutout of X". - NOT for replacing the background with new content or other edits (use `generate_image` with referenceImages), NOT for upscaling (`upscale_image`). For videos use `remove_video_background`. SOURCE IMAGE CONSTRAINTS (sam3): at most 4096px per side. Cost: 1 credit per image. If the subject can't be found in the image the task fails with "No detections found" — rephrase the subject and retry once, then tell the user what couldn't be found. BIREFNET-V2: 1 credit per image. PAID PLAN REQUIRED and not enabled for every workspace — only use it when `whoami` lists it in `availableBackgroundRemovalModels`. If it is rejected as unavailable, fall back to the default model. OUTPUT: a PNG with an alpha channel (true transparency). HANDLING USER ATTACHMENTS: - ChatGPT: pass an uploaded image as `imageFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `image.url`. - Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `image.url`. Outputs of `generate_image` can be passed directly as `image.url`. USER-FACING REPLIES: Talk about the cutout (what's being isolated and that the background becomes transparent), not the model or task machinery. Do not mention SAM3 or BiRefNet unless the user asks. Parameters: - model: `sam3` (default) | `birefnet-v2`. - subject: Required for sam3, ignored by birefnet-v2. What to KEEP, described in a few words (e.g. "the person", "the sneaker", "the dog on the left"). Max 200 characters. - variant: birefnet-v2 only. `general` (default), `general-2k`, `general-heavy`, `general-dynamic`, `matting` (hair/fur, soft alpha), or `portrait` (people). - image: `{url}` — source image. URL must be a Runway-hosted asset URL (from complete_upload or generate_image) or a public HTTPS image URL. - imageFile: ChatGPT-only uploaded image file param `{download_url, file_id, mime_type, file_name}`.
Remove the background from a video: SAM3 segmentation tracks the subject you describe across every frame and outputs a .webm video with a TRANSPARENT background (alpha channel) — the subject is kept, everything else is removed. Set `invert: true` to do the opposite: cut the subject OUT and keep the background. CREDIT MODE: Runway account availability is checked against the currently connected workspace. WHEN TO USE THIS: - "remove the background", "cut out the person", "green-screen this video", "isolate the product", "make the background transparent". - NOT for replacing the background with new content (use `edit_video` — it can restyle or swap backgrounds in place), NOT for content generation (`generate_video`), NOT for sharpening (`upscale_video`). SOURCE VIDEO CONSTRAINTS: at most 60 seconds (~1800 frames at 30fps). Cost: 1 credit/second of input video. `video.durationSeconds` is REQUIRED — the task is billed by duration and the backend rejects sources without one. OUTPUT: a .webm file with an alpha channel. Warn the user that some players show transparent regions as black/gray; the transparency is real and works in editors and compositing tools. AVAILABILITY: account-gated. If the task is rejected with an "Unknown task type" error, this workspace does not have access yet — tell the user the feature isn't enabled for their account instead of retrying. REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it. HANDLING USER ATTACHMENTS: - ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths like /mnt/data/... as `video.url`. Still pass `video.durationSeconds` (from file metadata). - Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`. USER-FACING REPLIES: Talk about the cutout (what's being isolated and that the background becomes transparent), not the model or task machinery. Do not mention SAM3 unless the user asks. Parameters: - subject: Required. What to KEEP, described in a few words (e.g. "the person", "the red car", "the dog on the left"). Max 200 characters. - video: `{url?, durationSeconds, assetId?}` — source video. URL must be a Runway-hosted /datasets/ asset URL (from complete_upload) or a public HTTPS video URL; omit it only when passing videoFile. durationSeconds is ALWAYS required. - videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. Pass the duration alongside it as `video: {durationSeconds}`. - keepSound: Optional, default true. Keep the source audio track in the output. - invert: Optional, default false. Remove the SUBJECT and keep the background instead.
Upscale an existing image to a higher resolution (2x, 4x, 8x, or 16x) using Runway's AI image upscaler. Use this to sharpen, denoise, and increase the resolution of an image — it does NOT change the content, composition, or style; for those edits use `generate_image` with `referenceImages`. CREDIT MODE: Runway account availability is checked against the currently connected workspace. PAID PLAN REQUIRED: This upscaler is not available on free Runway workspaces. If the connected workspace is on a free plan the task will be rejected with an account-limitation message — tell the user image upscaling isn't available in their current workspace and do not retry. There is no free upscaling alternative; do NOT try to fake an upscale with `generate_image`. WHEN TO USE: - The user wants a low-res / soft / blurry / grainy image made crisper or larger (e.g. "upscale this to 4x", "make this image higher resolution", "enhance / denoise this photo"). - The user has a finished image (their own upload, or an output from `generate_image`) and wants a higher-fidelity version of the exact same image. WHEN NOT TO USE: - To restyle, edit, remove/replace backgrounds or objects, or otherwise change the content → use `generate_image` with `referenceImages[0]`. - To generate a brand-new image → use `generate_image`. - To upscale a video → use `upscale_video`. INPUT: - Provide the source image as `image.url`: a Runway-hosted URL (from `complete_upload` or a `generate_image` output) or a public HTTPS image URL. The tool imports it into Runway automatically. - Pass `image.width`/`image.height` when known so scale-factor limits can be checked before submitting. The input must be at least 300x300px, and the output (width * scaleFactor x height * scaleFactor) must stay under ~25 megapixels — e.g. a 4000x3000 image only supports 1 step of 2x. FLAVOR / SCALE RULES: - flavor `sublime` (DEFAULT — artistic/illustrated images, smooth gradients, vibrant colors): scaleFactor 2, 4, 8, or 16. - flavor `photo` (photographic images, natural colors, realistic detail): scaleFactor 2 only. - flavor `photo_denoiser` (noise reduction for low-light/grainy photos): scaleFactor 2 only. HANDLING USER ATTACHMENTS: - Claude/Cursor/local agents: if the user attached a local image, run `init_upload` -> curl -> `complete_upload` first, then pass the returned URL as `image.url`. - ChatGPT: if it supplies an uploaded image as `imageFile`, pass it directly. NEVER pass local paths such as `/mnt/data/file.png` as `image.url`; use `imageFile` instead. USER-FACING REPLIES: Talk about the result ("upscaling your image 4x"), not the underlying engine. Do not mention the model/provider unless the user explicitly asks. Parameters: - image: `{ url, width?, height? }` — the source image to upscale. URL must be public HTTPS or Runway-hosted. Do NOT use ChatGPT local paths like `/mnt/data/...`; use `imageFile` for ChatGPT uploads. - imageFile: ChatGPT-only uploaded image file param `{download_url, file_id, mime_type, file_name}`. - scaleFactor: 2, 4, 8, or 16. Default: 2. Factors above 2 require flavor `sublime`. - flavor: `sublime` | `photo` | `photo_denoiser`. Default: `sublime`. Pick `photo` for photographs, `photo_denoiser` for grainy/low-light photos. - sharpen: 0-100. Default: 10. Higher = crisper edges. - smartGrain: 0-100. Default: 10. Adds natural film-like grain to avoid a plasticky look. - ultraDetail: 0-100. Default: 30. Higher = more invented micro-detail (can drift from the source).
Upscale an existing video to a higher resolution (up to 4K) using Runway's AI video upscaler. Use this to sharpen, clean up, and increase the resolution of footage — it does NOT change the content, motion, style, or duration; for those edits use `generate_video` (video-to-video). CREDIT MODE: Runway account availability is checked against the currently connected workspace. WHEN TO USE: - The user wants a low-res / soft / blurry video made crisper or larger (e.g. "upscale this to 4K", "make this video higher resolution", "enhance the quality of this clip"). - The user has a finished clip (their own upload, or an output from another tool that they re-uploaded) and wants a higher-fidelity version of the exact same video. WHEN NOT TO USE: - To restyle, edit, remove/replace backgrounds, remove objects, or otherwise change the content → use `generate_video` with `referenceVideo`. - To generate a brand-new video → use `generate_video`. INPUT: - Provide the source video as `video.url`. This must be a Runway-hosted `/datasets/<UUID>.<ext>` URL returned by `complete_upload`, OR a public HTTPS video URL (the tool imports it into Runway automatically). - Pass `video.durationSeconds` when known. Videos must be under 40 seconds. REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it. HANDLING USER ATTACHMENTS: - Claude/Cursor/local agents: if the user attached a local video, run `init_upload` -> curl -> `complete_upload` first to get a Runway-hosted asset URL, then pass it as `video.url`. - ChatGPT: if it supplies an uploaded video as `videoFile`, pass it directly. This tool will download the temporary ChatGPT file server-side, upload it to Runway, and use it as the source. NEVER pass local paths such as `/mnt/data/file.mp4` as `video.url`; use `videoFile` instead. REUSING ASSETS: A URL from `complete_upload` is a stable, hosted asset URL — pass it directly as `video.url`, never re-upload. Note: outputs of `generate_video` are NOT yet usable as `video.url` here (re-upload via `init_upload` if needed). USER-FACING REPLIES: Talk about the result ("upscaling your video to 4K"), not the underlying engine. Do not mention the model/provider unless the user explicitly asks. Parameters: - video: `{ url, durationSeconds? }` — the source video to upscale. URL must be public HTTPS or a Runway-hosted `/datasets` URL. Do NOT use ChatGPT local paths like `/mnt/data/...`; use `videoFile` for ChatGPT uploads. - videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. Use this instead of `video` only when ChatGPT provides a user-uploaded file.
Convert an existing standard (SDR) video into a true HDR delivery using Runway Ruby. This regrades the footage — expanding highlights, contrast and colour into an HDR container — it does NOT change the content, motion, style, or duration; for those edits use `generate_video` or `edit_video`. CREDIT MODE: Runway account availability is checked against the currently connected workspace. WHEN TO USE: - The user wants an HDR master or HDR delivery of a clip they already have ("make this HDR", "convert to HDR10", "give me an HLG version", "HDR ProRes for my edit"). WHEN NOT TO USE: - To restyle, edit or regenerate the footage → `generate_video` / `edit_video`. - To raise resolution or sharpen → `upscale_video`. SOURCE VIDEO CONSTRAINTS: at most 30 seconds. Cost: 20 credits per second of source video, or 40 credits per second above 4 megapixels (roughly above 4K). Pass `video.durationSeconds` when known so an over-long clip is rejected before any upload. OUTPUT FORMATS: - `hdr10` (default) — HDR10 master, the safest general-purpose delivery. - `hlg` — Hybrid Log-Gamma, for broadcast-style delivery. - `hdr_prores` — HDR ProRes `.mov` for editorial. Most browsers CANNOT play this file: tell the user to download it and open it in an editor or player, and do not describe it as viewable inline. `proresProfile` (422, 422 HQ, 4444) applies only to this format. AVAILABILITY: account-gated. If the call comes back saying HDR conversion is not enabled, or that the plan does not include HDR output, this workspace does not have access — tell the user instead of retrying or switching format. REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it. HANDLING USER ATTACHMENTS: - Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`. - ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths such as `/mnt/data/file.mp4` as `video.url`. Pass the duration alongside it as `video: {durationSeconds}`. USER-FACING REPLIES: Talk about the result ("converting your clip to HDR10"), not the underlying engine. Do not mention the model unless the user explicitly asks. Parameters: - video: `{url?, durationSeconds?, assetId?}` — the source video. URL must be a Runway-hosted `/datasets` URL (from `complete_upload`) or a public HTTPS video URL; omit it only when passing `videoFile`. - videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. - outputFormat: Optional, default `hdr10`. One of hdr10, hlg, hdr_prores. - proresProfile: Optional, only valid with `outputFormat: 'hdr_prores'`. One of 422, 422 HQ, 4444.
Convert an existing video — Runway-made or not — to a different frame rate (24, 25, 30, 48, 50, 60, 120, 23.98, 29.97, 59.94 fps) using Runway's Enhance Frame Rate. This synthesises the in-between frames for smooth motion at the new rate; it does NOT change the content, resolution, style, or duration of the footage. For those edits use `generate_video`, `edit_video` or `upscale_video`. CREDIT MODE: Runway account availability is checked against the currently connected workspace. WHEN TO USE: - The user needs a clip at a specific delivery frame rate: broadcast (25 or 59.94 fps), an editing timeline (25 or 30 fps), a client spec, or smoother high-frame-rate playback (48, 60, 120 fps). Runway generations are 24 fps, so "make this 30fps", "I need 59.94 for broadcast", "convert to 60fps" all belong here. WHEN NOT TO USE: - To raise resolution or sharpen → `upscale_video`. - To restyle, edit, extend or regenerate the footage → `generate_video` / `edit_video` / `expand_video`. - To grade into HDR → `convert_video_to_hdr`. - For a frame rate that is not in the list above: there is no custom fps. Offer the nearest preset instead. SOURCE VIDEO CONSTRAINTS: at most 300 seconds (5 minutes), up to 4K (3840x2160 in either orientation), aspect ratio at most 4:1. Standard (SDR) sources only — HDR inputs are not supported yet. Cost: 1 credit per 2 seconds of source video (minimum 1 credit), so a 60s clip costs 30 credits. Pass `video.durationSeconds` when known so an over-long clip is rejected before any upload. TARGET FRAME RATE: `targetFrameRate` is required. Common picks — `25` (PAL broadcast / EU timelines), `29.97` or `30` (NTSC / US timelines, social), `59.94` (US broadcast HFR), `48`, `50`, `60`, `120` (smooth / slow-motion sources). `23.98` and `24` are also available for conforming footage that is not already 24 fps. AVAILABILITY: paid plans only, and account-gated. If the call comes back saying the feature is not enabled or requires a paid plan, tell the user instead of retrying or switching frame rate. REUSING A VIDEO FROM THIS CHAT: if the source is a video a Runway tool generated earlier in this conversation, pass its task ID as `video.taskId` (no `url`, no `videoFile`). The server looks up the URL and duration. Do NOT ask the user to upload or download it. HANDLING USER ATTACHMENTS: - Claude/Cursor/local agents: upload local files first via `init_upload` -> curl -> `complete_upload`, then pass the returned asset URL as `video.url`. - ChatGPT: pass an uploaded video as `videoFile`; the tool imports it server-side. NEVER pass local paths such as `/mnt/data/file.mp4` as `video.url`. Pass the duration alongside it as `video: {durationSeconds}`. USER-FACING REPLIES: Talk about the result ("converting your clip to 59.94 fps"), not the underlying engine. Do not mention the model unless the user explicitly asks. Parameters: - video: `{url?, durationSeconds?, assetId?, taskId?}` — the source video. URL must be a Runway-hosted `/datasets` URL (from `complete_upload`) or a public HTTPS video URL; omit it only when passing `videoFile` or `video.taskId`. - videoFile: ChatGPT-only uploaded video file param `{download_url, file_id, mime_type, file_name}`. - targetFrameRate: Required. One of 24, 25, 30, 48, 50, 60, 120, 23.98, 29.97, 59.94 (fps, as a string).
Join 2-20 video clips end to end into one video file, in the order given, with hard cuts or crossfades. CREDIT MODE: Runway account availability is checked against the currently connected workspace. WHEN TO USE: - The user generated several clips (e.g. multiple `generate_video` shots of a story) and wants them as one video: "put these together", "combine the clips", "make it one video", "join them into a 30s ad". - The user has their own clips and wants them assembled in a specific order. WHEN NOT TO USE: - To generate a new multi-scene video from a story → `generate_multishot_video`. - To make one clip longer by generating more footage → `expand_video` / `generate_video`. - Side-by-side / split-screen layouts, trimming, text overlays or adding music are not supported by this tool. CLIPS: `clips` is the ordered list of sources. For each clip: - A video a Runway tool generated earlier in this conversation → pass `{taskId}` only. The server looks up the video and its duration. Do NOT ask the user to download or re-upload it. - Any other video → pass `{url, durationSeconds}`. `durationSeconds` is REQUIRED for URL clips (the stitch checks the total length against the files). Local agents: read it with `ffprobe`; upload local files first via `init_upload` -> curl -> `complete_upload`. - Clips should share an aspect ratio; mixed ratios are stitched but will not fill the frame consistently. TRANSITIONS: `transition` is `cut` (default, hard cuts) or `crossfade`. `crossfadeSeconds` (0.1-4.9, default 0.5) sets the overlap; each crossfade shortens the result by that amount. LIMITS & COST: total length at most 60 minutes. Cost: 1 credit per 20 seconds of total input (minimum 1 credit), so three 10s clips cost 2 credits. Paid Runway plans only — on a free workspace this returns an account-limitation result without creating a task; do not retry. USER-FACING REPLIES: Talk about the result ("joining your 3 clips into one 30s video"), not the underlying processing. Parameters: - clips: Required. 2-20 items in play order, each `{taskId}` or `{url, durationSeconds, assetId?}`. - transition: `'cut'` (default) | `'crossfade'`. - crossfadeSeconds: Crossfade length in seconds when `transition` is `'crossfade'`.
Initialize a file upload to Runway. Returns temporary upload URLs for direct upload. USER-FACING REPLIES: Do not mention provider-specific storage services. Say "Runway upload", "Runway-hosted asset", or "temporary upload URL" instead. USE THIS ONLY WHEN: The user has attached a NEW local file (image or video) that isn't already on Runway. CHATGPT: If ChatGPT supplies the uploaded file as `file`, pass it directly. This tool will download the temporary ChatGPT file server-side, upload it to Runway, complete the upload, and return the final asset URL. Do not ask the user to run curl. CHATGPT: Do NOT call this tool with only `filename`, `fileSize`, and `mimeType`; that only returns curl instructions and cannot upload bytes from ChatGPT. For image-to-video, call `generate_video` with `startImageFile`. For video-to-video, call `generate_video` with `referenceVideoFile`. DO NOT USE THIS FOR: - URLs returned by `generate_image`, `generate_video`, or `generate_multishot_video`. Those URLs are already stable Runway-hosted assets — pass them directly as `startFrame.url` / `referenceImages[].url` / `referenceVideo.url`. Re-uploading wastes a turn and creates a duplicate asset. - Public HTTPS image URLs: pass them directly to the generation/editing tool. CLAUDE/CURSOR/LOCAL WORKFLOW: 1. Get the file size and MIME type using bash: ```bash wc -c < "/absolute/path/to/your/file" | tr -d ' ' file --mime-type -b "/absolute/path/to/your/file" ``` 2. Call this tool with the filename, fileSize, and mimeType. 3. Use bash/curl to upload the file to the temporary upload URL(s). The curl commands this tool returns include `-D -` so the response headers (including ETag) are written to stdout. 4. Capture the ETag header from each PUT response (it looks like `etag: "abc123..."`; strip the surrounding quotes). 5. Call `complete_upload` with the uploadId AND the parts array — `parts` is required even for single-part uploads. 6. Use the returned assetUrl as startFrame.url / referenceImages[].url / referenceVideo.url in the generation tools. Parameters: - file: ChatGPT-only uploaded file param `{download_url, file_id, mime_type, file_name}`. If present, the server completes the upload and returns an asset URL directly. - filename: Name for the uploaded file (e.g., "video.mp4") - fileSize: Size in bytes (from wc -c) - mimeType: MIME type (from file --mime-type). Supported: image/jpeg, image/png, image/webp, image/gif, video/mp4, video/quicktime, video/webm
Finalize a file upload and get the asset URL. Call this AFTER successfully uploading the file bytes to the temporary upload URL(s) from `init_upload`. USER-FACING REPLIES: Do not mention provider-specific storage services. Say "Runway upload", "Runway-hosted asset", or "temporary upload URL" instead. `parts` is REQUIRED — even for single-part uploads. Each entry corresponds to one PUT response and carries the ETag from that response's `etag` header (strip surrounding quotes if present). Returns the asset URL to use as `startFrame` / `endFrame` / `referenceImages[].url` in the generation tools, or `referenceVideo.url` in `generate_video` for video-to-video edits.
Import an MP4 video into Runway from a URL.
Lists recent uploaded and generated assets for the authenticated workspace. Returns asset IDs, media types, task IDs when available, and reusable asset URLs. Use this to help the user pick recent images or videos as references.
Gets details for a Runway task by ID — used to check status and retrieve the result of a generation/edit task once it completes. Generation tools (`generate_image`, `generate_video`, `generate_multishot_video`, `run_workflow`) return immediately with a task ID. If an inline viewer rendered, do not poll; the viewer updates itself. If no viewer rendered, poll this tool until SUCCEEDED (wait 60-120s between calls for video/workflow). For `dynamic_workflow` tasks, the text includes overall %, per-step name/status, and output URLs when nodes finish. Parent `dynamic_workflow` SUCCEEDED can have empty artifacts — use the step Outputs, not artifacts[0]. Set `include_workflow_execution: true` only when you need node UUIDs. When status is SUCCEEDED the response includes the asset URL. USER-FACING REPLIES: describe the asset content; the `taskType` in the response encodes the model name and is for debugging only — keep it out of your reply unless the user explicitly asks which model was used.
Shows the connected workspace's remaining Runway credits plus available plans and top-up options as an inline card. Call it when the user asks about their credits, balance, plans, pricing, or how to buy more credits — and after a generation fails with an out-of-credits or low-credits result. After an out-of-credits failure, pass `creditsNeeded` (the approximate credit cost of that generation, from the failed tool's cost notes) to render a shortfall card. After a generation fails because the plan does not allow it, call it with `gatedCapability` set to the phrase from that error to render an upgrade card. Purchases are completed on runway.com; the card only links out. For Max workspaces, returns a single sentence with an account-management link instead of an inline card. When the inline card renders, keep your reply to one short sentence — the card already shows the numbers and buttons.
Records one interaction with the Runway home credits or account controls.
Cancels the free Runway Pro trial so it is not charged. The workspace keeps Pro until the trial end date. Only call this when the user explicitly asks to cancel their trial.
Record a diagnostic report about a Runway tool problem in this MCP server's logs. Examples of when to call: - A generation failed or produced unexpected results - You couldn't figure out which tool to use - The user said the result wasn't what they wanted - The prompting guide didn't have the info you needed The Runway team can inspect these logs to investigate tool problems. Be specific about what went wrong.
Records a privacy-safe interaction from the Runway media viewer.
