FilmAgent

AI co-pilot for filmmakers

Solution overview for Sales and Solution Architects

What FilmAgent is

FilmAgent is an AI co-pilot for filmmakers. It covers screenwriting, asset generation, specialized AI skills and shot editing, and takes the existing project into account.

It works through natural language and precise editing.

Technically: built on Pi, a minimal harness framework. It uses ByteDance models (Seedance, Seedream, Seed) and specialized Skills written from best practice.
  • FilmAgent: ask in words; the agent does the work.
  • Studio: the same film, stage by stage.
  • Canvas: card-by-card control of one shot or edit.
FilmAgent: session on the left, Generations on the right

Target customers

1 · AI-native film companies

"We want a Claude Code-like solution, but for film making."
"We want to build our platform on top of your workflows and use your best practices."
"We want agentic workflows." / "We want to use agents."

2 · Media platforms that need agentic solutions

Their end users have reference images, sketches, storyboards and reference videos, but don't know how to use AI tools.
FilmAgent is relevant to API customers who purchased Advanced Creation Rights.

Scenarios

Seven scenarios: video editing, cinematic video generation, previz, asset generation, interactive storyboarding, directing, audio design

The harness what sits between the director and the models

The harness is the agent loop that turns a request into finished work. For each request it plans the steps, calls one film tool per step, looks at what came back and decides the next step, until it can report. There is no fixed pipeline.

DirectorAsks in plain words, attaches references, names results as R1, T3 or @Name.
→
PlanThe planner model reads the request and the film, and chooses the next tool.
→
ActOne tool call: write, draw, shoot, edit, finish or assemble.
→
LookIt reads the result, including frames of every picture and take.
→
ReportWhat is done, what is not and why. The director picks.
Plan → Act → Look repeats until the request is done or a limit is reached
30 film toolsScreenplay, shot list, cast, variations, storyboard, previz, video, edit, finish, assemble.
SkillsThe agent's method, each model's prompt spec and each tool's prompts, as plain files.
Project contextThe cloud film is the memory: script, shots, plates and takes.
Run rulesLimits and checks in code that hold whatever the model decides.

What the harness holds

Tools30 film tools the agent can call. Each does one job and returns a named result: R for pictures, T for takes.
SkillsPrompts are written under each model's own spec (Seedance 2.5, Seedream, editing). Tool prompts and the agent's method are files a customer can read and change.
ContextOne cloud project per film. The agent reads only what a step needs, so a long project does not have to be reloaded into the model each time.
Assets by nameR1, T3 and @Name resolve to the right file and are numbered for the model. A prompt that cites an asset not in the render is refused in code.
Run rulesUp to 60 tool calls per request. A tool that refuses the same thing twice is closed for that request. A render's prompt must come from the prompt-writing step.
Director in the loopEvery step shows as it happens. The director can add a note while it works, stop it, and pick or reject results. Renders already started still land in the film.
ModelsPlanner: Seed 2.1 Turbo by default, Seed 2.0 Pro or Seed-Evolving selectable. Video: Seedance 2.5. Images: Seedream. Each slot is set by an environment variable.

Harness benefits

No workflow to buildA new kind of request needs no new pipeline. The agent composes the steps from the tools it has.
Model best practice built inPrompts follow each model's spec, so end users don't need prompt-writing skill to get a usable take.
Spend stays controlledDrafts render at 480p first and only picks are finished. Limits cap what one request can use.
Results can be auditedEvery step is visible, and the exact prompt is saved on every result.
Failures are reportedA failed render comes back with its reason. Nothing is retried or substituted silently.
Customizable, white labelSkills are files. Customers can add their own formats and rules, and build their product on the same blocks.
One film, three surfacesFilmAgent, Studio and Canvas open the same project, so work started in chat can be finished by hand.
The director keeps controlThe agent reports what it saw. Picking, rejecting and notes stay with the person.
Works on existing materialReferences, sketches, storyboards and footage can be attached and used by name.

Architecture Harness, Studio and Canvas

FilmAgent is a white-label solution: building blocks for the film industry that combine the Harness, Studio and Canvas spaces.

  • Harness (FilmAgent): one session drives the film through tools.
  • Cloud project: one film shared by all three.
  • Studio: Script, Cast & World, Scenes, Cut.
  • Canvas: Shot card, Previz, Edit card, Take Library, Final Cut.
FilmAgent pipeline diagram

Video editing

The agent edits existing footage of up to 30 seconds.

Scenarios

  • Background replacement, face swap
  • Outfit change, relighting
  • Dialogue line change, audio replacement
  • New endings
  • Equipment clean-up
  • Mask generation, blockout video
  • In-place editing for hybrid AI films
FilmAgent editing a take: a camel added to the background
The director asks for a change to take T6 in words; the agent edits it into T8.

Cinematic video generation FilmAgent

Generate as many takes as needed and steer them in natural language.

  • Automatic prompt writing via AI skills
  • Draft mode: 480p first, then the final
  • Takes up to 30 seconds, with sound
  • The agent reports what it saw in each take
FilmAgent video generation: the ask, the steps, three takes

Cinematic video generation Canvas Shot card

Canvas Shot card
The Shot card builds a prompt from references and launches the takes: model, length, ratio, keyframes, first frame lock.
A take from the Shot card, 15 s

Previz and animatics

The agent turns an idea into an overhead diagram of the blocking, then into an animatic.

Previz schematic
Schematic: set, actors, moves, cameras
Animatic from that schematic, 28 s
Previz agent on the Canvas
The Previz agent on the Canvas
Its animatic

Asset generation the Bible

This Canvas agent turns a script or brief into a Bible: the set of references reused across the whole project.

  • Character sheets
  • Locations
  • Props

In FilmAgent the same job is one request: "Cast it."

Cast and World on the Canvas
Cast and World with takes

Asset variations

The Variations agent changes what an asset looks like while keeping the structure of the reference: characters, locations, camera angles, lighting.

Character variations
Character: ten outfits from one sheet
Location variations
Location: angles and times of day
Variations placed in the Bible
Variations sit beside the plate

Interactive storyboarding from a brief

Quick storyboardOne sheet with several panels that tell the story.
Cinematic storyboardStart and end stills per shot: a closer look at each shot, and the keyframes it will use.

Each row can be edited and re-rendered, then sent to a Shot card.

Storyboard strip with start and end stills
Storyboard on the Canvas

Directing: control and masking

Control inputs reduce the error rate in complex shots, such as multi-character scenes or planned camera moves.

  • Masking: people become flat colour silhouettes
  • Editing: draw on a picture to say where
  • Directing: blocking, keyframes and reference video steer the take
A still
The same still with the figure as a silhouette
A still and its people mask

Audio design

From an image and a prompt, the Audio agent on the Canvas generates sound with Seed Audio 1.0.

  • Ambient sounds
  • Voice-over
  • SFX

Seedance takes also come with their own sound.

Audio agent on the Canvas

Skills

Skills let FilmAgent solve the request and get the most out of each model without pre-defined workflows. Customers can create new skills and change existing ones.

  • Model skills: best-practice prompt writing for each model, bound to its model slots.
  • FilmAgent: the agent's own method.
  • Tool prompts: shared by FilmAgent, Studio and Canvas.
Skills and prompts: model skills
Skills and prompts: tool prompts

How to install API key and endpoints

git clone git@github.com:RaiaN/FilmAgent.git && cd FilmAgent
npm install && cp .env.example .env.local

Open .env.local and fill in:

# required
MODELARK_API_KEY, MODELARK_API_BASE_URL
MODELARK_MODEL_REASONER, MODELARK_MODEL_SEEDREAM, MODELARK_MODEL_SEEDREAM_PRO
MODELARK_ASSET_ACCESS_KEY, MODELARK_ASSET_SECRET_KEY
MODELARK_TOS_BUCKET, MODELARK_TOS_REGION
# add for Seedance 2.5 (30 s takes, MOV 10-bit)
MODELARK_MODEL_SEEDANCE_25=ep-...
MODELARK_MODEL_SEEDANCE_25_PREMIUM=ep-...
# optional
MODELARK_MODEL_SEEDANCE, _FAST, _MINI
BYTEPLUSVOICE_API_KEY, MODELARK_MODEL_SEED_AUDIO

Needs Node 18+ and git access to the repository.

Then start it:

NODE_USE_SYSTEM_CA=1 npm run dev

Open http://localhost:3000.

Projects and media are stored in the TOS bucket named in the file.

Materials

Sizzle reel, 30 s
One-pagerOpen
Images and videos in this deck were made with Seedream and Seedance.