Audio-Visual Memory

Check the qwen-mm-plugins-omni-memory tools in your tool list for full schemas and parameters.

First decide whether this video needs a memory at all

Building one costs an omni call per 30s window, so over a short video it spends N calls indexing what fits in a single request. Decide by length, and by whether the user wants a memory.

How you learn the length: get_memory_status(video_path=...). It reports duration_min for the source video even when no memory exists, and its next_step already applies the table below — so the call you had to make anyway also answers this. (It reads the file with ffprobe; if that is unavailable the field is absent and building is the safe default.)

Two things that override the table:

• Several questions about the same video → build the memory, even a short one. A watch is stateless: every question re-uploads and re-watches, while a memory is paid for once and then answers for free. One question is cheaper as a watch; a conversation is cheaper as a memory.
• A memory already exists → use it. watch_and_answer re-uploads the video and sees only what is in it; the memory already holds every utterance with its speaker, and answers in milliseconds.

If a watch cannot get through, its result says so explicitly: fallback: "build_memory" plus the exact command. Run that and continue with the flow below. When the failure is throttling or a misconfigured endpoint there is no fallback field — retry or fix the configuration instead, because a build would hit the very same wall.

General Workflow

Everything below is the memory path. Memory must exist before you can query it. When a user asks about an audio-video, you are the one who retrieves memory and answers — every tool here returns evidence, not an answer.

1. get_memory_status(video_path=...) — always start here. It reports one of:
   • exists: false → also carries duration_min, and a next_step that applies the routing table above: short enough to watch instead, or build it (see "Building memory")
   • exists: true, complete: true → query it
   • exists: true, complete: false → truncated. Its answers cannot be trusted. An interrupted build still finalizes the library, so it looks normal otherwise; continue it with --mode resume
2. Query it (see "Retrieval workflow")
3. Answer from what you retrieved. The record already carries the audio's content — every utterance with its speaker, how it was said, and the non-speech sound around it — so this is where most questions end.

replay_and_answer is not step 4. It is an exception off the side of this flow, taken only when you can name a specific thing the record does not contain — and it re-watches only the few clips you name, so on everything else it knows less about the video than the memory does.

The Memory Layout