Media Understanding

Use qwen-mm-plugins-api to understand media through configured model services. The tools are grouped by model family:

• VL model (Qwen-VL, OpenAI-compatible endpoint): vision_chat, ocr, grounding.
• Omni model (Qwen-Omni — the AV tools combine video frames and audio; the ASR/music tools focus on audio): omni_asr, omni_asr_timestamped, omni_multi_speaker_asr, omni_av_caption, omni_av_grounding, omni_av_counting, omni_music_caption.
• Other services: transcribe_audio (Qwen3-ASR), segmentation (a SAM3 server).

Check the qwen-mm-plugins-api tools in your tool list for full schemas and parameters. For file reading, rendering, or metadata inspection, use core.

When to Use Which Tool

VL model (single images/videos, spatial reasoning):

• Ask a VLM about images/videos (caption, VQA, free-form) → vision_chat
• Extract text from an image → ocr
• Detect/locate objects in an image (bounding boxes, spatial WHERE) → grounding. It sends EXIF-corrected pixels, so returned 0–1000 boxes address the displayed image and can be passed directly to core crop/draw_bbox or search image_search.

Omni model (audio + video together, temporal reasoning; clips up to a few minutes):

• Transcribe speech, plain text → omni_asr (one continuous string, no timestamps)
• Transcribe with timestamps → omni_asr_timestamped (granularity = sentence or word; also returns SRT)
• Who said what → omni_multi_speaker_asr (diarization: speaker labels + timestamps + SRT; pass num_speakers if known)
• Describe the content over time → omni_av_caption (timestamped Markdown descriptions of visual content, dialogue, music, and sounds)
• Find WHEN something happens → omni_av_grounding (natural-language query → matching time segments; temporal localization)
• Count how many times an event/object/action occurs → omni_av_counting (target → total + per-occurrence timestamps)
• Analyze / caption a music track → omni_music_caption (whole-track tags — genre / moods / instruments / key / time signature / vocal profile — plus a dense English caption for music generation; audio-only, no timestamps)

Other services:

• Segment objects in an image (masks) → segmentation
• Transcribe speech from audio/video, fast and long-file friendly → transcribe_audio

Tips

Vision chat: pass images/videos + text prompt. Model precedence is explicit model → QWEN_MM_API_VL_MODEL → qwen3.7-plus. Use dry_run=true to inspect payloads.

VL/Omni endpoints: default to DashScope (DASHSCOPE_BASE_URL and DASHSCOPE_API_KEY). Use the existing base_url, api_key, and model arguments for another compatible endpoint, including a self-hosted service. The endpoint must support the selected tool's media payload and model.

OrcaRouter: configure ORCAROUTER_API_KEY, then pass base_url="https://api.orcarouter.ai/v1" and a gateway model ID. The server selects that key automatically; an explicit api_key overrides it.

OpenRouter: configure OPENROUTER_API_KEY, then pass base_url="https://openrouter.ai/api/v1" and an OpenRouter model ID, such as qwen/qwen3.7-plus for images. The server selects that key automatically; an explicit api_key overrides it. For video, use videos with a suitable model such as qwen/qwen3.8-max-0902. Local sampled frames are sent as ordered images; direct video URLs require video support from the model and provider.

Grounding: returns normalized boxes (0–1000). Set return_img=true to get the annotated image back, or draw them yourself with core's draw_bbox.

ASR (transcribe_audio): accepts audio or video, auto-chunks long files. Formats: srt (default), text, json. Uses DashScope with DASHSCOPE_API_KEY; configured ASR_SERVER_URLS provide a self-hosted fallback when the key is absent or DashScope fails. Needs ffmpeg for audio extraction and chunking.