• Qwen/Qwen3-Omni-30B-A3B-Instruct

• en
• fr
• de
• es
• it
• pt
• ru

• gguf

Qwen3-Omni

Overview

Introduction

Qwen3-Omni is the natively end-to-end multilingual omni-modal foundation models. It processes text, images, audio, and video, and delivers real-time streaming responses in both text and natural speech. We introduce several architectural upgrades to improve performance and efficiency. Key features:

• State-of-the-art across modalities: Early text-first pretraining and mixed multimodal training provide native multimodal support. While achieving strong audio and audio-video results, unimodal text and image performance does not regress. Reaches SOTA on 22 of 36 audio/video benchmarks and open-source SOTA on 32 of 36; ASR, audio understanding, and voice conversation performance is comparable to Gemini 2.5 Pro.
• Multilingual: Supports 119 text languages, 19 speech input languages, and 10 speech output languages.

• Novel Architecture: MoE-based Thinker–Talker design with AuT pretraining for strong general representations, plus a multi-codebook design that drives latency to a minimum.
• Real-time Audio/Video Interaction: Low-latency streaming with natural turn-taking and immediate text or speech responses.
• Flexible Control: Customize behavior via system prompts for fine-grained control and easy adaptation.
• Detailed Audio Captioner: Qwen3-Omni-30B-A3B-Captioner is now open source: a general-purpose, highly detailed, low-hallucination audio captioning model that fills a critical gap in the open-source community.

Model Architecture

Cookbooks for Usage Cases

Qwen3-Omni supports a wide range of multimodal application scenarios, covering various domain tasks involving audio, image, video, and audio-visual modalities. Below are several cookbooks demonstrating the usage cases of Qwen3-Omni and these cookbooks include our actual execution logs. You can first follow the QuickStart guide to download the model and install the necessary inference environment dependencies, then run and experiment locally—try modifying prompts or switching model types, and enjoy exploring the capabilities of Qwen3-Omni!

Category	Cookbook	Description	Open
Audio	Speech Recognition	Speech recognition, supporting multiple languages and long audio.
	Speech Translation	Speech-to-Text / Speech-to-Speech translation.
	Music Analysis	Detailed analysis and appreciation of any music, including style, genre, rhythm, etc.
	Sound Analysis	Description and analysis of various sound effects and audio signals.
	Audio Caption	Audio captioning, detailed description of any audio input.
	Mixed Audio Analysis	Analysis of mixed audio content, such as speech, music, and environmental sounds.
Visual	OCR	OCR for complex images.
	Object Grounding	Target detection and grounding.
	Image Question	Answering arbitrary questions about any image.
	Image Math	Solving complex mathematical problems in images, highlighting the capabilities of the Thinking model.
	Video Description	Detailed description of video content.
	Video Navigation	Generating navigation commands from first-person motion videos.
	Video Scene Transition	Analysis of scene transitions in videos.
Audio-Visual	Audio Visual Question	Answering arbitrary questions in audio-visual scenarios, demonstrating the model's ability to model temporal alignment between audio and video.
	Audio Visual Interaction	Interactive communication with the model using audio-visual inputs, including task specification via audio.
	Audio Visual Dialogue	Conversational interaction with the model using audio-visual inputs, showcasing its capabilities in casual chat and assistant-like behavior.
Agent	Audio Function Call	Using audio input to perform function calls, enabling agent-like behaviors.
Downstream Task Fine-tuning	Omni Captioner	Introduction and capability demonstration of Qwen3-Omni-30B-A3B-Captioner, a downstream fine-tuned model based on Qwen3-Omni-30B-A3B-Instruct, illustrating the strong generalization ability of the Qwen3-Omni foundation model.

QuickStart

Model Description and Download

Below is the description of all Qwen3-Omni models. Please select and download the model that fits your needs.

# Download through ModelScope (recommended for users in Mainland China)
pip install -U modelscope
modelscope download --model Qwen/Qwen3-Omni-30B-A3B-Instruct --local_dir ./Qwen3-Omni-30B-A3B-Instruct
modelscope download --model Qwen/Qwen3-Omni-30B-A3B-Thinking --local_dir ./Qwen3-Omni-30B-A3B-Thinking
modelscope download --model Qwen/Qwen3-Omni-30B-A3B-Captioner --local_dir ./Qwen3-Omni-30B-A3B-Captioner

Download through Hugging Face
pip install -U "huggingface_hub[cli]"
huggingface-cli download Qwen/Qwen3-Omni-30B-A3B-Instruct --local-dir ./Qwen3-Omni-30B-A3B-Instruct
huggingface-cli download Qwen/Qwen3-Omni-30B-A3B-Thinking --local-dir ./Qwen3-Omni-30B-A3B-Thinking
huggingface-cli download Qwen/Qwen3-Omni-30B-A3B-Captioner --local-dir ./Qwen3-Omni-30B-A3B-Captioner

Transformers Usage

#### Installation

The Hugging Face Transformers code for Qwen3-Omni has been successfully merged, but the PyPI package has not yet been released. Therefore, you need to install it from source using the following command. We strongly recommend that you create a new Python environment to avoid environment runtime issues.

# If you already have transformers installed, please uninstall it first, or create a new Python environment
pip uninstall transformers
pip install git+https://github.com/huggingface/transformers
pip install accelerate

We offer a toolkit to help you handle various types of audio and visual input more conveniently, providing an API-like experience. This includes support for base64, URLs, and interleaved audio, images, and videos. You can install it using the following command and make sure your system has ffmpeg installed:

pip install qwen-omni-utils -U

Additionally, we recommend using FlashAttention 2 when running with Hugging Face Transformers to reduce GPU memory usage. However, if you are primarily using vLLM for inference, this installation is not necessary, as vLLM includes FlashAttention 2 by default.

pip install -U flash-attn --no-build-isolation

Also, you should have hardware that is compatible with FlashAttention 2. Read more about it in the official documentation of the FlashAttention repository. FlashAttention 2 can only be used when a model is loaded in torch.float16 or torch.bfloat16.

#### Code Snippet

Here is a code snippet to show you how to use Qwen3-Omni with transformers and qwenomniutils:

import soundfile as sf

from transformers import Qwen3OmniMoeForConditionalGeneration, Qwen3OmniMoeProcessor
from qwenomniutils import processmminfo

MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Thinking"


model = Qwen3OmniMoeForConditionalGeneration.from_pretrained(
    MODEL_PATH,
    dtype="auto",
    device_map="auto",
    attnimplementation="flashattention_2",
)

processor = Qwen3OmniMoeProcessor.frompretrained(MODELPATH)

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/cars.jpg"},
            {"type": "audio", "audio": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/cough.wav"},
            {"type": "text", "text": "What can you see and hear? Answer in one short sentence."}
        ],
    },
]

Set whether to use audio in video
USEAUDIOIN_VIDEO = True

Preparation for inference
text = processor.applychattemplate(conversation, addgenerationprompt=True, tokenize=False)
audios, images, videos = processmminfo(conversation, useaudioinvideo=USEAUDIOINVIDEO)
inputs = processor(text=text, 
                   audio=audios, 
                   images=images, 
                   videos=videos, 
                   return_tensors="pt", 
                   padding=True, 
                   useaudioinvideo=USEAUDIOINVIDEO)
inputs = inputs.to(model.device).to(model.dtype)

Inference: Generation of the output text and audio
text_ids, audio = model.generate(inputs, 
                                 speaker="Ethan", 
                                 thinkerreturndictingenerate=True,
                                 useaudioinvideo=USEAUDIOINVIDEO)

text = processor.batchdecode(textids.sequences[:, inputs["input_ids"].shape[1] :],
                              skipspecialtokens=True,
                              cleanuptokenization_spaces=False)
print(text)
if audio is not None:
    sf.write(
        "output.wav",
        audio.reshape(-1).detach().cpu().numpy(),
        samplerate=24000,
    )

Here are some more advanced usage examples. You can expand the sections below to learn more.

from transformers import Qwen3OmniMoeForConditionalGeneration, Qwen3OmniMoeProcessor
from qwenomniutils import processmminfo

MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Thinking"


model = Qwen3OmniMoeForConditionalGeneration.from_pretrained(
    MODEL_PATH,
    dtype="auto",
    device_map="auto",
    attnimplementation="flashattention_2",
)
model.disable_talker()

processor = Qwen3OmniMoeProcessor.frompretrained(MODELPATH)

Conversation with image only
conversation1 = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/cars.jpg"},
            {"type": "text", "text": "What can you see in this image? Answer in one sentence."},
        ]
    }
]

Conversation with audio only
conversation2 = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/cough.wav"},
            {"type": "text", "text": "What can you hear in this audio?"},
        ]
    }
]

Conversation with pure text and system prompt
conversation3 = [
    {
        "role": "system",
        "content": [
            {"type": "text", "text": "You are Qwen-Omni."}
        ],
    },
    {
        "role": "user",
        "content": "Who are you?"
    }
]

Conversation with mixed media
conversation4 = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/cars.jpg"},
            {"type": "audio", "audio": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/cough.wav"},
            {"type": "text", "text": "What can you see and hear? Answer in one sentence."}
        ],
    }
]

Combine messages for batch processing
conversations = [conversation1, conversation2, conversation3, conversation4]

Set whether to use audio in video
USEAUDIOIN_VIDEO = True

Preparation for batch inference
text = processor.applychattemplate(conversations, addgenerationprompt=True, tokenize=False)
audios, images, videos = processmminfo(conversations, useaudioinvideo=USEAUDIOINVIDEO)

inputs = processor(text=text, 
                   audio=audios, 
                   images=images, 
                   videos=videos, 
                   return_tensors="pt", 
                   padding=True, 
                   useaudioinvideo=USEAUDIOINVIDEO)
inputs = inputs.to(model.device).to(model.dtype)

Batch inference does not support returning audio
text_ids, audio = model.generate(inputs,
                                 return_audio=False,
                                 thinkerreturndictingenerate=True,
                                 useaudioinvideo=USEAUDIOINVIDEO)

text = processor.batchdecode(textids.sequences[:, inputs["input_ids"].shape[1] :],
                              skipspecialtokens=True,
                              cleanuptokenization_spaces=False)
print(text)

model = Qwen3OmniMoeForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3-Omni-30B-A3B-Instruct",
    dtype="auto",
    device_map="auto",
    attnimplementation="flashattention_2",
)
model.disable_talker()

model = Qwen3OmniMoeForConditionalGeneration.from_pretrained(
    "Qwen/Qwen3-Omni-30B-A3B-Instruct",
    dtype="auto",
    device_map="auto",
    attnimplementation="flashattention_2",
)
...
textids,  = model.generate(..., return_audio=False)

text_ids, audio = model.generate(..., speaker="Ethan")

text_ids, audio = model.generate(..., speaker="Chelsie")

text_ids, audio = model.generate(..., speaker="Aiden")

vLLM Usage

#### Installation

We strongly recommend using vLLM for inference and deployment of the Qwen3-Omni series models. Since our code is currently in the pull request stage, and audio output inference support for the Instruct model will be released in the near future, you can follow the commands below to install vLLM from source. Please note that we recommend you create a new Python environment to avoid runtime environment conflicts and incompatibilities. For more details on compiling vLLM from source, please refer to the vLLM official documentation.

git clone -b qwen3_omni https://github.com/wangxiongts/vllm.git
cd vllm
pip install -r requirements/build.txt
pip install -r requirements/cuda.txt
export VLLMPRECOMPILEDWHEELLOCATION=https://wheels.vllm.ai/a5dd03c1ebc5e4f56f3c9d3dc0436e9c582c978f/vllm-0.9.2-cp38-abi3-manylinux1x86_64.whl
VLLMUSEPRECOMPILED=1 pip install -e . -v --no-build-isolation
If you meet an "Undefined symbol" error while using VLLMUSEPRECOMPILED=1, please use "pip install -e . -v" to build from source.
Install the Transformers
pip install git+https://github.com/huggingface/transformers
pip install accelerate
pip install qwen-omni-utils -U
pip install -U flash-attn --no-build-isolation

#### Inference

You can use the following code for vLLM inference. The limitmmperprompt parameter specifies the maximum number of each modality's data allowed per message. Since vLLM needs to pre-allocate GPU memory, larger values will require more GPU memory; if OOM issues occur, try reducing this value. Setting tensorparallelsize greater than one enables multi-GPU parallel inference, improving concurrency and throughput. In addition, maxnumseqs indicates the number of sequences that vLLM processes in parallel during each inference step. A larger value requires more GPU memory but enables higher batch inference speed. For more details, please refer to the vLLM official documentation. Below is a simple example of how to run Qwen3-Omni with vLLM:

import os
import torch

from vllm import LLM, SamplingParams
from transformers import Qwen3OmniMoeProcessor
from qwenomniutils import processmminfo

if name == 'main':
    # vLLM engine v1 not supported yet
    os.environ['VLLMUSEV1'] = '0'

MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
    # MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Thinking"

llm = LLM(
            model=MODELPATH, trustremotecode=True, gpumemory_utilization=0.95,
            tensorparallelsize=torch.cuda.device_count(),
            limitmmper_prompt={'image': 3, 'video': 3, 'audio': 3},
            maxnumseqs=8,
            maxmodellen=32768,
            seed=1234,
    )

sampling_params = SamplingParams(
        temperature=0.6,
        top_p=0.95,
        top_k=20,
        max_tokens=16384,
    )

processor = Qwen3OmniMoeProcessor.frompretrained(MODELPATH)

messages = [
        {
            "role": "user",
            "content": [
                {"type": "video", "video": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/draw.mp4"}
            ], 
        }
    ]

text = processor.applychattemplate(
        messages,
        tokenize=False,
        addgenerationprompt=True,
    )
    audios, images, videos = processmminfo(messages, useaudioin_video=True)

inputs = {
        'prompt': text,
        'multimodaldata': {},
        "mmprocessorkwargs": {
            "useaudioin_video": True,
        },
    }

if images is not None:
        inputs['multimodaldata']['image'] = images
    if videos is not None:
        inputs['multimodaldata']['video'] = videos
    if audios is not None:
        inputs['multimodaldata']['audio'] = audios

outputs = llm.generate([inputs], samplingparams=samplingparams)

print(outputs[0].outputs[0].text)

import os
import torch

from vllm import LLM, SamplingParams
from transformers import Qwen3OmniMoeProcessor
from qwenomniutils import processmminfo

def buildinput(processor, messages, useaudioinvideo):
    text = processor.applychattemplate(
        messages,
        tokenize=False,
        addgenerationprompt=True,
    )
    audios, images, videos = processmminfo(messages, useaudioinvideo=useaudioinvideo)

inputs = {
        'prompt': text,
        'multimodaldata': {},
        "mmprocessorkwargs": {
            "useaudioinvideo": useaudioinvideo,
        },
    }

if images is not None:
        inputs['multimodaldata']['image'] = images
    if videos is not None:
        inputs['multimodaldata']['video'] = videos
    if audios is not None:
        inputs['multimodaldata']['audio'] = audios

return inputs

if name == 'main':
    # vLLM engine v1 not supported yet
    os.environ['VLLMUSEV1'] = '0'

MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
    # MODEL_PATH = "Qwen/Qwen3-Omni-30B-A3B-Thinking"

llm = LLM(
            model=MODELPATH, trustremotecode=True, gpumemory_utilization=0.95,
            tensorparallelsize=torch.cuda.device_count(),
            limitmmper_prompt={'image': 3, 'video': 3, 'audio': 3},
            maxnumseqs=8,
            maxmodellen=32768,
            seed=1234,
    )

sampling_params = SamplingParams(
        temperature=0.6,
        top_p=0.95,
        top_k=20,
        max_tokens=16384,
    )

processor = Qwen3OmniMoeProcessor.frompretrained(MODELPATH)

# Conversation with image only
    conversation1 = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/cars.jpg"},
                {"type": "text", "text": "What can you see in this image? Answer in one sentence."},
            ]
        }
    ]

# Conversation with audio only
    conversation2 = [
        {
            "role": "user",
            "content": [
                {"type": "audio", "audio": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/cough.wav"},
                {"type": "text", "text": "What can you hear in this audio?"},
            ]
        }
    ]

# Conversation with pure text and system prompt
    conversation3 = [
        {
            "role": "system",
            "content": [
                {"type": "text", "text": "You are Qwen-Omni."}
            ],
        },
        {
            "role": "user",
            "content": "Who are you? Answer in one sentence."
        }
    ]

# Conversation with mixed media
    conversation4 = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/cars.jpg"},
                {"type": "audio", "audio": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/cookbook/asr_fr.wav"},
                {"type": "text", "text": "What can you see and hear? Answer in one sentence."}
            ],
        }
    ]

USEAUDIOIN_VIDEO = True

# Combine messages for batch processing
    conversations = [conversation1, conversation2, conversation3, conversation4]
    inputs = [buildinput(processor, messages, USEAUDIOINVIDEO) for messages in conversations]

outputs = llm.generate(inputs, samplingparams=samplingparams)

result = [outputs[i].outputs[0].text for i in range(len(outputs))]
    print(result)

# Qwen3-Omni-30B-A3B-Instruct for single GPU
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --port 8901 --host 127.0.0.1 --dtype bfloat16 --max-model-len 32768 --allowed-local-media-path / -tp 1
Qwen3-Omni-30B-A3B-Instruct for multi-GPU (example on 4 GPUs)
vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --port 8901 --host 127.0.0.1 --dtype bfloat16 --max-model-len 65536 --allowed-local-media-path / -tp 4
Qwen/Qwen3-Omni-30B-A3B-Thinking for single GPU
vllm serve Qwen/Qwen3-Omni-30B-A3B-Thinking --port 8901 --host 127.0.0.1 --dtype bfloat16 --max-model-len 32768 --allowed-local-media-path / -tp 1
Qwen/Qwen3-Omni-30B-A3B-Thinking for multi-GPU (example on 4 GPUs)
vllm serve Qwen/Qwen3-Omni-30B-A3B-Thinking --port 8901 --host 127.0.0.1 --dtype bfloat16 --max-model-len 65536 --allowed-local-media-path / -tp 4

curl http://localhost:8901/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": [
        {"type": "imageurl", "imageurl": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/cars.jpg"}},
        {"type": "audiourl", "audiourl": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/cough.wav"}},
        {"type": "text", "text": "What can you see and hear? Answer in one sentence."}
    ]}
    ]
    }'

usersystemprompt = "You are Qwen-Omni, a smart voice assistant created by Alibaba Qwen."
message = {
    "role": "system",
    "content": [
          {"type": "text", "text": f"{usersystemprompt} You are a virtual voice assistant with no gender or age.\nYou are communicating with the user.\nIn user messages, “I/me/my/we/our” refer to the user and “you/your” refer to the assistant. In your replies, address the user as “you/your” and yourself as “I/me/my”; never mirror the user’s pronouns—always shift perspective. Keep original pronouns only in direct quotes; if a reference is unclear, ask a brief clarifying question.\nInteract with users using short(no more than 50 words), brief, straightforward language, maintaining a natural tone.\nNever use formal phrasing, mechanical expressions, bullet points, overly structured language. \nYour output must consist only of the spoken content you want the user to hear. \nDo not include any descriptions of actions, emotions, sounds, or voice changes. \nDo not use asterisks, brackets, parentheses, or any other symbols to indicate tone or actions. \nYou must answer users' audio or text questions, do not directly describe the video content. \nYou should communicate in the same language strictly as the user unless they request otherwise.\nWhen you are uncertain (e.g., you can't see/hear clearly, don't understand, or the user makes a comment rather than asking a question), use appropriate questions to guide the user to continue the conversation.\nKeep replies concise and conversational, as if talking face-to-face."}
    ]
}

messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "/path/to/audio.wav"},
            {"type": "image", "image": "/path/to/image.png"},
            {"type": "video", "video": "/path/to/video.mp4"},
            {"type": "text", "text": "Analyze this audio, image, and video together."},
        ], 
    }
]

# In data preprocessing
audios, images, videos = processmminfo(messages, useaudioin_video=True)

# For Transformers
text = processor.applychattemplate(messages, addgenerationprompt=True, tokenize=False)
inputs = processor(text=text, audio=audios, images=images, videos=videos, return_tensors="pt", 
                   padding=True, useaudioin_video=True)
textids, audio = model.generate(..., useaudioinvideo=True)

For vLLM
text = processor.applychattemplate(messages, addgenerationprompt=True, tokenize=False)
inputs = {
    'prompt': text,
    'multimodaldata': {},
    "mmprocessorkwargs": {
        "useaudioin_video": True,
    },
}

messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "/path/to/audio.wav"},
            {"type": "image", "image": "/path/to/image.png"},
            {"type": "video", "video": "/path/to/video.mp4"},
            {"type": "text", "text": "Describe the audio, image and video."},
        ],
    },
]