For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our blog, GitHub, and Documentation.

Performance

\#: Results were generated using GPT-4o-20241120, as access to the native function calling API of GPT-4o-0327 was unavailable.

Quickstart

The following contains a code snippet illustrating how to use the model generate content based on given inputs.

from transformers import AutoModelForCausalLM, AutoTokenizer


model_name = "Qwen/Qwen3-235B-A22B-Instruct-2507"

load the tokenizer and the model
tokenizer = AutoTokenizer.frompretrained(modelname)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

prepare the model input
prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.applychattemplate(
    messages,
    tokenize=False,
    addgenerationprompt=True,
)
modelinputs = tokenizer([text], returntensors="pt").to(model.device)

conduct text completion
generated_ids = model.generate(
    model_inputs,
    maxnewtokens=16384
)
outputids = generatedids[0][len(modelinputs.inputids[0]):].tolist()

content = tokenizer.decode(outputids, skipspecial_tokens=True)

print("content:", content)

python -m sglang.launch_server --model-path Qwen/Qwen3-235B-A22B-Instruct-2507 --tp 8 --context-length 262144

• vLLM:

vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507 --tensor-parallel-size 8 --max-model-len 262144

For local use, applications such as Ollama, LMStudio, MLX-LM, llama.cpp, and KTransformers have also supported Qwen3.

Agentic Use

Qwen3 excels in tool calling capabilities. We recommend using Qwen-Agent to make the best use of agentic ability of Qwen3. Qwen-Agent encapsulates tool-calling templates and tool-calling parsers internally, greatly reducing coding complexity.

To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by yourself.

from qwen_agent.agents import Assistant


Define LLM
llm_cfg = {
    'model': 'Qwen3-235B-A22B-Instruct-2507',

# Use a custom endpoint compatible with OpenAI API:
    'modelserver': 'http://localhost:8000/v1',  # apibase
    'api_key': 'EMPTY',
}

Define Tools
tools = [
    {'mcpServers': {  # You can specify the MCP configuration file
            'time': {
                'command': 'uvx',
                'args': ['mcp-server-time', '--local-timezone=Asia/Shanghai']
            },
            "fetch": {
                "command": "uvx",
                "args": ["mcp-server-fetch"]
            }
        }
    },
  'code_interpreter',  # Built-in tools
]

Define Agent
bot = Assistant(llm=llmcfg, functionlist=tools)

Streaming generation
messages = [{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}]
for responses in bot.run(messages=messages):
    pass
print(responses)

Best Practices

To achieve optimal performance, we recommend the following settings:

1. Sampling Parameters:

1. Adequate Output Length: We recommend using an output length of 16,384 tokens for most queries, which is adequate for instruct models.
2. Standardize Output Format: We recommend using prompts to standardize model outputs when benchmarking.