For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our blog, GitHub, and Documentation.

Performance

\#: Results were generated using GPT-4o-20241120, as access to the native function calling API of GPT-4o-0327 was unavailable.

Quickstart

The following contains a code snippet illustrating how to use the model generate content based on given inputs.

from transformers import AutoModelForCausalLM, AutoTokenizer


model_name = "Qwen/Qwen3-30B-A3B-Instruct-2507"

load the tokenizer and the model
tokenizer = AutoTokenizer.frompretrained(modelname)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

prepare the model input
prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.applychattemplate(
    messages,
    tokenize=False,
    addgenerationprompt=True,
)
modelinputs = tokenizer([text], returntensors="pt").to(model.device)

conduct text completion
generated_ids = model.generate(
    model_inputs,
    maxnewtokens=16384
)
outputids = generatedids[0][len(modelinputs.inputids[0]):].tolist()

content = tokenizer.decode(outputids, skipspecial_tokens=True)

print("content:", content)

python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 --context-length 262144

• vLLM:

vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 --max-model-len 262144

For local use, applications such as Ollama, LMStudio, MLX-LM, llama.cpp, and KTransformers have also supported Qwen3.

Agentic Use

Qwen3 excels in tool calling capabilities. We recommend using Qwen-Agent to make the best use of agentic ability of Qwen3. Qwen-Agent encapsulates tool-calling templates and tool-calling parsers internally, greatly reducing coding complexity.

To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen-Agent, or integrate other tools by yourself.

from qwen_agent.agents import Assistant


Define LLM
llm_cfg = {
    'model': 'Qwen3-30B-A3B-Instruct-2507',

# Use a custom endpoint compatible with OpenAI API:
    'modelserver': 'http://localhost:8000/v1',  # apibase
    'api_key': 'EMPTY',
}

Define Tools
tools = [
    {'mcpServers': {  # You can specify the MCP configuration file
            'time': {
                'command': 'uvx',
                'args': ['mcp-server-time', '--local-timezone=Asia/Shanghai']
            },
            "fetch": {
                "command": "uvx",
                "args": ["mcp-server-fetch"]
            }
        }
    },
  'code_interpreter',  # Built-in tools
]

Define Agent
bot = Assistant(llm=llmcfg, functionlist=tools)

Streaming generation
messages = [{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}]
for responses in bot.run(messages=messages):
    pass
print(responses)

Processing Ultra-Long Texts

To support ultra-long context processing (up to 1 million tokens), we integrate two key techniques:

• Dual Chunk Attention (DCA): A length extrapolation method that splits long sequences into manageable chunks while preserving global coherence.
• MInference: A sparse attention mechanism that reduces computational overhead by focusing on critical token interactions.

Together, these innovations significantly improve both generation quality and inference efficiency for sequences beyond 256K tokens. On sequences approaching 1M tokens, the system achieves up to a 3× speedup compared to standard attention implementations.

For full technical details, see the Qwen2.5-1M Technical Report.

How to Enable 1M Token Context

[!NOTE]

To effectively process a 1 million token context, users will require approximately 240 GB of total GPU memory. This accounts for model weights, KV-cache storage, and peak activation memory demands.

#### Step 1: Update Configuration File

export MODELNAME=Qwen3-30B-A3B-Instruct-2507
huggingface-cli download Qwen/${MODELNAME} --local-dir ${MODELNAME}
mv ${MODELNAME}/config.json ${MODELNAME}/config.json.bak
mv ${MODELNAME}/config_1m.json ${MODELNAME}/config.json

#### Step 2: Launch Model Server

After updating the config, proceed with either vLLM or SGLang for serving the model.

#### Option 1: Using vLLM

To run Qwen with 1M context support:

git clone https://github.com/vllm-project/vllm.git
cd vllm
pip install -e .

Then launch the server with Dual Chunk Flash Attention enabled:

VLLMATTENTIONBACKEND=DUALCHUNKFLASHATTN VLLMUSE_V1=0 \
vllm serve ./Qwen3-30B-A3B-Instruct-2507 \
  --tensor-parallel-size 4 \
  --max-model-len 1010000 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 131072 \
  --enforce-eager \
  --max-num-seqs 1 \
  --gpu-memory-utilization 0.85

##### Key Parameters

First, clone and install the specialized branch:

git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[all]"

Launch the server with DCA support:

python3 -m sglang.launch_server \
    --model-path ./Qwen3-30B-A3B-Instruct-2507 \
    --context-length 1010000 \
    --mem-frac 0.75 \
    --attention-backend dualchunkflash_attn \
    --tp 4 \
    --chunked-prefill-size 131072

##### Key Parameters

1. Encountering the error: "The model's max sequence length (xxxxx) is larger than the maximum number of tokens that can be stored in the KV cache." or "RuntimeError: Not enough memory. Please try to increase --mem-fraction-static."

1. Encountering the error: "torch.OutOfMemoryError: CUDA out of memory."

1. Encountering the error: "Input prompt (xxxxx tokens) + lookahead slots (0) is too long and exceeds the capacity of the block manager." or "The input (xxx xtokens) is longer than the model's context length (xxx tokens)."

#### Long-Context Performance

We test the model on an 1M version of the RULER benchmark.

• All models are evaluated with Dual Chunk Attention enabled.
• Since the evaluation is time-consuming, we use 260 samples for each length (13 sub-tasks, 20 samples for each).

Best Practices

To achieve optimal performance, we recommend the following settings:

1. Sampling Parameters:

1. Adequate Output Length: We recommend using an output length of 16,384 tokens for most queries, which is adequate for instruct models.
2. Standardize Output Format: We recommend using prompts to standardize model outputs when benchmarking.

Citation

If you find our work helpful, feel free to give us a cite.

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report}, 
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.09388}, 
}

🚀 If you find these models useful

Help me test my AI-Powered Quantum Network Monitor Assistant with quantum-ready security checks:

👉 Quantum Network Monitor

The full Open Source Code for the Quantum Network Monitor Service available at my github repos ( repos with NetworkMonitor in the name) : Source Code Quantum Network Monitor. You will also find the code I use to quantize the models if you want to do it yourself GGUFModelBuilder

What I’m Testing

• Function calling against live network services
• How small can a model go while still handling:

🟡 TestLLM – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space):

• ✅ Zero-configuration setup
• ⏳ 30s load time (slow inference but no API costs) . No token limited as the cost is low.
• 🔧 Help wanted! If you’re into edge-device AI, let’s collaborate!

Other Assistants

• It performs very well but unfortunatly OpenAI charges per token. For this reason tokens usage is limited.
• Create custom cmd processors to run .net code on Quantum Network Monitor Agents
• Real-time network diagnostics and monitoring
• Security Audits
• Penetration testing (Nmap/Metasploit)

🔵 HugLLM – Latest Open-source models:

• 🌐 Runs on Hugging Face Inference API. Performs pretty well using the lastest models hosted on Novita.

💡 Example commands you could test: