Quantization made by Richard Erkhov.

Github

Discord

Request more models

Qwen2.5-32B-Instruct-abliterated - GGUF

• Model creator: https://huggingface.co/huihui-ai/
• Original model: https://huggingface.co/huihui-ai/Qwen2.5-32B-Instruct-abliterated/

Name	Quant method	Size
Qwen2.5-32B-Instruct-abliterated.Q2K.gguf	Q2K	11.47GB
Qwen2.5-32B-Instruct-abliterated.IQ3XS.gguf	IQ3XS	12.76GB
Qwen2.5-32B-Instruct-abliterated.IQ3S.gguf	IQ3S	13.45GB
Qwen2.5-32B-Instruct-abliterated.Q3KS.gguf	Q3K_S	13.4GB
Qwen2.5-32B-Instruct-abliterated.IQ3M.gguf	IQ3M	13.79GB
Qwen2.5-32B-Instruct-abliterated.Q3K.gguf	Q3K	14.84GB
Qwen2.5-32B-Instruct-abliterated.Q3KM.gguf	Q3K_M	14.84GB
Qwen2.5-32B-Instruct-abliterated.Q3KL.gguf	Q3K_L	16.06GB
Qwen2.5-32B-Instruct-abliterated.IQ4XS.gguf	IQ4XS	16.64GB
Qwen2.5-32B-Instruct-abliterated.Q40.gguf	Q40	17.36GB
Qwen2.5-32B-Instruct-abliterated.IQ4NL.gguf	IQ4NL	17.53GB
Qwen2.5-32B-Instruct-abliterated.Q4KS.gguf	Q4K_S	17.49GB
Qwen2.5-32B-Instruct-abliterated.Q4K.gguf	Q4K	18.49GB
Qwen2.5-32B-Instruct-abliterated.Q4KM.gguf	Q4K_M	18.49GB
Qwen2.5-32B-Instruct-abliterated.Q41.gguf	Q41	19.22GB
Qwen2.5-32B-Instruct-abliterated.Q50.gguf	Q50	21.08GB
Qwen2.5-32B-Instruct-abliterated.Q5KS.gguf	Q5K_S	21.08GB
Qwen2.5-32B-Instruct-abliterated.Q5K.gguf	Q5K	21.66GB
Qwen2.5-32B-Instruct-abliterated.Q5KM.gguf	Q5K_M	21.66GB
Qwen2.5-32B-Instruct-abliterated.Q51.gguf	Q51	22.95GB
Qwen2.5-32B-Instruct-abliterated.Q6K.gguf	Q6K	25.04GB
Qwen2.5-32B-Instruct-abliterated.Q80.gguf	Q80	32.43GB

Original model description:

---

library_name: transformers
license: apache-2.0
license_link: https://huggingface.co/huihui-ai/Qwen2.5-32B-Instruct-abliterated/blob/main/LICENSE
language:

• en

pipeline_tag: text-generation
base_model: Qwen/Qwen2.5-32B-Instruct
tags:

• chat
• abliterated
• uncensored

---

huihui-ai/Qwen2.5-32B-Instruct-abliterated

This is an uncensored version of Qwen2.5-32B-Instruct created with abliteration (see this article to know more about it).

Special thanks to @FailSpy for the original code and technique. Please follow him if you're interested in abliterated models.

Usage

You can use this model in your applications by loading it with Hugging Face's transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer

Load the model and tokenizer
model_name = "huihui-ai/Qwen2.5-32B-Instruct-abliterated"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.frompretrained(modelname)

Initialize conversation context
initial_messages = [
    {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."}
]
messages = initial_messages.copy()  # Copy the initial conversation context

Enter conversation loop
while True:
    # Get user input
    user_input = input("User: ").strip()  # Strip leading and trailing spaces

# If the user types '/exit', end the conversation
    if user_input.lower() == "/exit":
        print("Exiting chat.")
        break

# If the user types '/clean', reset the conversation context
    if user_input.lower() == "/clean":
        messages = initial_messages.copy()  # Reset conversation context
        print("Chat history cleared. Starting a new conversation.")
        continue

# If input is empty, prompt the user and continue
    if not user_input:
        print("Input cannot be empty. Please enter something.")
        continue

# Add user input to the conversation
    messages.append({"role": "user", "content": user_input})

# Build the chat template
    text = tokenizer.applychattemplate(
        messages,
        tokenize=False,
        addgenerationprompt=True
    )

# Tokenize input and prepare it for the model
    modelinputs = tokenizer([text], returntensors="pt").to(model.device)

# Generate a response from the model
    generated_ids = model.generate(
        model_inputs,
        maxnewtokens=8192
    )

# Extract model output, removing special tokens
    generated_ids = [
        outputids[len(inputids):] for inputids, outputids in zip(modelinputs.inputids, generated_ids)
    ]
    response = tokenizer.batchdecode(generatedids, skipspecialtokens=True)[0]

# Add the model's response to the conversation
    messages.append({"role": "assistant", "content": response})

# Print the model's response
    print(f"Qwen: {response}")