C4AI Command R7B Arabic - Quantized Versions in GGUF Format

This repository contains quantized versions of the C4AI Command R7B Arabic model, provided in GGUF format. These quantized versions are designed to reduce model size and improve inference speed while maintaining reasonable performance.

Available Quantized Versions

The following GGUF quantized versions are available:

• Q2K
• Q3KM
• Q40
• Q4KM
• Q5KS
• Q5KM
• Q6K
• Q80
• F16 Quantization

Original Repository

The original model was developed by Cohere and Cohere For AI. You can find it here:

https://huggingface.co/CohereForAI/c4ai-command-r7b-arabic-02-2025

License

These quantized versions follow the same licensing terms as the original model: CC-BY-NC-4.0, with an additional requirement to comply with C4AI’s Acceptable Use Policy. By using these models, you agree to abide by these terms.

Available Models

The GGUF files available in this repository are listed below:

Installation

To use these GGUF models, you can use:

1. llama-cpp-python (Python Library)

Install with:

pip install llama-cpp-python

2. llama.cpp (C++ Library)

If you prefer a non-Python workflow, you can use the llama.cpp C++ implementation.

3. LM Studio (GUI Interface)

LM Studio provides an easy-to-use graphical interface for running GGUF models locally. You can download it from:

https://lmstudio.ai

4. GPT4All (Cross-Platform GUI & CLI)

GPT4All supports running GGUF models across various operating systems. You can install it from:

https://gpt4all.io

5. Ollama (Local Model Runner)

Ollama is a lightweight tool for running LLMs locally. Download it from:

https://ollama.com

Downloading the Models

You can download the GGUF files from this repository using the huggingface_hub library:

from huggingfacehub import hfhub_download

hfhubdownload(
    repo_id="eltay89/c4ai-command-r7b-arabic-02-2025-gguf",
    filename="c4ai-command-r7b-arabic-02-2025-Q4KM.gguf",
    local_dir="."
)

Alternatively, download the files directly from the repository’s page on Hugging Face.

Usage

Using llama-cpp-python in Python

from llama_cpp import Llama

Load the model (replace with the path to your downloaded GGUF file)
llm = Llama(modelpath="path/to/c4ai-command-r7b-arabic-02-2025-Q4K_M.gguf")

Generate text
output = llm("مرحبا، كيف حالك؟", max_tokens=100, temperature=0.3)
print(output['choices'][0]['text'])

Replace "path/to/c4ai-command-r7b-arabic-02-2025-Q4KM.gguf" with the actual path to your downloaded GGUF file.

The prompt "مرحبا، كيف حالك؟" translates to "Hello, how are you?" in Arabic.

Chat Templates

LM Studio Chat Template

To get clean, conversational outputs in LM Studio, use this chat template:

Before System: <|STARTOFTURNTOKEN|><|SYSTEMTOKEN|>  
After System: <|ENDOFTURN_TOKEN|>  
Before User: <|STARTOFTURNTOKEN|><|USERTOKEN|>  
After User: <|ENDOFTURN_TOKEN|>  
Before Assistant: <|STARTOFTURNTOKEN|><|CHATBOTTOKEN|><|START_RESPONSE|>  
After Assistant: <|ENDOFTURN_TOKEN|>  
Additional Stop Strings: <|ENDRESPONSE|>, <|ENDOFTURNTOKEN|>, <|STARTTHINKING|>, <|ENDTHINKING|>, <|STARTACTION|>, <|ENDACTION|>, <|STARTTOOLRESULT|>, <|ENDTOOLRESULT|>

Ollama Chat Template

For Ollama, create a Modelfile with this content:

FROM ./c4ai-command-r7b-arabic-02-2025-Q4KM.gguf

TEMPLATE """
<|STARTOFTURNTOKEN|><|SYSTEMTOKEN|> {{ system }} <|ENDOFTURN_TOKEN|>
<|STARTOFTURNTOKEN|><|USERTOKEN|> {{ prompt }} <|ENDOFTURN_TOKEN|>
<|STARTOFTURNTOKEN|><|CHATBOTTOKEN|><|STARTRESPONSE|> {{ response }} <|ENDOFTURNTOKEN|>
"""

PARAMETER stop "<|END_RESPONSE|>"
PARAMETER stop "<|ENDOFTURN_TOKEN|>"
PARAMETER stop "<|START_THINKING|>"
PARAMETER stop "<|END_THINKING|>"
PARAMETER stop "<|START_ACTION|>"
PARAMETER stop "<|END_ACTION|>"
PARAMETER stop "<|STARTTOOLRESULT|>"
PARAMETER stop "<|ENDTOOLRESULT|>"

Run:

ollama create c4ai-command-r7b-arabic -f Modelfile
ollama run c4ai-command-r7b-arabic

Contact

For questions or issues, please refer to the original repository or contact Cohere For AI at [email protected].