π Model Description
license: apache-2.0 language:
- en
- chat
- anthracite-org/c2logs32kllama3qwen2v1.2nosystem
- anthracite-org/kalo-opus-instruct-22k-no-refusal-no-system
- anthracite-org/kalo-opus-instruct-3k-filtered-no-system
- anthracite-org/nopmclaudewritingfixed
- anthracite-org/kaloopusmisc240827nosystem
- anthracite-org/kalomiscpart2no_system
This repo contains GGUF quants of the model. If you need the original weights, please find them here.
This is a series of models designed to replicate the prose quality of the Claude 3 models, specifically Sonnet and Opus.
This model is fine-tuned on top of mistralai/Mistral-Nemo-Instruct-2407.
Prompting
A typical input would look like this:<s>[INST] SYSTEM MESSAGE
USER MESSAGE[/INST] ASSISTANT MESSAGE</s>[INST] USER MESSAGE[/INST]SillyTavern templates
Below are Instruct and Context templates for use within SillyTavern.
context template
default SillyTavern template works fineinstruct template
default SillyTavern template works fineAxolotl config
See axolotl config
base_model: mistralai/Mistral-Nemo-Instruct-2407
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
hubmodelid: anthracite-org/magnum-v4-12b-r2
hubstrategy: "allcheckpoints"
pushdatasetto_hub:
hfuseauth_token: true
plugins:
- axolotl.integrations.liger.LigerPlugin
liger_rope: true
ligerrmsnorm: true
liger_swiglu: true
ligerfusedlinearcrossentropy: true
loadin8bit: false
loadin4bit: false
strict: false
datasets:
- path: anthracite-org/c2logs32kllama3qwen2v1.2no_system
type: custommistralv3tekken
- path: anthracite-org/kalo-opus-instruct-22k-no-refusal-no-system
type: custommistralv3tekken
- path: anthracite-org/kalo-opus-instruct-3k-filtered-no-system
type: custommistralv3tekken
- path: anthracite-org/nopmclaudewriting_fixed
type: custommistralv3tekken
- path: anthracite-org/kaloopusmisc240827no_system
type: custommistralv3tekken
- path: anthracite-org/kalomiscpart2nosystem
type: custommistralv3tekken
#chat_template: chatml
shufflemergeddatasets: true
#defaultsystemmessage: "You are an assistant that responds to the user."
datasetpreparedpath: /workspace/data/magnum-12b-data
valsetsize: 0.0
output_dir: /workspace/data/12b-fft-out
sequence_len: 32768
sample_packing: true
padtosequence_len: true
adapter:
loramodeldir:
lora_r:
lora_alpha:
lora_dropout:
loratargetlinear:
lorafaninfanout:
wandb_project: 12b-magnum-fft
wandb_entity:
wandb_watch:
wandb_name: v4-r2-attempt-01
wandblogmodel:
gradientaccumulationsteps: 2
microbatchsize: 1
num_epochs: 2
optimizer: adamwbnb8bit
lr_scheduler: cosine
learning_rate: 0.00001
trainoninputs: false
groupbylength: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
earlystoppingpatience:
resumefromcheckpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
warmup_steps: 40
evalsperepoch:
evaltablesize:
evalmaxnew_tokens:
savesperepoch: 2
debug:
deepspeed: deepspeed_configs/zero2.json
weight_decay: 0.1
fsdp:
fsdp_config:
special_tokens:
pad_token: <pad>
Credits
We'd like to thank Recursal / Featherless for sponsoring the compute for this train, Featherless has been hosting our Magnum models since the first 72 B and has given thousands of people access to our models and helped us grow.We would also like to thank all members of Anthracite who made this finetune possible.
Datasets
- anthracite-org/c2logs32kllama3qwen2v1.2nosystem
- anthracite-org/kalo-opus-instruct-22k-no-refusal-no-system
- anthracite-org/kalo-opus-instruct-3k-filtered-no-system
- anthracite-org/nopmclaudewritingfixed
- anthracite-org/kaloopusmisc240827nosystem
- anthracite-org/kalomiscpart2nosystem
Training
The training was done for 2 epochs. We used 8xH100s GPUs graciously provided by Recursal AI / Featherless AI for the full-parameter fine-tuning of the model.