• generatedfromtrainer

• name: >-

• ruslandev/tagengo-rus-gpt-4o

Llama-3 8B GPT-4o-RU1.0

[[Dataset]](https://huggingface.co/datasets/ruslandev/tagengo-rus-gpt-4o)

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct.
The idea behind this model is to train on a dataset derived from a smaller subset of the tagengo-gpt4, but with improved data quality.
I tried to achieve higher data quality by prompting GPT-4o, the latest OpenAI's LLM with better multilingual capabilities. The training objective is primarily focused on the Russian language (80% of the training examples).
After training for 1 epoch on 2 NVIDIA A100 the model shows promising results on the MT-Bench evaluation benchmark, surpassing GPT-3.5-turbo and being on par with Suzume in Russian language scores,
even though the latter is trained on 8x bigger and more diverse dataset.

How to use

The easiest way to use this model on your own computer is to use the GGUF version of this model (ruslandev/llama-3-8b-gpt-4o-ru1.0-gguf) using a program such as llama.cpp.
If you want to use this model directly with the Huggingface Transformers stack, I recommend using my framework gptchain.

git clone https://github.com/RuslanPeresy/gptchain.git
cd gptchain
pip install -r requirements-train.txt
python gptchain.py chat -m ruslandev/llama-3-8b-gpt-4o-ru1.0 \
	--chatml true \
	-q '[{"from": "human", "value": "Из чего состоит нейронная сеть?"}]'

Evaluation scores

I achieved the following scores on Ru/En MT-Bench:

Training procedure

base_model: meta-llama/Meta-Llama-3-8B-Instruct
model_type: LlamaForCausalLM
tokenizer_type: AutoTokenizer  # PreTrainedTokenizerFast


loadin8bit: false
loadin4bit: false
strict: false

datasets:
  - path: ruslandev/tagengo-rus-gpt-4o
    type: sharegpt
    conversation: llama-3
datasetpreparedpath: /home/ubuntu/llmtraining/axolotl/llama3-8b-gpt-4o-ru/preparedtagengo_rus
valsetsize: 0.01
outputdir: /home/ubuntu/llmtraining/axolotl/llama3-8b-gpt-4o-ru/outputllama38bgpt4o_ru

sequence_len: 8192
sample_packing: true
padtosequence_len: true
evalsamplepacking: false

use_wandb: false
#wandb_project: axolotl
#wandbentity: wandbentity
#wandbname: llama38bgpt4oru

gradientaccumulationsteps: 2
microbatchsize: 2
num_epochs: 1
optimizer: pagedadamw8bit
lr_scheduler: cosine
learning_rate: 1e-5

trainoninputs: false
groupbylength: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
gradientcheckpointingkwargs:
  use_reentrant: false
earlystoppingpatience:
resumefromcheckpoint:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
evalsperepoch: 5
evaltablesize:
savesperepoch: 1
debug:
deepspeed: /home/ubuntu/axolotl/deepspeed_configs/zero2.json
weight_decay: 0.0
special_tokens:
  padtoken: <|endof_text|>

Training hyperparameters

The following hyperparameters were used during training:

• learningrate: 1e-05
• trainbatchsize: 2
• evalbatchsize: 2
• seed: 42
• distributedtype: multi-GPU
• numdevices: 2
• gradientaccumulationsteps: 2
• totaltrainbatchsize: 8
• totalevalbatchsize: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lrschedulertype: cosine
• lrschedulerwarmupsteps: 10
• num_epochs: 1

Training results

Framework versions

• Transformers 4.41.1
• Pytorch 2.2.2+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1