Quantization made by Richard Erkhov.

Github

Discord

Request more models

OpenELM-450M-Instruct - GGUF

• Model creator: https://huggingface.co/apple/
• Original model: https://huggingface.co/apple/OpenELM-450M-Instruct/

Name	Quant method	Size
OpenELM-450M-Instruct.Q2K.gguf	Q2K	0.18GB
OpenELM-450M-Instruct.IQ3XS.gguf	IQ3XS	0.19GB
OpenELM-450M-Instruct.IQ3S.gguf	IQ3S	0.2GB
OpenELM-450M-Instruct.Q3KS.gguf	Q3K_S	0.2GB
OpenELM-450M-Instruct.IQ3M.gguf	IQ3M	0.21GB
OpenELM-450M-Instruct.Q3K.gguf	Q3K	0.23GB
OpenELM-450M-Instruct.Q3KM.gguf	Q3K_M	0.23GB
OpenELM-450M-Instruct.Q3KL.gguf	Q3K_L	0.24GB
OpenELM-450M-Instruct.IQ4XS.gguf	IQ4XS	0.24GB
OpenELM-450M-Instruct.Q40.gguf	Q40	0.25GB
OpenELM-450M-Instruct.IQ4NL.gguf	IQ4NL	0.25GB
OpenELM-450M-Instruct.Q4KS.gguf	Q4K_S	0.25GB
OpenELM-450M-Instruct.Q4K.gguf	Q4K	0.27GB
OpenELM-450M-Instruct.Q4KM.gguf	Q4K_M	0.27GB
OpenELM-450M-Instruct.Q41.gguf	Q41	0.28GB
OpenELM-450M-Instruct.Q50.gguf	Q50	0.3GB
OpenELM-450M-Instruct.Q5KS.gguf	Q5K_S	0.3GB
OpenELM-450M-Instruct.Q5K.gguf	Q5K	0.31GB
OpenELM-450M-Instruct.Q5KM.gguf	Q5K_M	0.31GB
OpenELM-450M-Instruct.Q51.gguf	Q51	0.32GB
OpenELM-450M-Instruct.Q6K.gguf	Q6K	0.35GB
OpenELM-450M-Instruct.Q80.gguf	Q80	0.45GB

Original model description:

---

license: other
license_name: apple-sample-code-license
license_link: LICENSE

---

OpenELM

Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, Mohammad Rastegari

We introduce OpenELM, a family of Open Efficient Language Models. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. We pretrained OpenELM models using the CoreNet library. We release both pretrained and instruction tuned models with 270M, 450M, 1.1B and 3B parameters.

Our pre-training dataset contains RefinedWeb, deduplicated PILE, a subset of RedPajama, and a subset of Dolma v1.6, totaling approximately 1.8 trillion tokens. Please check license agreements and terms of these datasets before using them.

Usage

We have provided an example function to generate output from OpenELM models loaded via HuggingFace Hub in generateopenelm.py.

You can try the model by running the following command:

python generateopenelm.py --model apple/OpenELM-450M-Instruct --hfaccesstoken [HFACCESSTOKEN] --prompt 'Once upon a time there was' --generatekwargs repetition_penalty=1.2

Please refer to this link to obtain your hugging face access token.

Additional arguments to the hugging face generate function can be passed via generatekwargs. As an example, to speedup the inference, you can try lookup token speculative generation by passing the promptlookupnumtokens argument as follows:

python generateopenelm.py --model apple/OpenELM-450M-Instruct --hfaccesstoken [HFACCESSTOKEN] --prompt 'Once upon a time there was' --generatekwargs repetitionpenalty=1.2 promptlookupnumtokens=10

Alternatively, try model-wise speculative generation with an assistive model by passing a smaller model through the assistantmodel argument, for example:

python generateopenelm.py --model apple/OpenELM-450M-Instruct --hfaccesstoken [HFACCESSTOKEN] --prompt 'Once upon a time there was' --generatekwargs repetitionpenalty=1.2 --assistantmodel [SMALLER_MODEL]

Main Results

Zero-Shot

Model Size	ARC-c	ARC-e	BoolQ	HellaSwag	PIQA	SciQ	WinoGrande	Average
OpenELM-270M	26.45	45.08	53.98	46.71	69.75	84.70	53.91	54.37
OpenELM-270M-Instruct	30.55	46.68	48.56	52.07	70.78	84.40	52.72	55.11
OpenELM-450M	27.56	48.06	55.78	53.97	72.31	87.20	58.01	57.56
OpenELM-450M-Instruct	30.38	50.00	60.37	59.34	72.63	88.00	58.96	59.95
OpenELM-1_1B	32.34	55.43	63.58	64.81	75.57	90.60	61.72	63.44
OpenELM-1_1B-Instruct	37.97	52.23	70.00	71.20	75.03	89.30	62.75	65.50
OpenELM-3B	35.58	59.89	67.40	72.44	78.24	92.70	65.51	67.39
OpenELM-3B-Instruct	39.42	61.74	68.17	76.36	79.00	92.50	66.85	69.15

LLM360

Model Size	ARC-c	HellaSwag	MMLU	TruthfulQA	WinoGrande	Average
OpenELM-270M	27.65	47.15	25.72	39.24	53.83	38.72
OpenELM-270M-Instruct	32.51	51.58	26.70	38.72	53.20	40.54
OpenELM-450M	30.20	53.86	26.01	40.18	57.22	41.50
OpenELM-450M-Instruct	33.53	59.31	25.41	40.48	58.33	43.41
OpenELM-1_1B	36.69	65.71	27.05	36.98	63.22	45.93
OpenELM-1_1B-Instruct	41.55	71.83	25.65	45.95	64.72	49.94
OpenELM-3B	42.24	73.28	26.76	34.98	67.25	48.90
OpenELM-3B-Instruct	47.70	76.87	24.80	38.76	67.96	51.22

OpenLLM Leaderboard

Model Size	ARC-c	CrowS-Pairs	HellaSwag	MMLU	PIQA	RACE	TruthfulQA	WinoGrande	Average
OpenELM-270M	27.65	66.79	47.15	25.72	69.75	30.91	39.24	53.83	45.13
OpenELM-270M-Instruct	32.51	66.01	51.58	26.70	70.78	33.78	38.72	53.20	46.66
OpenELM-450M	30.20	68.63	53.86	26.01	72.31	33.11	40.18	57.22	47.69
OpenELM-450M-Instruct	33.53	67.44	59.31	25.41	72.63	36.84	40.48	58.33	49.25
OpenELM-1_1B	36.69	71.74	65.71	27.05	75.57	36.46	36.98	63.22	51.68
OpenELM-1_1B-Instruct	41.55	71.02	71.83	25.65	75.03	39.43	45.95	64.72	54.40
OpenELM-3B	42.24	73.29	73.28	26.76	78.24	38.76	34.98	67.25	54.35
OpenELM-3B-Instruct	47.70	72.33	76.87	24.80	79.00	38.47	38.76	67.96	55.73

See the technical report for more results and comparison.

Evaluation

Setup

Install the following dependencies:

# install public lm-eval-harness

harness_repo="public-lm-eval-harness"
git clone https://github.com/EleutherAI/lm-evaluation-harness ${harness_repo}
cd ${harness_repo}
use main branch on 03-15-2024, SHA is dc90fec

git checkout dc90fec
pip install -e .
cd ..


66d6242 is the main branch on 2024-04-01 
pip install datasets@git+https://github.com/huggingface/datasets.git@66d6242
pip install tokenizers>=0.15.2 transformers>=4.38.2 sentencepiece>=0.2.0

Evaluate OpenELM

# OpenELM-450M-Instruct
hf_model=apple/OpenELM-450M-Instruct

this flag is needed because lm-eval-harness set addbostoken to False by default, but OpenELM uses LLaMA tokenizer which requires addbostoken to be True
tokenizer=meta-llama/Llama-2-7b-hf
addbostoken=True
batch_size=1

mkdir lmevaloutput

shot=0
task=arcchallenge,arceasy,boolq,hellaswag,piqa,race,winogrande,sciq,truthfulqa_mc2
lm_eval --model hf \
        --modelargs pretrained=${hfmodel},trustremotecode=True,addbostoken=${addbostoken},tokenizer=${tokenizer} \
        --tasks ${task} \
        --device cuda:0 \
        --num_fewshot ${shot} \
        --outputpath ./lmevaloutput/${hfmodel//\//}${task//,/_}-${shot}shot \
        --batchsize ${batchsize} 2>&1 | tee ./lmevaloutput/eval-${hfmodel//\//}${task//,/}-${shot}shot.log

shot=5
task=mmlu,winogrande
lm_eval --model hf \
        --modelargs pretrained=${hfmodel},trustremotecode=True,addbostoken=${addbostoken},tokenizer=${tokenizer} \
        --tasks ${task} \
        --device cuda:0 \
        --num_fewshot ${shot} \
        --outputpath ./lmevaloutput/${hfmodel//\//}${task//,/_}-${shot}shot \
        --batchsize ${batchsize} 2>&1 | tee ./lmevaloutput/eval-${hfmodel//\//}${task//,/}-${shot}shot.log

shot=25
task=arcchallenge,crowspairs_english
lm_eval --model hf \
        --modelargs pretrained=${hfmodel},trustremotecode=True,addbostoken=${addbostoken},tokenizer=${tokenizer} \
        --tasks ${task} \
        --device cuda:0 \
        --num_fewshot ${shot} \
        --outputpath ./lmevaloutput/${hfmodel//\//}${task//,/_}-${shot}shot \
        --batchsize ${batchsize} 2>&1 | tee ./lmevaloutput/eval-${hfmodel//\//}${task//,/}-${shot}shot.log

shot=10
task=hellaswag
lm_eval --model hf \
        --modelargs pretrained=${hfmodel},trustremotecode=True,addbostoken=${addbostoken},tokenizer=${tokenizer} \
        --tasks ${task} \
        --device cuda:0 \
        --num_fewshot ${shot} \
        --outputpath ./lmevaloutput/${hfmodel//\//}${task//,/_}-${shot}shot \
        --batchsize ${batchsize} 2>&1 | tee ./lmevaloutput/eval-${hfmodel//\//}${task//,/}-${shot}shot.log

Bias, Risks, and Limitations

The release of OpenELM models aims to empower and enrich the open research community by providing access to state-of-the-art language models. Trained on publicly available datasets, these models are made available without any safety guarantees. Consequently, there exists the possibility of these models producing outputs that are inaccurate, harmful, biased, or objectionable in response to user prompts. Thus, it is imperative for users and developers to undertake thorough safety testing and implement appropriate filtering mechanisms tailored to their specific requirements.

Citation

If you find our work useful, please cite:

@article{mehtaOpenELMEfficientLanguage2024,
	title = {{OpenELM}: {An} {Efficient} {Language} {Model} {Family} with {Open} {Training} and {Inference} {Framework}},
	shorttitle = {{OpenELM}},
	url = {https://arxiv.org/abs/2404.14619v1},
	language = {en},
	urldate = {2024-04-24},
	journal = {arXiv.org},
	author = {Mehta, Sachin and Sekhavat, Mohammad Hossein and Cao, Qingqing and Horton, Maxwell and Jin, Yanzi and Sun, Chenfan and Mirzadeh, Iman and Najibi, Mahyar and Belenko, Dmitry and Zatloukal, Peter and Rastegari, Mohammad},
	month = apr,
	year = {2024},
}

@inproceedings{mehta2022cvnets, 
     author = {Mehta, Sachin and Abdolhosseini, Farzad and Rastegari, Mohammad}, 
     title = {CVNets: High Performance Library for Computer Vision}, 
     year = {2022}, 
     booktitle = {Proceedings of the 30th ACM International Conference on Multimedia}, 
     series = {MM '22} 
}