• en

• text-generation-inference
• transformers
• gemma
• gguf
• llama.cpp

Uploaded model

• Developed by: pmking27
• License: apache-2.0
• Finetuned from model : pmking27/PrathameshLLM-2B

Provided Quants Files

Run one of the following commands, according to your system:

# Base ctransformers with no GPU acceleration
pip install llama-cpp-python
With NVidia CUDA acceleration
CMAKEARGS="-DLLAMACUBLAS=on" pip install llama-cpp-python
Or with OpenBLAS acceleration
CMAKEARGS="-DLLAMABLAS=ON -DLLAMABLASVENDOR=OpenBLAS" pip install llama-cpp-python
Or with CLBLast acceleration
CMAKEARGS="-DLLAMACLBLAST=on" pip install llama-cpp-python
Or with AMD ROCm GPU acceleration (Linux only)
CMAKEARGS="-DLLAMAHIPBLAS=on" pip install llama-cpp-python
Or with Metal GPU acceleration for macOS systems only
CMAKEARGS="-DLLAMAMETAL=on" pip install llama-cpp-python

In windows, to set the variables CMAKE_ARGS in PowerShell, follow this format; eg for NVidia CUDA:
$env:CMAKEARGS = "-DLLAMAOPENBLAS=on"
pip install llama-cpp-python

Model Download Script

import os
from huggingfacehub import hfhub_download

Specify model details
modelrepoid = "pmking27/PrathameshLLM-2B-GGUF"  # Replace with the desired model repo
filename = "PrathameshLLM-2B.Q4KM.gguf"  # Replace with the specific GGUF filename
local_folder = "."  # Replace with your desired local storage path

Create the local directory if it doesn't exist
os.makedirs(localfolder, existok=True)

Download the model file to the specified local folder
filepath = hfhubdownload(repoid=modelrepoid, filename=filename, cachedir=local_folder)

print(f"GGUF model downloaded and saved to: {filepath}")

Replace modelrepoid and filename with the desired model repository ID and specific GGUF filename respectively. Also, modify local_folder to specify where you want to save the downloaded model file.

#### Simple llama-cpp-python Simple inference example code

from llama_cpp import Llama

llm = Llama(
  model_path = filepath,  # Download the model file first
  n_ctx = 32768,  # The max sequence length to use - note that longer sequence lengths require much more resources
  n_threads = 8,            # The number of CPU threads to use, tailor to your system and the resulting performance
  ngpulayers = 35         # The number of layers to offload to GPU, if you have GPU acceleration available
)
Defining the Alpaca prompt template

alpaca_prompt = """
Instruction:

{}


Input:
{}

Response:
{}"""

output = llm(
  alpaca_prompt.format(
        '''
        You're an assistant trained to answer questions using the given context.

context:

General elections will be held in India from 19 April 2024 to 1 June 2024 to elect the 543 members of the 18th Lok Sabha. The elections will be held in seven phases and the results will be announced on 4 June 2024. This will be the largest-ever election in the world, surpassing the 2019 Indian general election, and will be the longest-held general elections in India with a total span of 44 days (excluding the first 1951–52 Indian general election). The incumbent prime minister Narendra Modi who completed a second term will be contesting elections for a third consecutive term.

Approximately 960 million individuals out of a population of 1.4 billion are eligible to participate in the elections, which are expected to span a month for completion. The Legislative assembly elections in the states of Andhra Pradesh, Arunachal Pradesh, Odisha, and Sikkim will be held simultaneously with the general election, along with the by-elections for 35 seats among 16 states.
        ''', # instruction
        "In how many phases will the general elections in India be held?", # input
        "", # output - leave this blank for generation!
    ), #Alpaca Prompt
  max_tokens = 512,  # Generate up to 512 tokens
  stop = ["<eos>"],   #stop token
  echo = True        # Whether to echo the prompt
)

output_text = output['choices'][0]['text']
start_marker = "### Response:"
end_marker = "<eos>"
startpos = outputtext.find(startmarker) + len(startmarker)
endpos = outputtext.find(endmarker, startpos)

Extracting the response text
responsetext = outputtext[startpos:endpos].strip()

print(response_text)

#### Simple llama-cpp-python Chat Completion API Example Code

from llama_cpp import Llama
llm = Llama(modelpath = filepath, chatformat="gemma")  # Set chat_format according to the model you are using
message=llm.createchatcompletion(
    messages = [
        {"role": "system", "content": "You are a story writing assistant."},
        {
            "role": "user",
            "content": "Write a story about llamas."
        }
    ]
)
message['choices'][0]["message"]["content"]