Overview
This guide provides comprehensive instructions on adding, customizing, and deleting models within the Jan platform.
Add Models
There are various ways to add models to Jan.
Currently, Jan natively supports the following model formats:
- GGUF (through a llama.cpp engine)
- TensorRT (through a TRT-LLM engine)
Download from Jan Hub
Jan Hub provides three convenient methods to access machine learning models. Here’s a clear step-by-step guide for each method:
1. Download from the Recommended List
The Recommended List is a great starting point if you're looking for popular and pre-configured models that work well and quickly on most computers.
- Open the Jan app and navigate to the Hub.
- Select models, clicking the
v
dropdown for more information. Models with theRecommended
label will likely run faster on your computer. - Click Download to download the model.
2. Download with HuggingFace Model's ID or URL
If you need a specific model from Hugging Face (opens in a new tab), Jan Hub lets you download it directly using the model’s ID or URL.
Only GGUF
models are supported for this feature.
- Go to the Hugging Face (opens in a new tab).
- Select the model you want to use.
- Copy the Model's ID or URL, for example:
MaziyarPanahi/Mixtral-8x22B-Instruct-v0.1-GGUF
orhttps://huggingface.co/MaziyarPanahi/Mixtral-8x22B-Instruct-v0.1-GGUF
. - Return to the Jan app and click on the Hub tab.
- Paste the URL or the model ID you have copied into the search bar.
- The app will show all available versions of the model.
- Click Download to download the model.
3. Download with Deep Link
You can also use Jan's deep link feature to download a specific model from Hugging Face (opens in a new tab). The deep link format is: jan://models/huggingface/<model's ID>
.
The deep link feature cannot be used for models that require:
- API Token.
- Acceptance of usage agreement.
You will need to download such models manually.
- Go to the Hugging Face (opens in a new tab).
- Select the model you want to use.
- Copy the Model's ID or URL, for example:
TheBloke/Magicoder-S-DS-6.7B-GGUF
. - Enter the deep link URL with your chosen model's ID in your browser. For example:
jan://models/huggingface/TheBloke/Magicoder-S-DS-6.7B-GGUF
- A prompt will appear, click Open to open the Jan app.
- The app will show all available versions of the model.
- Click Download to download the model.
Import or Symlink Local Models
You can also point to existing model binary files on your local filesystem. This is the easiest and most space-efficient way if you have already used other local AI applications.
- Navigate to the Hub.
- Click on
Import Model
at the top. - Select to import using
.GGUF
file or a folder. - Select the model or the folder containing multiple models.
- Optionally, check the box to symlink the model files instead of copying them over the Jan Data Folder. This saves disk space.
Windows users should drag and drop the model file, as Click to Upload might not show the model files in Folder Preview.
Add a Model Manually
You can also add a specific model that is not available within the Hub section by following the steps below:
- Open the Jan app.
- Click the gear icon (⚙️) on the bottom left of your screen.
- Under the Settings screen, click Advanced Settings.
- Open the Jan Data folder.
- Head to the
~/jan/models/
. - Make a new model folder and put a file named
model.json
in it. - Insert the following
model.json
default code:
{ "id": "<unique_identifier_of_the_model>", "object": "<type_of_object, e.g., model, tool>", "name": "<name_of_the_model>", "version": "<version_number>", "description": "<brief_description_of_the_model>", "format": "<format_of_the_model_api_or_other>", "settings": "<additional_settings_as_needed>", "parameters": { "max_tokens": "<maximum_number_of_tokens_the_model_can_generate>", "temperature": "<temperature_setting_for_randomness_in_generation>" }, "metadata": { "author": "<name_of_the_creator_or_organization>", "tags": ["<list_of_relevant_tags_describing_the_model>"] }, "engine": "<engine_or_platform_the_model_runs_on>", "source": "<url_or_source_of_the_model_information>"}
There are two important fields in model.json
that you need to set:
Settings
This is the field where you can set your engine configurations. There are two important fields that you need to define for your local models:
Term | Description |
---|---|
ctx_len | Defined based on the model's context size. |
prompt_template | Defined based on the model's trained template (e.g., ChatML, Alpaca). |
To set up the prompt_template
based on your model, follow the steps below:
- Visit Hugging Face (opens in a new tab), an open-source machine-learning platform.
- Find the current model that you're using (e.g., Gemma 7b it (opens in a new tab)).
- Review the text and identify the template.
Parameters
parameters
are the adjustable settings that affect how your model operates or processes the data.
The fields in parameters
are typically general and can be the same across models. Here is an example of model parameters:
To see the complete list of a model's parameters, please see below.
"parameters":{ "temperature": 0.7, "top_p": 0.95, "stream": true, "max_tokens": 4096, "frequency_penalty": 0, "presence_penalty": 0}
Delete Models
To delete a model:
- Go to Settings.
- Go to My Models.
- Select the three dots next and select
Delete model
.