Model Parameters
A model has three main parameters to configure:
- Inference Parameters
- Model Parameters
- Engine Parameters
Inference Parameters
Inference parameters are settings that control how an AI model generates outputs. These parameters include the following:
| Parameter | Description |
|---|---|
| Temperature | - Influences the randomness of the model's output. - A higher temperature leads to more random and diverse responses, while a lower temperature produces more predictable outputs. |
| Top P | - Sets a probability threshold, allowing only the most likely tokens whose cumulative probability exceeds the threshold to be considered for generation. - A lower top-P value (e.g., 0.9) may be more suitable for focused, task-oriented applications, while a higher top-P value (e.g., 0.95 or 0.97) may be better for more open-ended, creative tasks. |
| Stream | - Enables real-time data processing, which is useful for applications needing immediate responses, like live interactions. It accelerates predictions by processing data as it becomes available. - Turned on by default. |
| Max Tokens | - Sets the upper limit on the number of tokens the model can generate in a single output. - A higher limit benefits detailed and complex responses, while a lower limit helps maintain conciseness. |
| Stop Sequences | - Defines specific tokens or phrases that signal the model to stop producing further output. - Use common concluding phrases or tokens specific to your application’s domain to ensure outputs terminate appropriately. |
| Frequency Penalty | - Modifies the likelihood of the model repeating the same words or phrases within a single output, reducing redundancy in the generated text. - Increase the penalty to avoid repetition in scenarios where varied language is preferred, such as creative writing or content generation. |
| Presence Penalty | - Encourages the generation of new and varied concepts by penalizing tokens that have already appeared, promoting diversity and novelty in the output. - Use a higher penalty for tasks requiring high novelty and variety, such as brainstorming or ideation sessions. |
Model Parameter
Model parameters are the settings that define and configure the model's behavior. These parameters include the following:
| Parameter | Description |
|---|---|
| Prompt Template | - This predefined text or framework generates responses or predictions. It is a structured guide that the AI model fills in or expands upon during the generation process. - For example, a prompt template might include placeholders or specific instructions that direct how the model should formulate its outputs. |
Engine Parameters
Engine parameters are the settings that define how the model processes input data and generates output. These parameters include the following:
| Parameter | Description |
|---|---|
| Context Length | - This parameter determines the maximum input amount the model can generate responses. The maximum context length varies with the model used. This setting is crucial for the model’s ability to produce coherent and contextually appropriate outputs. - For tasks like summarizing long documents that require extensive context, use a higher context length. A lower setting can quicken response times and lessen computational demand for simpler queries or brief interactions. |
By default, Jan sets the Context Length to the maximum supported by your model, which may slow down response times. For lower-spec devices, reduce Context Length to 1024 or 2048, depending on your device's specifications, to improve speed.
Customize the Model Settings
Adjust model settings for a specific conversation or across all conversations:
A Specific Conversation
To customize model settings for a specific conversation only:
- Create a new thread.
- Expand the right panel.
- Change settings under the
modeldropdown.
All Conversations
To customize default model settings for all conversations:
- Open any thread.
- Select the three dots next to the
modeldropdown. - Select
Edit global defaults for [model]. - Edit the default settings directly in the
model.json. - Save the file and refresh the app.
