Skip to content
This repository has been archived by the owner on Jan 7, 2025. It is now read-only.

Commit

Permalink
Update alternative_models.rst (#504)
Browse files Browse the repository at this point in the history
  • Loading branch information
jakethekoenig authored Jan 28, 2024
1 parent 0a2988d commit 9e411ac
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/source/user/alternative_models.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ In addition, Mentat uses the :code:`gpt-4-1106-preview` model by default. When u
🦙 Local Models
---------------

In our experiments we have not found any non-openai models to be as good as even gpt-3.5-turbo with Mentat. That being said it is possible to use Mentat with other models with just a few steps. Mentat uses the OpenAI SDK to retrieve chat completions. This means that setting the `OPENAI_API_BASE` environment variable is enough to use any model that has the same response schema as OpenAI. To use models with different response schemas, we recommend setting up a litellm proxy as described [here](https://docs.litellm.ai/docs/proxy/quick_start) and pointing `OPENAI_API_BASE` to the proxy. You can use local models run with ollama with the following steps:
In our experiments we have not found any non-openai models to be as good as even gpt-3.5-turbo with Mentat. That being said it is possible to use Mentat with other models with just a few steps. Mentat uses the OpenAI SDK to retrieve chat completions. This means that setting the `OPENAI_API_BASE` environment variable is enough to use any model that has the same response schema as OpenAI. To use models with different response schemas, we recommend setting up a litellm proxy as described `here <https://docs.litellm.ai/docs/proxy/quick_start>`__ and pointing `OPENAI_API_BASE` to the proxy. You can use local models run with ollama with the following steps:

First run ollama. Replace mixtral with whichever model you want to use.

Expand Down

0 comments on commit 9e411ac

Please sign in to comment.