Kit Dev: Run an LLM Locally

If you're using Kit with LLMs you can quickly run the model locally to speed integration, testing, or experimentation.

To run the ModelKit locally, first create a new directory for your LLM:

mkdir kitdev
cd kitdev

Now unpack an LLM ModelKit - there are several on Jozu Hub, but here we're using Phi3 Mini because of its size:

kit unpack jozu.ml/jozu/phi3:3.8b-mini-instruct-4k-q4_K_M

Now start your LLM dev server locally using the kit dev start command:

kit dev start .

In the command output you'll see a URL you can use to interact with the LLM (there's a command flag to always use the same port). You can control parameters of the model, change the prompt, or chat with the LLM.

If you need to get logs use the dev logs command:

kit dev logs

When you're done don't forget to stop the Kit dev server:

kit dev stop

Questions or suggestions? Drop an issue in our GitHub repository or join our Discord server to get support or share your feedback.