┏━┓┏━╸   ┏┳┓╻┏┓╻╻
┃ ┃┃╺┓╺━╸┃┃┃┃┃┗┫┃
┗━┛┗━┛   ╹ ╹╹╹ ╹╹

OffGPT Mini is a special LLM that

• runs on a server that costs 20 Dollar or less
• is able to serve thousands of users without problems
• while consuming several watts only
• is simple to host by non technical individuals
• shows good practice when using containerization

Relevant files are in app/my-app/public .

The codebase is small. So as a user or as a hoster, it is recommended to have a look at the code. Check or let check that data does not leave the machine, that no malicious lines are in this project.

To run it, issue

docker compose up -d 

How to install docker compose on your OS is written on their official website.

When the container is running, you can access the interface by opening a browser, and going on

http://localhost:3000

To use the LLM server, you have to first download a small model in so called gguf format.

In the interface ( on localhost:3000 ) you can upload this local gguf file and start chatting fully locally.

#Technical hints:

You can run docker with quadlets too, btw. Or with podman, which I personally defer as it is part of the google / big tech / Kubernetes ecosystem.

As you can see in the docker-compose.yaml file, the whole container such as its volumes are read only, and all kernel capabilities could be dropped. That is possible, because here we have actual static serving. Which makes the hosting of this project simple and safe, also for non technical users.

The interface is mainly defined by the index.html file.

Which means you can customize the interface easily.