content/blog/2024-01-13-local-llm.md


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103

+++
date = 2024-01-13
title = "Running Local LLMs on macOS and iOS"
description = "Finding some useful applications for running local LLMs on macOS and iOS."
+++

## Requirements

I've recently started playing with large language models (LLMs), mostly in the
popular chatbot form, as part of my job and have decided to see if there's a
consistent and reliable way to interact with these models on Apple devices
without sacrificing privacy or requiring in-depth technical setup.

My requirements for this test:

- Open source platform
- On-device model files
- Minimal required configuration
- Preferably pre-built, but a simple build process is acceptable

I tested a handful of apps and have summarized my favorite (so far) for macOS
and iOS below.

> TL;DR - Here are the two that met my requirements and I have found the easiest
> to install and use so far:

- macOS: [Ollama](https://ollama.ai/)
- iOS  : [LLM Farm](https://llmfarm.site/)

## macOS

[Ollama](https://ollama.ai/) is a simple Go application for macOS and Linux that
can run various LLMs locally.

For macOS, you can download the pplication on the [Ollama download
page](https://ollama.ai/download/mac) and install it by unzipping the
`Ollama.app` file and moving it to the `Applications` folder.

If you prefer the command line, you can run these commands after the download
finished:

```sh
cd ~/Downloads && \
unzip Ollama-darwin.zip && \
mv ~/Downloads/Ollama.app /Applications/
```

After running the app, the app will ask you to open a terminal and run the
default `llama2` model, which will open an interactive chat session in the
terminal. You can startfully using the application at this point.

![Ollama](https://img.cleberg.net/blog/20240113-local-llm/ollama.png "Ollama")

If you don't want to use the default `llama2` model, you can download and run
additional models found on the [Models](https://ollama.ai/library) page.

To see the information for the currently-used model, you can run the `/show
info` command in the chat.

![Model Info](https://img.cleberg.net/blog/20240113-local-llm/ollama_info.png "Model Info")

### Community Integrations

I highly recommend browsing the [Community
Integrations](https://github.com/jmorganca/ollama#community-integrations)
section of the project to see how you would prefer to extend Ollama beyond a
simple command-line interface. There are options for APIs, browser UIs, advanced
terminal configurations, and more.

![Ollama SwiftUI](https://img.cleberg.net/blog/20240113-local-llm/ollama-swiftui.png "Ollama SwifTUI")

## iOS

While there are a handful of decent macOS options, it was quite difficult to
find an iOS app that offered an open source platform without an extensive
configuration and building process. I found LLM Farm to be decent enough in
quality to sit at the top of my list -  however, it's definitely not user
friendly enough for me to consider using it on a daily basis.

[LLM Farm](https://llmfarm.site/) is available on TestFlight, so there's no
manual build process required. However, you can view the [LLMFarm
repository](https://github.com/guinmoon/LLMFarm) if you wish.

The caveat is that you will have to manually download the model files from the
links in the
[models.md](https://github.com/guinmoon/LLMFarm/blob/main/models.md) file to
your iPhone to use the app - there's currently no option in the app to reach out
and grab the latest version of any supported model.

Once you have a file downloaded, you simply create a new chat and select the
downloaded model file and ensure the inference matches the requirement in the
`models.md` file.

See below for a test of the ORCA Mini v3 model:

| Chat List                                                                                              | Chat                                                                                 |
|--------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
| ![LLM Farm Chats](https://img.cleberg.net/blog/20240113-local-llm/llm_farm_chats.png "LLM Farm Chats") | ![LLM Farm](https://img.cleberg.net/blog/20240113-local-llm/llm_farm.png "LLM Farm")

[Enchanted](https://github.com/AugustDev/enchanted) is also an iOS for private
AI models, but it requires a public-facing Ollama API, which did not meet my
"on device requirement." Nonetheless, it's an interesting looking app and I will
likely set it up to test soon.