#+title: Running Local LLMs on macOS and iOS
#+date: 2024-01-13
#+description: Finding some useful applications for running local LLMs on macOS and iOS.
#+filetags: :apple:

* Requirements
I've recently started playing with large language models (LLMs), mostly
in the popular chatbot form, as part of my job and have decided to see
if there's a consistent and reliable way to interact with these models
on Apple devices without sacrificing privacy or requiring in-depth
technical setup.

My requirements for this test:

- Open source platform
- On-device model files
- Minimal required configuration
- Preferably pre-built, but a simple build process is acceptable

I tested a handful of apps and have summarized my favorite (so far) for
macOS and iOS below.

#+begin_quote
TL;DR - Here are the two that met my requirements and I have found the
easiest to install and use so far:

#+end_quote

- macOS: [[https://ollama.ai/][Ollama]]
- iOS : [[https://llmfarm.site/][LLM Farm]]

* macOS
[[https://ollama.ai/][Ollama]] is a simple Go application for macOS and
Linux that can run various LLMs locally.

For macOS, you can download the pplication on the
[[https://ollama.ai/download/mac][Ollama download page]] and install it
by unzipping the =Ollama.app= file and moving it to the =Applications=
folder.

If you prefer the command line, you can run these commands after the
download finished:

#+begin_src sh
cd ~/Downloads && \
unzip Ollama-darwin.zip && \
mv ~/Downloads/Ollama.app /Applications/
#+end_src

After running the app, the app will ask you to open a terminal and run
the default =llama2= model, which will open an interactive chat session
in the terminal. You can startfully using the application at this point.

#+caption: Ollama
[[https://img.cleberg.net/blog/20240113-local-llm/ollama.png]]

If you don't want to use the default =llama2= model, you can download
and run additional models found on the
[[https://ollama.ai/library][Models]] page.

To see the information for the currently-used model, you can run the
=/show info= command in the chat.

#+caption: Model Info
[[https://img.cleberg.net/blog/20240113-local-llm/ollama_info.png]]

** Community Integrations
I highly recommend browsing the
[[https://github.com/jmorganca/ollama#community-integrations][Community
Integrations]] section of the project to see how you would prefer to
extend Ollama beyond a simple command-line interface. There are options
for APIs, browser UIs, advanced terminal configurations, and more.

#+caption: Ollama SwiftUI
[[https://img.cleberg.net/blog/20240113-local-llm/ollama-swiftui.png]]

* iOS
While there are a handful of decent macOS options, it was quite
difficult to find an iOS app that offered an open source platform
without an extensive configuration and building process. I found LLM
Farm to be decent enough in quality to sit at the top of my list -
however, it's definitely not user friendly enough for me to consider
using it on a daily basis.

[[https://llmfarm.site/][LLM Farm]] is available on TestFlight, so
there's no manual build process required. However, you can view the
[[https://github.com/guinmoon/LLMFarm][LLMFarm repository]] if you wish.

The caveat is that you will have to manually download the model files
from the links in the
[[https://github.com/guinmoon/LLMFarm/blob/main/models.md][models.md]]
file to your iPhone to use the app - there's currently no option in the
app to reach out and grab the latest version of any supported model.

Once you have a file downloaded, you simply create a new chat and select
the downloaded model file and ensure the inference matches the
requirement in the =models.md= file.

See below for a test of the ORCA Mini v3 model:

| Chat List                                                              | Chat                                                             |
|------------------------------------------------------------------------+------------------------------------------------------------------|
| [[https://img.cleberg.net/blog/20240113-local-llm/llm_farm_chats.png]] | [[https://img.cleberg.net/blog/20240113-local-llm/llm_farm.png]] |

[[https://github.com/AugustDev/enchanted][Enchanted]] is also an iOS for
private AI models, but it requires a public-facing Ollama API, which did
not meet my "on device requirement." Nonetheless, it's an interesting
looking app and I will likely set it up to test soon.