1 files changed, 108 insertions, 0 deletions
diff --git a/blog/local-llm/index.org b/blog/local-llm/index.org
new file mode 100644
index 0000000..ccde66e
--- /dev/null
+++ b/blog/local-llm/index.org
@@ -0,0 +1,108 @@
+#+title: Running Local LLMs on macOS and iOS
+#+date: 2024-01-13
+#+description: Finding some useful applications for running local LLMs on macOS and iOS.
+#+filetags: :apple:
+
+* Requirements
+I've recently started playing with large language models (LLMs), mostly
+in the popular chatbot form, as part of my job and have decided to see
+if there's a consistent and reliable way to interact with these models
+on Apple devices without sacrificing privacy or requiring in-depth
+technical setup.
+
+My requirements for this test:
+
+- Open source platform
+- On-device model files
+- Minimal required configuration
+- Preferably pre-built, but a simple build process is acceptable
+
+I tested a handful of apps and have summarized my favorite (so far) for
+macOS and iOS below.
+
+#+begin_quote
+TL;DR - Here are the two that met my requirements and I have found the
+easiest to install and use so far:
+
+#+end_quote
+
+- macOS: [[https://ollama.ai/][Ollama]]
+- iOS : [[https://llmfarm.site/][LLM Farm]]
+
+* macOS
+[[https://ollama.ai/][Ollama]] is a simple Go application for macOS and
+Linux that can run various LLMs locally.
+
+For macOS, you can download the pplication on the
+[[https://ollama.ai/download/mac][Ollama download page]] and install it
+by unzipping the =Ollama.app= file and moving it to the =Applications=
+folder.
+
+If you prefer the command line, you can run these commands after the
+download finished:
+
+#+begin_src sh
+cd ~/Downloads && \
+unzip Ollama-darwin.zip && \
+mv ~/Downloads/Ollama.app /Applications/
+#+end_src
+
+After running the app, the app will ask you to open a terminal and run
+the default =llama2= model, which will open an interactive chat session
+in the terminal. You can startfully using the application at this point.
+
+#+caption: Ollama
+[[https://img.cleberg.net/blog/20240113-local-llm/ollama.png]]
+
+If you don't want to use the default =llama2= model, you can download
+and run additional models found on the
+[[https://ollama.ai/library][Models]] page.
+
+To see the information for the currently-used model, you can run the
+=/show info= command in the chat.
+
+#+caption: Model Info
+[[https://img.cleberg.net/blog/20240113-local-llm/ollama_info.png]]
+
+** Community Integrations
+I highly recommend browsing the
+[[https://github.com/jmorganca/ollama#community-integrations][Community
+Integrations]] section of the project to see how you would prefer to
+extend Ollama beyond a simple command-line interface. There are options
+for APIs, browser UIs, advanced terminal configurations, and more.
+
+#+caption: Ollama SwiftUI
+[[https://img.cleberg.net/blog/20240113-local-llm/ollama-swiftui.png]]
+
+* iOS
+While there are a handful of decent macOS options, it was quite
+difficult to find an iOS app that offered an open source platform
+without an extensive configuration and building process. I found LLM
+Farm to be decent enough in quality to sit at the top of my list -
+however, it's definitely not user friendly enough for me to consider
+using it on a daily basis.
+
+[[https://llmfarm.site/][LLM Farm]] is available on TestFlight, so
+there's no manual build process required. However, you can view the
+[[https://github.com/guinmoon/LLMFarm][LLMFarm repository]] if you wish.
+
+The caveat is that you will have to manually download the model files
+from the links in the
+[[https://github.com/guinmoon/LLMFarm/blob/main/models.md][models.md]]
+file to your iPhone to use the app - there's currently no option in the
+app to reach out and grab the latest version of any supported model.
+
+Once you have a file downloaded, you simply create a new chat and select
+the downloaded model file and ensure the inference matches the
+requirement in the =models.md= file.
+
+See below for a test of the ORCA Mini v3 model:
+
+| Chat List                                                              | Chat                                                             |
+|------------------------------------------------------------------------+------------------------------------------------------------------|
+| [[https://img.cleberg.net/blog/20240113-local-llm/llm_farm_chats.png]] | [[https://img.cleberg.net/blog/20240113-local-llm/llm_farm.png]] |
+
+[[https://github.com/AugustDev/enchanted][Enchanted]] is also an iOS for
+private AI models, but it requires a public-facing Ollama API, which did
+not meet my "on device requirement." Nonetheless, it's an interesting
+looking app and I will likely set it up to test soon.