feat: add local-llm post

1 files changed, 94 insertions, 0 deletions

diff --git a/content/blog/2024-01-13-local-llm.md b/content/blog/2024-01-13-local-llm.md
new file mode 100644
index 0000000..35dc3db
--- /dev/null
+++ b/content/blog/2024-01-13-local-llm.md
@@ -0,0 +1,94 @@
++++
+date = 2024-01-13
+title = "Running Local LLMs on macOS and iOS"
+description = "Finding some useful applications for running local LLMs on macOS and iOS."
++++
+
+## Requirements
+
+I've recently started playing with large language models (LLMs), mostly in the
+popular chatbot form, as part of my job and have decided to see if there's a
+consistent and reliable way to interact with these models on Apple devices
+without sacrificing privacy or requiring in-depth technical setup.
+
+My requirements for this test:
+
+- Open source platform
+- On-device model files
+- Minimal required configuration
+- Preferably pre-built, but a simple build process is acceptable
+
+I tested a handful of apps and have summarized my favorite (so far) for macOS
+and iOS below.
+
+> TL;DR - Here are the two that met my requirements and I have found the easiest
+> to install and use so far:
+
+- macOS: [Ollama](https://ollama.ai/)
+- iOS  : [LLM Farm](https://llmfarm.site/)
+
+## macOS
+
+[Ollama](https://ollama.ai/) is a simple Go application for macOS and Linux that
+can run various LLMs locally.
+
+For macOS, you can download the pplication on the [Ollama download
+page](https://ollama.ai/download/mac) and install it by unzipping the
+`Ollama.app` file and moving it to the `Applications` folder.
+
+If you prefer the command line, you can run these commands after the download
+finished:
+
+```sh
+cd ~/Downloads && \
+unzip Ollama-darwin.zip && \
+mv ~/Downloads/Ollama.app /Applications/
+```
+
+After running the app, the app will ask you to open a terminal and run the
+default `llama2` model, which will open an interactive chat session in the
+terminal. You can startfully using the application at this point.
+
+![Ollama](https://img.cleberg.net/blog/20240113-local-llm/ollama.png "Ollama")
+
+If you don't want to use the default `llama2` model, you can download and run
+additional models found on the [Models](https://ollama.ai/library) page.
+
+To see the information for the currently-used model, you can run the `/show
+info` command in the chat.
+
+![Model Info](https://img.cleberg.net/blog/20240113-local-llm/ollama_info.png
+!"Model Info")
+
+## iOS
+
+While there are a handful of decent macOS options, it was quite difficult to
+find an iOS app that offered an open source platform without an extensive
+configuration and building process. I found LLM Farm to be decent enough in
+quality to sit at the top of my list -  however, it's definitely not user
+friendly enough for me to consider using it on a daily basis.
+
+[LLM Farm](https://llmfarm.site/) is available on TestFlight, so there's no
+manual build process required. However, you can view the [LLMFarm
+repository](https://github.com/guinmoon/LLMFarm) if you wish.
+
+The caveat is that you will have to manually download the model files from the
+links in the
+[models.md](https://github.com/guinmoon/LLMFarm/blob/main/models.md) file to
+your iPhone to use the app - there's currently no option in the app to reach out
+and grab the latest version of any supported model.
+
+Once you have a file downloaded, you simply create a new chat and select the
+downloaded model file and ensure the inference matches the requirement in the
+`models.md` file.
+
+See below for a test of the ORCA Mini v3 model:
+
+| Chat List                                                                                              | Chat                                                                                 |
+|--------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
+| ![LLM Farm Chats](https://img.cleberg.net/blog/20240113-local-llm/llm_farm_chats.png "LLM Farm Chats") | ![LLM Farm](https://img.cleberg.net/blog/20240113-local-llm/llm_farm.png "LLM Farm")
+
+[Enchanted](https://github.com/AugustDev/enchanted) is also an iOS for private
+AI models, but it requires a public-facing Ollama API, which did not meet my
+"on device requirement." Nonetheless, it's an interesting looking app and I will
+likely set it up to test soon.