diff options
Diffstat (limited to 'content/blog/2024-01-13-local-llm.org')
| -rw-r--r-- | content/blog/2024-01-13-local-llm.org | 94 |
1 files changed, 43 insertions, 51 deletions
diff --git a/content/blog/2024-01-13-local-llm.org b/content/blog/2024-01-13-local-llm.org index d6f173b..0240d81 100644 --- a/content/blog/2024-01-13-local-llm.org +++ b/content/blog/2024-01-13-local-llm.org @@ -5,11 +5,10 @@ * Requirements -I've recently started playing with large language models (LLMs), mostly -in the popular chatbot form, as part of my job and have decided to see -if there's a consistent and reliable way to interact with these models -on Apple devices without sacrificing privacy or requiring in-depth -technical setup. +I've recently started playing with large language models (LLMs), mostly in the +popular chatbot form, as part of my job and have decided to see if there's a +consistent and reliable way to interact with these models on Apple devices +without sacrificing privacy or requiring in-depth technical setup. My requirements for this test: @@ -18,12 +17,12 @@ My requirements for this test: - Minimal required configuration - Preferably pre-built, but a simple build process is acceptable -I tested a handful of apps and have summarized my favorite (so far) for -macOS and iOS below. +I tested a handful of apps and have summarized my favorite (so far) for macOS +and iOS below. #+begin_quote -TL;DR - Here are the two that met my requirements and I have found the -easiest to install and use so far: +TL;DR - Here are the two that met my requirements and I have found the easiest +to install and use so far: #+end_quote - macOS: [[https://ollama.ai/][Ollama]] @@ -31,13 +30,12 @@ easiest to install and use so far: * macOS -[[https://ollama.ai/][Ollama]] is a simple Go application for macOS and -Linux that can run various LLMs locally. +[[https://ollama.ai/][Ollama]] is a simple Go application for macOS and Linux that can run various LLMs +locally. -For macOS, you can download the pplication on the -[[https://ollama.ai/download/mac][Ollama download page]] and install it -by unzipping the =Ollama.app= file and moving it to the =Applications= -folder. +For macOS, you can download the pplication on the [[https://ollama.ai/download/mac][Ollama download page]] and +install it by unzipping the =Ollama.app= file and moving it to the +=Applications= folder. If you prefer the command line, you can run these commands after the download finished: @@ -48,49 +46,43 @@ unzip Ollama-darwin.zip && \ mv ~/Downloads/Ollama.app /Applications/ #+end_src -After running the app, the app will ask you to open a terminal and run -the default =llama2= model, which will open an interactive chat session -in the terminal. You can startfully using the application at this point. +After running the app, the app will ask you to open a terminal and run the +default =llama2= model, which will open an interactive chat session in the +terminal. You can startfully using the application at this point. -If you don't want to use the default =llama2= model, you can download -and run additional models found on the -[[https://ollama.ai/library][Models]] page. +If you don't want to use the default =llama2= model, you can download and run +additional models found on the [[https://ollama.ai/library][Models]] page. To see the information for the currently-used model, you can run the =/show info= command in the chat. ** Community Integrations -I highly recommend browsing the -[[https://github.com/jmorganca/ollama#community-integrations][Community -Integrations]] section of the project to see how you would prefer to -extend Ollama beyond a simple command-line interface. There are options -for APIs, browser UIs, advanced terminal configurations, and more. +I highly recommend browsing the [[https://github.com/jmorganca/ollama#community-integrations][Community Integrations]] section of the project to +see how you would prefer to extend Ollama beyond a simple command-line +interface. There are options for APIs, browser UIs, advanced terminal +configurations, and more. * iOS -While there are a handful of decent macOS options, it was quite -difficult to find an iOS app that offered an open source platform -without an extensive configuration and building process. I found LLM -Farm to be decent enough in quality to sit at the top of my list - -however, it's definitely not user friendly enough for me to consider -using it on a daily basis. - -[[https://llmfarm.site/][LLM Farm]] is available on TestFlight, so -there's no manual build process required. However, you can view the -[[https://github.com/guinmoon/LLMFarm][LLMFarm repository]] if you wish. - -The caveat is that you will have to manually download the model files -from the links in the -[[https://github.com/guinmoon/LLMFarm/blob/main/models.md][models.md]] -file to your iPhone to use the app - there's currently no option in the -app to reach out and grab the latest version of any supported model. - -Once you have a file downloaded, you simply create a new chat and select -the downloaded model file and ensure the inference matches the -requirement in the =models.md= file. - -[[https://github.com/AugustDev/enchanted][Enchanted]] is also an iOS for -private AI models, but it requires a public-facing Ollama API, which did -not meet my "on device requirement." Nonetheless, it's an interesting -looking app and I will likely set it up to test soon. +While there are a handful of decent macOS options, it was quite difficult to +find an iOS app that offered an open source platform without an extensive +configuration and building process. I found LLM Farm to be decent enough in +quality to sit at the top of my list - however, it's definitely not user +friendly enough for me to consider using it on a daily basis. + +[[https://llmfarm.site/][LLM Farm]] is available on TestFlight, so there's no manual build process +required. However, you can view the [[https://github.com/guinmoon/LLMFarm][LLMFarm repository]] if you wish. + +The caveat is that you will have to manually download the model files from the +links in the [[https://github.com/guinmoon/LLMFarm/blob/main/models.md][models.md]] file to your iPhone to use the app - there's currently no +option in the app to reach out and grab the latest version of any supported +model. + +Once you have a file downloaded, you simply create a new chat and select the +downloaded model file and ensure the inference matches the requirement in the +=models.md= file. + +[[https://github.com/AugustDev/enchanted][Enchanted]] is also an iOS for private AI models, but it requires a public-facing +Ollama API, which did not meet my "on device requirement." Nonetheless, it's an +interesting looking app and I will likely set it up to test soon. |