Running LLMs Locally Just Got Easier With Ollama

In the past years, organizations such as OpenAI, Anthropic, Mistral and others have provided us access to low cost yet high performance large language models (LLMs) without the stress of managing the infrastructure. Despite the organizations' promise not to exploit the user's information it handled from their respective models, some businesses remain wary and prefer on-premises systems that process and retain data.

Aside from transparency concerns, people are also skeptical of network latency and downtime associated with using LLMs from these established organizations. To address these issues, people started to initiate and run LLMs locally on personal computers and/or managed servers.

In this article, we'll introduce a remarkable tool designed to run open-source LLMs locally—this is Ollama.

What is Ollama?

Open-source LLMs are often distributed in the GGUF format, formerly known as GGML. Running these types of models, however, involves significant work since the community has to create a low-level software program to allow their execution. Unfortunately, using this program requires a deep understanding of command lines and specific terminology, deeming it inconvenient to use for the average user.

Ollama, on the other hand, has streamlined the process of running LLMs, making it more accessible to a wider audience. It is an easy-to-use tool that allows users to interact and run open-source LLMs locally, within their desktops, supporting a large model library containing models such as Lama 2, Mistal 7B, and Openchat just to name a few. By bringing it locally to your machine, you control the entire flow which gives a lot of potential. The best part of using Ollama is that it is FREE!

As of this posting, Ollama is only available for macOS and Linux. A Windows OS version is still in the works and will be available soon, so be sure to check their website at ollama.ai.

Installing Ollama

Head over to Ollama's official website to download and start the installation process.

1. Click the Download button.
2. Choose the OS version for your device. Note that only macOS and Linux options are available at the moment. For this specific guide, we will be referring to the macOS installation procedure.
3. Once downloaded, extract the file (usually in ZIP format) and open the Ollama application.
4. A 'Welcome to Ollama' window should now appear. Click Next button then click Install button to install the command line.
5. The next screen will show 'Run your first model.' In order to run the Ollama framework, you need to copy the provided command (default command is 'ollama run llama2') and click Finish. Paste and run the copied command on to your preferred terminal. Ollama should be running the Llama2 model at this point.
6. Alternatively, if you prefer a different model to begin with, you can proceed doing so by heading to the [Models library page](https://ollama.ai/library) of the Ollama Website. Select the model you want to run and copy the command under CLI Instruct. Head back to the 'Run your first model' screen and paste the copied command and click Finish.
7. One of the best perks of Ollama is that switching to another model is more convenient and efficient. You just need to copy the command of the model you want to switch from the Models library page and paste it to the terminal and hit Enter. Ollama will automatically start downloading the new model.
8. Once downloading of the model is complete, you can now start experimenting and/or provide your queries. The possibilities are endless so make sure to explore it diligently.

Check out this link from Mike Bird on another example of a model you can try on Ollama: https://twitter.com/MikeBirdTech/status/1721911715187626260

Pros and Cons of Ollama

Like any other programs, Ollama has its fair share of pros and cons that must be discussed to set proper expectations to people who are not familiar with the tool.

Pros

• Easy Setup: The installation process is straightforward and user-friendly.
• Cost Effective: It's totally FREE to use and can be hosted locally. No need to spend on costly maintenance and infrastructures.
• Model Diversity: Offers an entire library of highly capable open-source models.
• Data Privacy: Keeps your data in-house and safe, without having the fear of uncertainty that your data might be exploited.
• Customization: Highly customizable to suit every user's needs. Switching models is literally just copying and pasting a command.

For those seeking data privacy with hands-on AI experimentation, Ollama is shaping up to be an invaluable tool. As it continues developing amid the open-source AI movement, we can expect an even more refined user experience and functionality.

Cons

• Complexity: It lacks the streamlined nature of OpenAI or Mistral API endpoints.
• Scalability: Additional expertise and steps are required to scale and host Ollama in a cloud environment.
• Local Limitations: Hosting larger models locally is challenging due to hardware constraints, often necessitating cloud deployment to manage the load effectively.

Takeaways

The whole point of this article is to show the value for running highly capable LLMs on your local machine to avoid having to use externally hosted proprietary LLMs which you don't have control over. Ollama offers the convenience of having your own LLM that's free such as the one's offered by OpenAI, Mistral, and the likes.

For those seeking data privacy with hands-on AI experimentation, Ollama is shaping up to be an invaluable tool. As it continues developing amid the open-source AI movement, we can expect an even more refined user experience and functionality.

Overall, Ollama has effectively lowered the barrier to locally leveraging powerful language models. This enables broader AI literacy and creativity at the edge. The project shows promising potential to democratize access to the latest advancements in natural language processing.

Sources:

• https://www.youtube.com/watch?v=rIRkxZSn-A8&t=4s
• [https://twitter.com/MikeBirdTech/status/1721911715187626260
• [https://www.linkedin.com/pulse/21-8-running-llms-locally-ollama-won-bae-suh-itzxc
• [https://medium.com/@CompXBio/ollama-run-your-local-llm-0111dafae66a
• [https://www.youtube.com/watch?v=MGr1V4LyGFA
• [https://eightify.app/summary/computer-science-and-technology/run-llms-locally-with-ollama-simplifying-the-process