Run Directly on Your PC! Personal Open-Source LLM: The Complete Guide to Building Your Local AI Assistant

G: Hello everyone, who is interested in the future of AI and technology! 🌟

Cloud-based large language models (LLMs) are shaking up the world, but have you ever worried about things like, ‘Is my data safe?’, ‘Is the usage fee too expensive?’, or ‘Can I use it without internet?’ 🤔 Don’t worry anymore! You can resolve all these concerns by running a personal open-source LLM directly on your PC.

Today, I will tell you everything about running open-source LLMs locally to transform your PC into a powerful AI workstation. From preparations to core concepts, actual execution methods, and 10 recommended models optimized for a local environment, with this one guide, you too can become a local AI master! 🚀

Section 1: Why should I run an LLM on my PC? 🤔

Cloud LLM services are good, but local execution offers you imagination-beyond freedom and benefits.

• 1. Privacy Protection and Data Security 🔒:

  • This is the biggest advantage. All processing occurs within your PC, without the need to expose sensitive personal or corporate confidential data to external parties. 🕵️‍♂️
  • You can use it with peace of mind, without worrying about data being stored on cloud servers or used for training.

• 2. Cost Savings 💸:

  • There’s no need to pay API usage fees or subscription fees. Once set up, you can use it unlimitedly without additional cost. In the long run, this is a huge saving!

• 3. Fast Response Speed and Offline Use ⚡️✈️:

  • You can get instant responses without network delays, unaffected by internet connection status.
  • You can utilize your AI assistant anytime, even on an airplane or in an environment without internet access.

• 4. Infinite Customization and Experimentation 🛠️:

  • You can customize the model as desired, such as adjusting the model’s parameters directly or attempting additional training (fine-tuning) with specific datasets.
  • You can freely switch and test various open-source models.

• 5. Transparency and Control 🛡️:

  • You can understand the model’s operation more deeply and verify what data it was trained on.
  • You can have complete control over the AI system.

Section 2: Before You Start, Check Your Preparations! 🛠️

Running a local LLM is not as difficult as it might seem, but it requires a few basic preparations.

• 1. Hardware (Most Important!):

  • RAM (Memory): Minimum 16GB, 32GB or more recommended 🥇. LLM models use a lot of memory when loaded.
  • GPU (Graphics Card):
    • NVIDIA GeForce RTX 30 series or higher (VRAM 8GB or more) recommended! More CUDA cores are better. This most significantly affects LLM inference speed.
    • VRAM (Video Memory): 8GB is a starting point, and 12GB, 24GB or more allows running larger and more powerful models. It’s different from general RAM!
    • AMD GPUs are possible depending on ROCm support, but NVIDIA has much better compatibility. Intel Arc GPUs are also seeing expanded support through OpenVINO, etc.
  • CPU: Latest multi-core processor (Intel Core i5/Ryzen 5 or higher). If GPU memory is insufficient, the model may load with CPU RAM.
  • SSD: NVMe SSD recommended. Model file sizes are large, so fast loading speed is crucial. Secure at least 100GB of free space.

• 2. Software:

  • Operating System: Windows 10/11, macOS (Apple Silicon M1/M2/M3), Linux (Ubuntu, etc.).
  • Python (Optional but convenient): Version 3.9 or higher. Useful for installing libraries using pip.
  • Git: Required to download model files.
  • Conda or venv (Virtual Environment): It is recommended to use a virtual environment to avoid dependency conflicts between Python projects.

Section 3: Understanding Core Concepts – The Secret to Local LLM Execution 💡

It may seem complex, but with just a few concepts, you can easily understand the principles of local LLM execution.

• 1. Quantization:

  • LLMs are fundamentally trained with precision like FP16 (16-bit floating point). However, this makes the model file too large and burdens the VRAM of consumer GPUs.
  • Quantization is a technique that compresses model weights into lower bits (e.g., 4-bit, 8-bit). 📉
  • For example, an FP16 model with 7B (7 billion) parameters requires about 14GB of VRAM, but a 4-bit quantized model needs only about 4GB!
  • Of course, as the quantization level increases, the model’s accuracy may slightly decrease, but it’s often imperceptible in most cases. Q4_K_M (K-quantization, medium) is the most popular method, offering a good balance of performance and quality.

• 2. GGUF / GGML File Format:

  • Originally, LLM models are saved in frameworks like PyTorch or TensorFlow.
  • GGML is a library written in C/C++, optimized specifically for efficiently running LLMs on CPUs and GPUs (CUDA, Metal, etc.).
  • GGUF is the model file format for the new version of GGML. Files with the .gguf extension can be easily loaded locally using GGML-based tools like llama.cpp. This is key to local LLM execution! 💾

• 3. Prompt Engineering:

  • Whether it’s a local or cloud LLM, to get the AI to generate the desired response, it’s crucial to write the question (prompt) clearly and specifically. ✨
  • Example: Instead of “What’s the weather like today?”, it’s better to ask specifically, “What’s the weather like in Seoul today? What’s the temperature and what’s the chance of rain?”

Section 4: Powerful Tools for Local LLM Execution! 💪

Now, let’s introduce some representative tools that can run actual local LLMs. These tools allow you to easily use numerous GGUF models.

1. Oobabooga Text Generation WebUI (Most Flexible and Powerful) 💻

• Features: A web-based UI that uses llama.cpp as its backend. It offers the most versatile features, including model loading, chat interface, prompt engineering functions, and API server capabilities. Functionality can be extended through various extensions.
• Installation and Usage (Brief):
  1. Git Clone:

     git clone https://github.com/oobabooga/text-generation-webui.git
     cd text-generation-webui

  2. Run Installation Script:
     • Windows: start_windows.bat
     • Linux/WSL: ./start_linux.sh
     • macOS: ./start_macos.sh
     • (The script automatically handles the installation of required dependencies.)
  3. Download Model: Download the desired .gguf model file from sources like Hugging Face into the text-generation-webui/models folder. (e.g., TheBloke/Mistral-7B-OpenOrca-GGUF/mistral-7b-openorca.Q4_K_M.gguf)
  4. Run UI and Load Model:
     • After running the script, access the web browser (http://127.0.0.1:7860).
     • In the ‘Model’ tab, select the downloaded model and click the ‘Load’ button.
     • Go to the ‘Chat’ tab and start conversing with the AI!
     • GPU Acceleration: When running, add arguments like --load-in-4bit (if using bitsandbytes) or --gpu-layers 30 (if using GGML/GGUF, 30 is the number of layers to offload to the GPU) to maximize GPU utilization.

2. LM Studio (Easiest All-in-One Solution) 🤩

• Features: A desktop application that allows you to search, download, and run GGUF models all at once. It’s like an app store where you can easily choose a model and run it with one click, highly recommended for beginners. It can also be used as an API through its built-in local server.
• Installation and Usage:
  1. Download and run the installation file for your OS from the LM Studio official website.
  2. Once the app is running, enter the desired model name (e.g., Mistral, Llama2) in the search bar and search.
  3. Select the desired .gguf model from the search results and click the ‘Download’ button.
  4. Go to the ‘Chat’ tab, select the downloaded model, and you can immediately start a conversation.
  5. Activate the GPU option in settings to utilize GPU acceleration.

3. Jan (Open-Source Desktop App) 💖

• Features: Similar to LM Studio, it’s provided as a desktop application and is very easy to use. Being open-source is a big advantage, and it’s useful when you want to quickly test various models. It also provides API server functionality.
• Installation and Usage:
  1. Download and run the installation file from the Jan official website or GitHub repository.
  2. After running the app, search for and download the desired model in the ‘Models’ section.
  3. Once the download is complete, you can start a conversation immediately in the ‘Chat’ section.
  4. Check the GPU acceleration settings in Settings.

4. Ollama (Developer-Friendly CLI/API) 👨‍💻

• Features: Designed to easily download and run LLMs via the command-line interface (CLI). It allows starting a model with a single command like ollama run llama2, similar to Docker. It has a built-in API server, making it very convenient for programmatically utilizing LLMs. Recently, it also supports a desktop UI.
• Installation and Usage:
  1. Download and install the version for your OS from the Ollama official website.
  2. Open a terminal/command prompt and enter the following command to run the Llama 2 model:

     ollama run llama2

  3. Ollama will automatically download the model, and a conversation prompt will appear.
  4. If you want to run other models, just change the model name, like ollama run mistral.
  5. You can check the list of currently downloaded models with the ollama list command.

Section 5: 10 Recommended Open-Source LLMs That Shine on Your PC! ✨

Among numerous open-source models, we have carefully selected 10 models suitable for local execution and offering excellent performance. Most can be found on Hugging Face as TheBloke’s or GGUF quantized versions.

1. Llama 2 (7B, 13B, 70B) 🦙

   • Features: A model released by Meta, a leading open-source LLM. It forms the basis for various fine-tuned models. 7B and 13B are sufficiently usable on personal PCs, while 70B shows powerful performance on high-end GPUs (24GB+ VRAM).
   • Reason for Recommendation: Most widely used, stable, and many fine-tuned versions are available.
   • Recommended Size: 7B (Q4_K_M), 13B (Q4_K_M)

2. Mistral 7B Instruct 🌬️

   • Features: A 7B model released by Mistral AI, but boasts much stronger performance than Llama 2 of the same size. It is also fast and shows excellent inference capabilities.
   • Reason for Recommendation: The best value-for-money model that provides high-quality output with less VRAM.
   • Recommended Size: 7B (Q4_K_M)

3. Mixtral 8x7B Instruct 🤯

   • Features: A Mixture of Experts (MoE) structured model, where 8 expert 7B models collaborate. During actual inference, it operates at a similar speed to a 12.9B model while showing overwhelming performance equivalent to 8x7B.
   • Reason for Recommendation: Worth trying if you want to experience top-tier performance. Note that VRAM consumption is high, so a GPU with 24GB or more is recommended.
   • Recommended Size: 8x7B (Q4_K_M) – Minimum 28GB VRAM recommended

4. Zephyr 7B Beta / Zephyr-7B-gemma-v0.1 🚀

   • Features: A conversational fine-tuned model based on Mistral 7B, developed by Hugging Face. It specializes in chat, allowing natural conversations, and shows excellent performance despite its small size.
   • Reason for Recommendation: Optimal for use as a personal assistant or chatbot.
   • Recommended Size: 7B (Q4_K_M)

5. OpenOrca / Orca 2 (7B, 13B) 🐳

   • Features: Models developed inspired by Microsoft’s Orca project. They show strength in complex reasoning and instruction following capabilities.
   • Reason for Recommendation: Powerful for instruction-based tasks and questions requiring logical thinking.
   • Recommended Size: 7B, 13B (Q4_K_M)

6. Phi-2 (Microsoft) 🧠

   • Features: A “small” model with 2.7B parameters released by Microsoft, but it shows astonishing performance that belies its size. Trained on educational datasets, it excels in coding and logical reasoning.
   • Reason for Recommendation: Suitable when you want to experience a high-quality LLM even on a low-spec PC.
   • Recommended Size: 2.7B (Q4_K_M) – 4GB VRAM is sufficient!

7. Qwen 7B Instruct / Qwen 1.5 Series 🇨🇳

   • Features: A model developed by Alibaba Cloud, showing strong performance in benchmarks. Especially excellent in multilingual support. The recent Qwen 1.5 series boasts even more improved performance.
   • Reason for Recommendation: Good if you need diverse language support, including Korean.
   • Recommended Size: 7B, 14B (Q4_K_M)

8. Stable Beluga 13B 🐋

   • Features: A fine-tuned model based on Llama 2 13B, primarily showing excellent performance in question-answering and conversational scenarios. It also has strong instruction-following abilities.
   • Reason for Recommendation: Good when looking for a stable and balanced 13B-class model.
   • Recommended Size: 13B (Q4_K_M)

9. Dolphin 2.2.1-Mistral-7B 🐬

   • Features: An “Untensored” model built on Mistral 7B, providing more unrestricted answers due to less filtering. (Caution required when using)
   • Reason for Recommendation: Can be used for unrestricted experimentation on specific topics or creative writing.
   • Recommended Size: 7B (Q4_K_M)

10. SOLAR 10.7B Instruct (Kakao Brain) ☀️

    • Features: A Mistral-based model developed by Kakao Brain, achieving good performance with a smaller model using a unique method called SLiC (Stack-aligned Little is Capable). Its Korean language ability is particularly outstanding.
    • Reason for Recommendation: Highly recommended for users for whom Korean performance is very important and who also want decent English performance.
    • Recommended Size: 10.7B (Q4_K_M)

Section 6: Tips for Smarter Local LLM Utilization! 💡

• 1. Choose Quantization Level:
  • If VRAM is insufficient, try a lower quantization level like Q3_K_M or Q2_K. However, Q4_K_M is the most common, with little performance degradation.
  • If VRAM is sufficient, you can use Q5_K_M or Q8_0 to further improve quality.
• 2. GPU Layer Optimization (--gpu-layers / n_gpu_layers):
  • In Oobabooga or LM Studio, you can specify the number of model layers to load onto the GPU. For example, --gpu-layers 30 means offload 30 layers of the model to the GPU.
  • Send as many layers to the GPU as VRAM allows. The remaining part will be loaded into CPU RAM.
• 3. Practice Prompt Engineering:
  • Practice asking clear and specific questions to get the model to provide the desired answers. Role-playing and providing examples are effective.
• 4. Experiment with Various Models:
  • Each model has different strengths and weaknesses. A specific model might be better for coding, while another is better for creative writing. Download and experiment with multiple models for different tasks.
• 5. Utilize Communities:
  • You can get the latest model information, usage tips, and troubleshooting methods from communities like Hugging Face model cards and Reddit’s r/LocalLLaMA.

Section 7: Troubleshooting and Additional Tips ⚠️

• 1. If “Out of Memory” or “CUDA Error” occurs:
  • Insufficient VRAM: Try lowering the quantization level of the model you are using (e.g., Q5_K_M -> Q4_K_M).
  • Reduce GPU layers: Reduce the --gpu-layers value to lessen the burden on the GPU.
  • Close other programs: Close all background programs that use GPU memory.
  • Update drivers: Keep NVIDIA/AMD graphics drivers up to date.
• 2. If LLM response speed is too slow or odd:
  • Check GPU acceleration: Verify that GPU acceleration is properly enabled in the tool you are using. (CUDA, ROCm, Metal, etc.)
  • Check model: You might have loaded too large a model or used an incorrect quantization file.
  • Prompt issue: The prompt might be ambiguous, or the topic might not have been learned by the model.
• 3. If installation errors occur:
  • Check Python version: Confirm that you are using the required Python version.
  • Use virtual environment: Create a virtual environment with a command like conda create -n llm_env python=3.10 to avoid conflicts.
  • Search error message: Google the specific error message to find solutions.

Conclusion: The Future of AI Unfolding in Your Hands! 🚀

Now, your PC is ready to transform from a simple computer into a powerful AI assistant. You can use your own LLM anytime, anywhere, without worries about personal data leaks or cost burdens.

Of course, the initial setup might seem a bit complicated, but once you successfully get it running, you’ll be amazed by its convenience and possibilities. It will dramatically boost your productivity in various fields such as coding, writing, brainstorming, and learning.

Don’t hesitate, try it right now! If you have any questions, feel free to ask in the comments. We support your AI journey! 💖