Skip to content
All tags

#gguf

3 posts
ai deep-dive 認識 AI 模型

Quantization & Inference Optimization: Running a 70B Model on Your Laptop

A 70B model needs ~140GB VRAM in FP16, but 4-bit quantization shrinks it to ~35GB. With llama.cpp's partial CPU offloading, it can run on consumer hardware. GGUF naming conventions (Q4_K_M, Q5_K_S) tell you the precision-size tradeoff. KV cache is why long conversations slow down.

#quantization#gguf#kv-cache#inference#vram#ollama#llama-cpp#ai-model
ai guide

The Complete Unsloth Guide: Fine-Tune and Run LLMs Locally, Faster

Unsloth is the fastest, most VRAM-efficient local LLM fine-tuning tool — 2× training speed and 70% less VRAM. In 2026 it added a Desktop app that bundles inference, training, image/video generation, web search, and agent integration into a complete local AI workstation.

#unsloth#llm#fine-tuning#local-inference#lora#qlora#gguf#self-hosted
ai guide

llama.cpp — From Pure C++ to an LLM Inference Engine on Consumer Hardware

llama.cpp is the most widely used local LLM inference engine, implemented in pure C/C++. It supports CPU, Metal, CUDA, Vulkan, and other backends, and uses the GGUF quantization format to run multi-billion-parameter models on consumer hardware.

#llama-cpp#gguf#quantization#llm-inference#apple-silicon#metal#cuda#local-llm