{"id":3456,"date":"2026-09-04T16:32:50","date_gmt":"2026-09-04T20:32:50","guid":{"rendered":"https:\/\/blog.lufamily.ca\/kang\/?p=3456"},"modified":"2026-09-04T16:32:53","modified_gmt":"2026-09-04T20:32:53","slug":"qwen3-8-27b-local-model","status":"publish","type":"post","link":"https:\/\/blog.lufamily.ca\/kang\/2026\/09\/04\/qwen3-8-27b-local-model\/","title":{"rendered":"Qwen3.8 27B Local Model"},"content":{"rendered":"\n

Near the end of last year, we built an AI Server<\/a> with two RTX3090 GPU card resulting in 48GB of VRAM. Back then we were running Qwen3 14B model locally on the machine.<\/p>\n\n\n\n

Agentic coding 8 months ago was very instructional driven. We would have to concentrate on small tasks and explain the steps in the detail to the AI. There were also many mistakes. For every 5 tasks, there may be bugs based on duplication, syntax errors, misunderstanding, or simply incompetence on behalf of the AI.<\/p>\n\n\n\n

How far have we come? Today using the same hardware, we are now running Qwen3.8 27B model. Things are moving very fast. I think since the initial creation of the machine, we have updated the LLM model more than three times, and the vLLM inference serving engine more than four times.<\/p>\n\n\n\n

With this current model, we can truly give complex tasks. One major example is that we were unhappy with Perplexica. Its recent release does not perform very well with integrating web search results. We ended up asking the AI to create a new chat application from scratch using an architecture and technology platforms of its choosing. We collaborate with the user interface layout and design.<\/p>\n\n\n\n

The final result is stunning. Below is a screenshot of our custom chat application.<\/p>\n\n\n

\n
\"\"
LUChat: Custom Chat Application using Qwen3.8 27B model with integrated web searches and access to our corporate Intranet.<\/figcaption><\/figure>\n<\/div>\n\n\n

We used very similar steps to install vLLM (v0.28.0 as of this writing<\/code><\/em>):<\/p>\n\n\n

\npython3 -m venv .venv\nsource .venv\/bin\/activate\n\npip install torch-c-dlpack-ext\npip install torch torchvision --index-url https:\/\/download.pytorch.org\/whl\/cu130\npip install vllm\n<\/pre><\/div>\n\n\n
We also modified the systemd<\/code> unit file so that vLLM starts with Qwen3.8 27B model when it boots. The contents of the \/etc\/systemd\/system\/vllm.service file<\/code> is below:<\/p>\n\n\n
\n[Unit]\nDescription=vLLM OpenAI Compatible Server\nAfter=network.target\n\n[Service]\n# User and Group to run the service as (e.g., 'youruser', 'yourgroup')\nUser=kang\nGroup=kang\n\n# Set the working directory\nWorkingDirectory=\/home\/kang\/py_vllm\n\nEnvironment=CUDA_VISIBLE_DEVICES=0,1\n\n# Need for vLLM 0.27.1\nEnvironment=FLASHINFER_DISABLE_VERSION_CHECK=1\n\n# New recommendation from lued\/Qwen3.8-27B-INT8-W8A16-MTP\nEnvironment NCCL_P2P_DISABLE=1\nEnvironment NCCL_CUMEM_ENABLE=0\nEnvironment VLLM_WORKER_MULTIPROC_METHOD=spawn\nEnvironment OMP_NUM_THREADS=1\nEnvironment VLLM_USE_FLASHINFER_SAMPLER=1\nEnvironment PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True,max_split_size_mb:512\n\n# The command to start the vLLM server\n# Qwen3.8 27B\nExecStart=\/home\/kang\/py_vllm\/.venv\/bin\/python -m vllm.entrypoints.openai.api_server --model lued\/Qwen3.8-27B-INT8-W8A16-MTP \\\n  --served-model-name Qwen3.8-27B \\\n  --tensor-parallel-size 2 \\\n  --pipeline-parallel-size 1 \\\n  --dtype float16 \\\n  --performance-mode balanced \\\n  --max-model-len 262144 \\\n  --gpu-memory-utilization 0.93 \\\n  --max-num-seqs 4 \\\n  --max-num-batched-tokens 4096 \\\n  --kv-cache-dtype fp8_e4m3 \\\n  --enable-prefix-caching \\\n  --enable-chunked-prefill \\\n  --mamba-cache-mode align \\\n  --prefix-match-unit 16 \\\n  --enable-prompt-tokens-details \\\n  --reasoning-parser qwen3 \\\n  --tool-call-parser qwen3_xml \\\n  --enable-auto-tool-choice \\\n  --disable-custom-all-reduce \\\n  --trust-remote-code \\\n  --default-chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \\\n  --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"repetition_penalty":1.0,"presence_penalty":0.8}' \\\n  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' --host 0.0.0.0 --port 8000 --api-key XXXXXXXX\n\n# Restart the service if it fails\nRestart=always\n\n[Install]\nWantedBy=multi-user.target\n<\/pre><\/div>\n\n\n
We used the opencode.ai<\/a> harness to build the chat-bot. To configure the harness, below is the ~\/.config\/opencode\/opencode.jsonc<\/code>:<\/p>\n\n\n
\n{\n  "$schema": "https:\/\/opencode.ai\/config.json",\n  "permission": {\n    "lsp": "allow",\n  },\n  "provider": {\n    "vLLM": {\n      "npm": "@ai-sdk\/openai-compatible",\n      "name": "vLLM (local)",\n      "options": {\n        "baseURL": "http:\/\/0.0.0.0:8000\/v1",\n        "timeout": false,\n        "chunkTimeout": 2400000,\n      },\n      "models": {\n        "Qwen3.8-27B": {\n          "name": "Qwen3.8-27B",\n          "options": {\n            "max_tokens": 262144,\n            "max_completion_tokens": 132072,\n            "temperature": 1.0,\n            "top_p": 0.95,\n            "top_k": 20,\n            "min_p": 0,\n            "presence_penalty": 0.0,\n            "repetition_penalty": 1.0,\n          },\n        },\n      },\n    },\n  },\n}\n\n<\/pre><\/div>\n\n\n
Who would have thought that the first RUST and REACT application that I ended up  writing<\/s> (producing) is one with AI.<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Near the end of last year, we built an AI Server with two RTX3090 GPU card resulting in 48GB of VRAM. Back then we were running Qwen3 14B model locally on the machine. Agentic coding 8 months ago was very instructional driven. We would have to concentrate on small tasks and explain the steps in … Continue reading “Qwen3.8 27B Local Model”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[111],"tags":[142,165,141,213,28,195],"class_list":["post-3456","post","type-post","status-publish","format-standard","hentry","category-tech","tag-ai","tag-cuda","tag-llm","tag-opencode","tag-technology","tag-vllm"],"jetpack_shortlink":"https:\/\/wp.me\/p7V6i8-TK","jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/posts\/3456","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/comments?post=3456"}],"version-history":[{"count":12,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/posts\/3456\/revisions"}],"predecessor-version":[{"id":3469,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/posts\/3456\/revisions\/3469"}],"wp:attachment":[{"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/media?parent=3456"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/categories?post=3456"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/tags?post=3456"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}