{"id":3303,"date":"2025-12-23T13:58:18","date_gmt":"2025-12-23T18:58:18","guid":{"rendered":"https:\/\/blog.lufamily.ca\/kang\/?p=3303"},"modified":"2025-12-23T13:58:19","modified_gmt":"2025-12-23T18:58:19","slug":"new-ai-server","status":"publish","type":"post","link":"https:\/\/blog.lufamily.ca\/kang\/2025\/12\/23\/new-ai-server\/","title":{"rendered":"New AI Server"},"content":{"rendered":"\n

In a previous post<\/a>, I commented on our AI server containing an old P40 GPU failed. We replaced our server with the following parts.<\/p>\n\n\n\n

Component<\/th>Description<\/th><\/tr><\/thead>
CPU<\/td>AMD Ryzen 9 9900X 4.4 GHz 12-Core Processor<\/td><\/tr>
CPU Cooler<\/td>Thermalright Peerless Assassin 120 SE 66.17 CFM CPU Cooler<\/td><\/tr>
Motherboard<\/td>Asus B650E MAX GAMING WIFI ATX AM5 Motherboard<\/td><\/tr>
Memory<\/td>Crucial Pro 64 GB (2 x 32 GB) DDR5-6000 CL40 Memory<\/td><\/tr>
Storage<\/td>Samsung 990 Pro 2 TB M.2-2280 PCle 4.0 X4 NVME Solid State Drive<\/td><\/tr>
GPU<\/td>2 x EVGA FTW3 ULTRA GAMING GeForce RTX 3090 24 GB Video Card (refurbished)<\/td><\/tr>
Case<\/td>Fractal Design Meshify 3 XL ATX Full Tower Case<\/td><\/tr>
Power Supply<\/td>SeaSonic PRIME TX-1600 ATX 3.1 1600 W 80+ Titanium Certified Fully Modular ATX Power Supply<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

I purchased all of our components at Amazon and the total (including shipping and taxes) came to to be $6,271.22. The most expensive parts were the GPU ($2,979.98), the power supply ($903.95), and then the memory ($843.19). All prices are quoted in Canadian dollars.<\/p>\n\n\n\n

I had no issues in building the computer.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n

As you can see above, after the CPU cooler and the GPU’s were installed you can barely see the motherboard. Although there are still PCIe slots available, there is no more room to actually place new PCIe cards. We still have two more DIMM slots, so we can consider a future memory upgrade.<\/p>\n\n\n\n

One of the main concerns I had was to plug this computer into an electrical socket that will not trip any of my breakers. The 1,600W power supply is awfully close to the maximum theoretical limit of a 15A breaker in our house, which would be around 1,800W. This server is too powerful for any of my current UPS units or power bars. It will have to be connected directly to a wall on a circuit that is not loaded by other appliances.<\/p>\n\n\n\n

After testing the memory using MemTest<\/a>, I installed Ubuntu Server 24.04.3 LTS. To prepare the machine for AI work load, I will then need to install Nvidia CUDA.<\/p>\n\n\n\n

Installing CUDA<\/h2>\n\n\n\n

The first step that I did was to install the Nvidia CUDA. I followed the steps here<\/a> for Ubuntu. I specifically follow the Network Repository Installation directions.<\/p>\n\n\n\n

\u276f sudo su -\n# wget https:\/\/developer.download.nvidia.com\/compute\/cuda\/repos\/ubuntu2404\/x86_64\/cuda-keyring_1.1-1_all.deb\n# dpkg -i cuda-keyring_1.1-1_all.deb\n\n# apt install cuda-toolkit\n# apt install nvidia-gds\n# reboot<\/code><\/pre>\n\n\n\n
After the reboot, I tested CUDA by doing the following:<\/p>\n\n\n\n
# nvidia-smi              <\/code><\/pre>\n\n\n\n
Installing vLLM<\/h2>\n\n\n\n
I then proceeded to install vLLM using the Quick Start<\/a> guide for Nividia CUDA.<\/p>\n\n\n\n
However I just use the Quick Start guide as a guidance. Ultimately I followed the following steps:<\/p>\n\n\n\n
\u276f sudo apt install build-essential python3.12-dev\n\n\u276f mkdir py_vllm\n\u276f cd py_vllm\n\n\u276f python3 -m venv vllm_cuda13_env\n\u276f source vllm_cuda13_env\/bin\/activate\n\n\u276f pip install torch-c-dlpack-ext\n\u276f pip install torch torchvision --index-url https:\/\/download.pytorch.org\/whl\/cu130\n\u276f pip install vllm --pre --extra-index-url https:\/\/wheels.vllm.ai\/nightly<\/code><\/pre>\n\n\n\n
I tried to run vLLM using Podman but I always run out of memory for certain models, so I chose the Python method of deployment.<\/p>\n\n\n\n
I then try to run it with Qwen\/Qwen3-14B<\/a> from Hugging Face. Since I have two GPU’s, I set the tensor-parallel-size<\/code> to 2.<\/p>\n\n\n\n
export VLLM_USE_V1=1 vllm serve Qwen\/Qwen3-14B --tensor-parallel-size=2<\/code><\/pre>\n\n\n\n
It took a minute or two to download the model and initialize the GPU’s. Once it is up and running, I verified that it was running by using a simple curl command.<\/p>\n\n\n\n
\u276f curl http:\/\/localhost:8000\/v1\/models | jq .\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100   463  100   463    0     0   615k      0 --:--:-- --:--:-- --:--:--  452k\n{\n  \"object\": \"list\",\n  \"data\": [\n    {\n      \"id\": \"Qwen\/Qwen3-14B\",\n      \"object\": \"model\",\n      \"created\": 1766511858,\n      \"owned_by\": \"vllm\",\n      \"root\": \"Qwen\/Qwen3-14B\",\n      \"parent\": null,\n      \"max_model_len\": 40960,\n      \"permission\": [\n        {\n          \"id\": \"modelperm-bc2e247073d50d67\",\n          \"object\": \"model_permission\",\n          \"created\": 1766511858,\n          \"allow_create_engine\": false,\n          \"allow_sampling\": true,\n          \"allow_logprobs\": true,\n          \"allow_search_indices\": false,\n          \"allow_view\": true,\n          \"allow_fine_tuning\": false,\n          \"organization\": \"*\",\n          \"group\": null,\n          \"is_blocking\": false\n        }\n      ]\n    }\n  ]\n}<\/code><\/pre>\n\n\n\n
To deploy a model, I created the following systemd<\/code> unit file in \/etc\/systemd\/system<\/code> called vllm.service<\/code>. This way vLLM will automatically start when the host is rebooted.<\/p>\n\n\n\n
[Unit]\nDescription=vLLM OpenAI Compatible Server\nAfter=network.target\n\n[Service]\n# User and Group to run the service as (e.g., 'youruser', 'yourgroup')\nUser=kang\nGroup=kang\n# Set the working directory\nWorkingDirectory=\/home\/kang\/py_vllm\nEnvironment=VLLM_USE_V1=1\n\n# The command to start the vLLM server\n# Use 'exec' to ensure systemd correctly manages the process\nExecStart=\/home\/kang\/py_vllm\/vllm_cuda13_env\/bin\/python -m vllm.entrypoints.openai.api_server --model Qwen\/Qwen3-14B --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 --enable-auto-tool-choice --tool-call-parser hermes\n\n# Restart the service if it fails\nRestart=always\n\n[Install]\nWantedBy=multi-user.target<\/code><\/pre>\n\n\n\n
I used 0.0.0.0<\/code> as the host so that any machine on the network can connect to the service. If you use 127.0.0.1<\/code>, only local sessions can connect.<\/p>\n\n\n\n
To enable the above service, I had to do the following:<\/p>\n\n\n\n
\u276f sudo systemctl daemon-reload\n\u276f sudo systemctl enable vllm.service\n\u276f sudo systemctl start vllm.service<\/code><\/pre>\n\n\n\n
I also enabled tooling for my Opencode.ai<\/a> experiments. vLLM ended up using all of 48GB VRAM on both GPU’s for the Qwen LLM as well as for caching. Impressive!<\/p>\n\n\n\n
Installing Podman and Prepare for Quadlets<\/h2>\n\n\n\n
For everyday chats, I also configured a version of Perplexica<\/a>. I chose to use Podman to install this, specifically using Podman Quadlet<\/a>. The idea is to run Perplexica under my user id (kang<\/code>), instead of running it as root<\/code>. Our first step is to install Podman and prepare our user account for quadlets.<\/p>\n\n\n\n
Note aside from explicit sudo<\/code> references all other commands are run as the user.<\/p>\n\n\n\n
Install Podman:<\/p>\n\n\n\n
sudo apt install podman<\/code><\/pre>\n\n\n\n
The container requires user and group ids so we need to map id spaces to my user account.<\/p>\n\n\n\n
sudo usermod --add-subuids 100000-170000 --add-subgids 100000-170000 ${USER}\n\ncat \/etc\/subuid\ncat \/etc\/subgid<\/code><\/pre>\n\n\n\n
We need to have an active user session for the container after a reboot, so we need my account to linger around.<\/p>\n\n\n\n
sudo loginctl enable-linger ${USER}<\/code><\/pre>\n\n\n\n
We need to proactively increase the kernel key size to avoid any exceeding quota situations like, \u201cDisk quota exceeded: OCI runtime error\u201d<\/code>. Not just for this container, but also for any other future containers.<\/p>\n\n\n\n
echo \"kernel.keys.maxkeys=1000\" | sudo tee -a \/etc\/sysctl.d\/custom.conf<\/code><\/pre>\n\n\n\n
Lastly, we need to prepare two directories for the containers. The first will house the systemd<\/code> unit definition of the container. The second is a directory that will act as local storage for the container.<\/p>\n\n\n\n
mkdir -p $HOME\/.config\/containers\/systemd\nmkdir -p $HOME\/containers\/storage<\/code><\/pre>\n\n\n\n
If we have any previous containers running, we need to perform a system migrate. I did not perform this, because I ensure that I had no other Podman containers running. You also can enable the auto update feature for podman. I also did not do this, as I prefer to this manually.<\/p>\n\n\n\n
podman system migrate\nsystemctl --user enable --now podman-auto-update<\/code><\/pre>\n\n\n\n
For a more control networking experience and behaviour, we want to create our own container network. This will also help with DNS resolution. We need to create the network definition in $HOME\/.config\/containers\/systemd\/${USER}.network<\/code> be sure to replace ${USER}<\/code> reference below with the actual user account name.<\/p>\n\n\n\n
[Unit]\nDescription=${USER} network\nAfter=podman-user-wait-network-online.service\n \n[Network]\nNetworkName=${USER}\nSubnet=10.168.0.0\/24\nGateway=10.168.0.1\nDNS=192.168.168.198\n \n[Install]\nWantedBy=default.target<\/code><\/pre>\n\n\n\n
We can then enable this network with the following commands:<\/p>\n\n\n\n
systemctl --user daemon-reload\nsystemctl --user start ${USER}-network\n\npodman network ls<\/code><\/pre>\n\n\n\n
The last command just verifies that the network is running and visible to Podman.<\/p>\n\n\n\n
Installing Perplexica<\/h2>\n\n\n\n
Now that our Quadlet environment for the user account is all prepared, we can then proceed to install Perplexica.<\/p>\n\n\n\n
First we need to create two local directories that Perplexica will use.<\/p>\n\n\n\n
mkdir -p $HOME\/containers\/storage\/perplexica\/data\nmkdir -p $HOME\/containers\/storage\/perplexica\/uploads<\/code><\/pre>\n\n\n\n
We then need to define the container in $HOME\/.config\/containers\/systemd\/perplexica.container<\/code>:
<\/p>\n\n\n\n
[Unit]\nDescription=Perplexica\n\n[Container]\nContainerName=perplexica\nImage=docker.io\/itzcrazykns1337\/perplexica:latest\nAutoUpdate=registry\n\nHealthCmd=curl http:\/\/localhost:3000\nHealthInterval=15m\nUserNS=keep-id:uid=1000,gid=1000\n\nNetwork=kang.network\nHostName=perplexica\nPublishPort=3000:3000\n\nVolume=%h\/containers\/storage\/perplexica\/data:\/home\/perplexica\/data\nVolume=%h\/containers\/storage\/perplexica\/uploads:\/home\/perplexica\/uploads\n\n[Service]\nRestart=always\nTimeoutStartSec=300\n\n[Install]\nWantedBy=default.target<\/code><\/pre>\n\n\n\n
Be sure to double check your account uid<\/code> and gid<\/code> is 1000. If not, then replace the above appropriately.<\/p>\n\n\n\n
Now we can start Perplexica.<\/p>\n\n\n\n
systemctl --user daemon-reload\nsystemctl --user start perplexica<\/code><\/pre>\n\n\n\n
Note that the above commands are run with the user account and not with sudo<\/code> or as root<\/code>. Also note the --user<\/code> option.<\/p>\n\n\n\n
Once the service is running, you can get its logs by doing the following:<\/p>\n\n\n\n
journalctl --user -u perplexica<\/code><\/pre>\n\n\n\n
You can also see all containers running as quadlets using:<\/p>\n\n\n\n
systemctl --user status<\/code><\/pre>\n\n\n\n
With Perplexica running, we can proceed to its Web UI (http:\/\/localhost:3000<\/code>) using our browser and point to our vLLM instance by creating an OpenAI<\/strong> connection type.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Once the connection is established, you can proceed to add the Chat and Embedding Models. In our case I used Qwen\/Qwen3-14B<\/code> as the model key. This is the same as the model id that vLLM is currently serving. The model name can be anything you assign.<\/p>\n\n\n\n
That is it! We now have a local chat service with Perplexica, and I can use the OpenAI compatible API with vLLM.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Here is an example of using CURL with the API:<\/p>\n\n\n\n
\u276f curl -X POST \"http:\/\/localhost:8000\/v1\/responses\" \\\n  -H \"Content-Type: application\/json\" \\\n  -d '{\n    \"model\": \"Qwen\/Qwen3-14B\",\n    \"input\": \"How are you today?\"\n  }' | jq -r '.output[0].content[0].text'\n  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n100  1791  100  1721  100    70    448     18  0:00:03  0:00:03 --:--:--   466\n<think>\nOkay, the user asked, \"How are you today?\" I need to respond appropriately. First, I should acknowledge their greeting and express that I'm doing well. Since I don't have feelings, I can't experience emotions, but I can simulate a friendly response. I should keep it positive and open-ended to encourage further conversation. Maybe add an emoji to keep it friendly. Also, I should invite them to ask questions or share something. Let me check if the response is natural and not too robotic. Avoid any technical jargon. Make sure it's concise but warm. Alright, that should work.\n<\/think>\n\nI'm doing great, thank you! I'm always ready to chat and help out. How about you? \ud83d\ude0a What's on your mind today?<\/code><\/pre>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
In a previous post, I commented on our AI server containing an old P40 GPU failed. We replaced our server with the following parts. Component Description CPU AMD Ryzen 9 9900X 4.4 GHz 12-Core Processor CPU Cooler Thermalright Peerless Assassin 120 SE 66.17 CFM CPU Cooler Motherboard Asus B650E MAX GAMING WIFI ATX AM5 Motherboard … Continue reading “New AI Server”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[111],"tags":[142,165,141,196,28,195],"class_list":["post-3303","post","type-post","status-publish","format-standard","hentry","category-tech","tag-ai","tag-cuda","tag-llm","tag-perplexica","tag-technology","tag-vllm"],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p7V6i8-Rh","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/posts\/3303","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/comments?post=3303"}],"version-history":[{"count":15,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/posts\/3303\/revisions"}],"predecessor-version":[{"id":3324,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/posts\/3303\/revisions\/3324"}],"wp:attachment":[{"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/media?parent=3303"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/categories?post=3303"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/tags?post=3303"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}