My rack is finished for now (because I’m out of money).
Last time I posted I had some jank cables going through the rack and now we’re using patch panels with color coordinated cables!
But as is tradition, I’m thinking about upgrades and I’m looking at that 1U filler panel. A mini PC with a 5060ti 16gb or maybe a 5070 12gb would be pretty sick to move my AI slop generating into my tiny rack.
I’m also thinking about the PI cluster at the top. Currently that’s running a Kubernetes cluster that I’m trying to learn on. They’re all PI4 4GB, so I was going to start replacing them with PI5 8/16GB. Would those be better price/performance for mostly coding tasks? Or maybe a discord bot for shitposting.
Thoughts? MiniPC recs? Wanna bully me for using AI? Please do!
You must log in or register to comment.
Wow! very cool rack you got there. I too started using mini pcs for local test servers or general home servers. But unlike yours mine are just dumped behind the screen on my desk (3 in total). For LLM stuff atm I use 16GB radeon but thats connected to my desktop. In the future I would love to build a proper rack like yours and perhaps move the GPU to a dedicated minipc.
As for the upgrades, like what others stated already, I would just go for more pc’s rather then rpi.
The PIs were honestly because I had them.
I think I’d rather use them for something else like robotics or a Birdnet pi.
But the pi rack was like $20 and hilarious.
The objectively correct answer for more compute is more mini PCs though. And I’m really thinking about the Mac Mini option for AI.
This will get downvoted to oblivion because this is Lemmy:
Get a Mac Mini. Any M-series model with 32GB of memory will run local models at decent speeds and will be cheaper than just a 5xxx series GPU
And it’ll fit your cool rack 😀
How much did this cost?
The Lenovo Thinkcentre M715q were $400 total after upgrades. I fortunately had 3 32 GB kits of ram from my work’s e-waste bin but if I had to add those it would probably be $550 ish The rack was $120 from 52pi I bought 2 extra 10in shelves for $25 each the Pi cluster rack was also $50 (shit I thought it was $20. Not worth) Patch Panel was $20 There’s a UPS that was $80 And the switch was $80
So in total I spent $800 on this set up
To fully replicate from scratch you would need to spend $160 on raspberry pis and probably $20 on cables
So $1000 theoratically
A single raspberry pi 5 can host an entire website
True, but I have an addiction and that’s buying stuff to cope with all the drawbacks of late stage capitalism.
I am but a consumer who must be given reasons to consume.
Scatter them about the place. Make appliances smart. Would be more interesting than a cluster.
Ohh nice, I want it. Don’t really know what I would use all of it for, but I want it (but don’t want to pay for it).
Currently been thinking of getting an N150 mini PC. Setup proxmox and a few VMs. At the very least pihole, location to dump some backups and also got a web server for a few projects.
Looking good! Funny I happen across this post when I’m working on mine as well. As I type this I’m playing with a little 1.5” transparent OLED that will poke out of the rack beside each pi, scrolling various info (cpu load/temp, IP, LAN traffic, node role, etc)
What OLED specifically and what will you be using to drive it?
Waveshare 1.51” transparent OLED. Comes with driver board, ribbon & jumpers. If you type it in Amazon it’s the only one that pops, just make sure it says transparent. Plugs into GPIO of my Pi 5s. The Amazon listing has a user guide you can download so make sure to do that. I was having trouble figuring it out until I saw that thing. Runs off a python script but once I get it behaving like I want I’ll add it to systemd so it boots on startup.
Imma dummy so I used ChatGPT for most of it, full …ahem… transparency. 🤷🏻♂️
I’m modeling a little bracket in spaceclaim today & will probably print it in transparent PETG. I’ll post a pic when I’m done!
Well, I always advocate for using the stuff you have. I don’t think a Discord bot needs four new RasPi 5. That’s likely to run on a single RasPi3. And as long as they’re sitting idle, it doesn’t really matter which model number they have… So go ahead and put something on your hardware, and buy new one once you’ve maxed out your current setup.
I’m not educated on Bazzite. Maybe tools like Distrobox or other container solutions can help running AI workloads on the gaming rig. It’s likely easier to run a dedicated AI server, but I started learning about quantization, tested some models on my main computer with the help of ollama, KoboldCPP and some random Docker/Podman containers. I’m not saying this is the preferrable solution. But definitely enough to get started with AI. And you can always connect the computers within your local network, write some server applications and have them hook into ollama’s API and it doesn’t really matter whether that runs on your gaming pc or a server (as long as the computer in question is turned on…)
You could probably run several discord bots on a Raspberry Pi 3, provided they aren’t public and popular
Ollama and all that runs on it its just the firewall rules and opening it up to my network that’s the issue.
I cannot get ufw, iptables, or anything like that running on it. So I usually just ssh into the PC and do a CLI only interaction. Which is mostly fine.
I want to use OpenWebUI so I can feed it notes and books as context, but I need the API which isn’t open on my network.
Thanks for the info. Some day I’ll try the shiny modern distros and learn the little peculiarities. I use a weird mix of Debian, NixOS and LMDE and it’s relatively straightforward to add firewall rules to those, both dynamically to nftables and to the persistent config… And I believe Debian didn’t even come with firewalling out of the box… But I understand Debian might not be the best choice for gaming and there is for example some extra work involved to get the latest Nvidia drivers. Neither is it an atomic distro.
Honestly if you’re not gaming or playing with new hardware, there is absolutely no point.
I’ve considered swapping this computer over to Fedora for a hot minute, but it really is a gaming PC and I should stop trying to break it.
This is so pretty 😍🤩💦!!
I’ve been considering a micro rack to support the journey, but primarily for house old laptop chassis as I convert them into proxmox resources.
Any thoughts or comments on you choice of this rack?
Not really a lot of thought went into rack choice. I wanted something smaller and more powerful than my several optiplexs I had.
I also decided I didn’t want storage to happen here anymore because I am stupid and only knew how to pass through disks for Truenas. So I had 4 truenas servers on my network and I hated it.
This was just what I wanted at a price I was good with at Like $120. There’s a 3D printable version but I wasn’t interested in that. I do want to 3D print racks and I want to make my own custom ones for the Pis to save space.
But this set up is way cheaper if you have a printer and some patience.
for your usecase, i’d get an external gpu to plug in to one of these juicy thinkstations right there. bonus for the modularity of having an actual gpu instead of relying on whatever crappy laptop gpu minipc manufacturers put in there.
you could probably virtualize a sick gaming setup with this rig too. stream it to your phone/laptop.
nice setup btw.
I’m afraid I’m going to have to deduct one style point for the misalignment of the labels on the mini PCs.
That’s fair and justified. I have the label maker right now in my hands. I can fix this at any moment and yet I choose not to.
I’m man feeding orphans to the orphan crushing machine. I can stop this at any moment.
The machine must keep running!
Honestly, If you are delving into Kubernetes, just add some more of those 1L PCs in there. I tend to find them on ebay cheaper than Pi’s. Last year I snagged 4x 1L Dells with 16GB RAM for $250 shipped. I swapped some RAM around, added some new SSD’s and now have 3x Kube masters, 3x Kube worker nodes and a few VMs running a Proxmox cluster across 3 of the 1L’s with 32GB and a 512GbB SSD each and its been great. The other one became my wife’s new desktop.
Big plus, there are so many more x86_64 containers out there compared to Pi compatible ARM ones.
I have a question about ai usage on this: how do you do this? Every time I see ai usage some sort of 4090 or 5090 is mentioned, so I am curious what kind of ai usage you can do here
I’m running ai on an old 1080 ti. You can run ai on almost anything, but the less memory you have the smaller (ie. dumber) your models will have to be.
As for the “how”, I use Ollama and Open WebUI. It’s pretty easy to set up.
Similar setup here with a 7900xtx, works great and the 20-30b models are honestly pretty good these days. Magistral, Qwen 3 Coder, GPT-OSS are most of what I use
I tried a couple of times with Jen ai and local llama, but somehow does not work that well for me.
But at the same time i have a 9070xt, so, not exactly optimal
Your options are to run smaller models or wait. llama3.2:3b fits on my 1080 Ti VRAM and is sufficiently fast. Bigger models will get split between VRAM and RAM and run slower but it’ll work.
Not all models are Gen AI style LLMs. I run GPU based speech to text models on my GPU too for my smart home.
With a RTX 3060 12gb, I have been perfectly happy with the quality and speed of the responses. It’s much slower than my 5060ti which I think is the sweet spot for text based LLM tasks. A larger context window provided by more vram or a web based AI is cool and useful, but I haven’t found the need to do that yet in my use case.
As you may have guessed, I can’t fit a 3060 in this rack. That’s in a different server that houses my NAS. I have done AI on my 2018 Epyc server CPU and its just not usable. Even with 109gb of ram, not usable. Even clustered, I wouldn’t try running anything on these machines. They are for docker containers and minecraft servers. Jeff Geerling probably has a video on trying to run an AI on a bunch of Raspberry Pis. I just saw his video using Ryzen AI Strix boards and that was ass compared to my 3060.
But to my use case, I am just asking AI to generate simple scripts based on manuals I feed it or some sort of writing task. I either get it to take my notes on a topic and make an outline that makes sense and I fill it in or I feed it finished writings and ask for grammatical or tone fixes. Thats fucking it and it boggles my mind that anyone is doing anything more intensive then that. I am not training anything and 12gb VRAM is plenty if I wanna feed like 10-100 pages of context. Would it be better with a 4090? Probably, but for my uses I haven’t noticed a difference in quality between my local LLM and the web based stuff.
So is not on this rack. OK because for a second I was thinking somehow you were able to run ai tasks with some sort of small cluster.
I have nowadays a 9070xt on my system. I just dabbled on this, but until now I havent been that successful. Maybe I will read more into it to understand better.
Ollama + Gemma/Deepseek is a great start. I have only ran AI on my AMD 6600XT and that wasn’t great and everything that I know is that AMD is fine for gaming AI tasks these days and not really LLM or Gen AI tasks.
A RTX 3060 12gb is the easiest and best self hosted option in my opinion. New for <$300 and used even less. However, I was running with a Geforce 1660 ti for a while and thats <$100
deleted by creator
NSFW
I didn’t even know these sorts of mini racks existed. now I’m going to have to get one for all my half sized preamps if they’ll fit. That would solve like half the problems with my studio room and may help bring back some of my spark for making music.
I have no recs. Just want to say I’m so excited to see this. I can probably build an audio patch panel.
https://en.wikipedia.org/wiki/ThinkCentre because I didn’t knew it existed.
These are M715q Thinkcentres with a Ryzen Pro 5 2400GE
Oh and my home office set up uses Tiny in One monitors so I configured these by plugging them into my monitor which was sick.
I’m a huge fan of this all in one idea that is upgradable.
