What’s happening on your servers? Any interesting news things you tried?
I didn’t do anyone other than updating Mastodon (native deployment) lately due to a lack of time. Reading so much about Immich caused me to consider trying it in parallel to Nextcloud but I’m not sure if I want to have everything twice.
Not quite homelab, but I’m about to install Linux Mint on my mom’s laptop and that had me thinking about creating an off-site backup in her place again since she has a fiber connection. I’m still not sure about the potential design though, but currently my only backup is in the same rack as the live stuff.
I finally got my ISP to enable bridge mode on my modem.
I also learned that I didn’t lose port forwarding and related services because I had been moved behind CGNAT or transitioned to IPv6 – they simply no longer offer port forwarding to residential customers. Ruminate on the implications of that statement so I’m not the only one with blood pressure in the high hundreds.
My ISP did the same thing recently and what was most annoying is they didn’t admit to changing anything, while trying to sell me a business account.
This weekend I setup Pangolin on a budget VPS and forwarded it back home. I don’t have my VPN backup but it fixed Plex and I can access my security cameras again.
Port forwarding is done at the router/firewall, so if ports can’t be transferred its a cgnat thing they are doing. Like a Non CGNAT IP on the internet can be sent a packet on any port.
No, I got it from the horse’s mouth: my WAN address was publicly routable all along, the ISP just disabled those NAT-related features remotely.
Oh shit, that’s terrible.
the implication of that is weird to me. I’m not saying that the horse is wrong, but thats such a non-standard solution. That’s implementing a CGNAT restriction without the benefits of CGNAT. They would need to only allow internal to external connections unless the connection was already established. How does standard communication still function if it was that way, I know that would break protocols like basic UDP, since that uses a fire and forget without internal prompting.
It’s perfectly reasonable from the perspective of corporate scum: take away a standard feature, then sell it back as an extra. As far as I know, the modem still had UPnP for applications that rely on it.
I am playing around with Podman Quadlet and that’s one hell of a rabbit hole. I have everything up and running, and now I need to configure the containers, and probably will deal with other pain points, etc.
The good thing is that I have documented the whole process so it is reproducible but it took me quite some time to figure out everything.
Would you mind sharing your process in a write up?
I will definitely do that, I just want to finish the whole setup.
So, serious question, should I self-host my servers in AWS?
Why would you?
Currently working on a networking problem. I have multiple Proton VPN connections on my Mikrotik router. Main reason being for fail over in case one endpoint reaches capacity, goes unresponsive, etc.
It’s a bit tricky since Proton issues the same peer and gateway IP for each connection. Haven’t quite got it working the way I want it to yet.
I threw a thinkcenter in my laundry room and did the bare minimum to securely SSH into it (fail2ban, nonstandard port, root login disabled, can’t login with password, etc), to be used as a testing platform for building my workplace a new website.
Just gotta relearn HTML/CSS and figure out what platform to use.
Also set up traefik/Authelia/maybe Anubis for the new domain and block any access outside of my home or workplace.
Finally finished setting up and testing a Peertube instance. The video stuff and object storage related things certainly make it more involved than other fediverse software, but overall it is working quite nicely. Just need to find some workable solution to using GPU acceleration in containers, but I think I mostly figured it out (might work after a server restart, but my sweet, sweet uptime makes me procrastinate on that 😅 ).
How much storage do you think you’ll need with caching external content? Does Peertube even do that?
Not automatically, but you can configure it to mirror certain video channels or individual videos. But I have not looked into that too much yet.
As for storage: a typical video you would find on such a platform with the different stored video resolutions and so on will take between 0.5 and 3 GB… depending on the length and how well it compresses.
I installed immich and began migrating our phones away from Google.
Set up Zipline to share bigger files with my friends.
I’ve been deploying Gitea (or Forgejo, still can’t decide), but I’ve fallen into the Ansible rabbit hole and can’t get out. Also learned Terraform in the last week and I’m still on the fence about using it in my homelab. It’s nice for the cloud but I don’t think it’s as useful on-prem.
Forgejo has everything Gitea has, with more and being more open
My concern when it forked was that forgejo would last a few months and then fizzle out.
That doesn’t seem to be the case.
Yeah, I evaluated my position since and now I’m trying to deploy Forgejo, but I’m still stuck in the IaC rabbit hole and can’t crawl out
Working on setup reserve proxy properly. With all this research and testing, im going to be ans expert in the area, just to never speak about to another human being… except on and another post
I’ve been making another attempt to replace Docker with Podman. The issue is I can’t connect to my server through a web browser. I think it’s a firewall issue.
Networking and networking troubleshooting is a bit confusing for me and that’s the least favourite part about self hosting for me. Turns out I actually enjoy writing scripts more and the challenge of writing POSIX scripts especially.
If I can figure it out, I’ll probably write a guide for setting up Podman and Caddy on Alpine Linux since there isn’t a lot of recent information out there from what I found in my searches so far.
Good luck 🫡 I made the switch about half a year ago and went all in on rootless quadlets while I was at it. It was a pretty nightmarish couple weeks figuring out things like user id mappings and rootless permissions, but I got there eventually. Landed on a super neat Traefik config that should work for anyone and makes spinning up new quadlets with their own reverse proxied subdomains really simple. I should really post it somewhere…
In the end I wouldn’t exactly say it was worth it… but it sure feels cool to be fully moved into a more open/native container implementation.
Yeah, I mainly just want to move away to more open projects. When I first started, everyone kept suggesting using Cloudflare. After half a year using their service, I just felt icky the entire time.
In the past couple months I was able to move away and chose to protect myself by learning how to harden my server as well as hiding my server behind multiple layers of obscurity.
With my current setup, the only site traffic I get has only been myself and my custom ssh port only gets hit by bots about 3-10 times a week according to my logs. Only time will tell how effective my layers of obscurity will hold up but so far it seems to satisfy my needs better than I was expecting.
Once I get podman in a state I like, I’ll pretty much be all open sourced and all I’ll have to do for myself is be in maintenance mode unless I care to add a new service. I like to keep things simple so I don’t normally go crazy adding new services anyways.
Rootless podman cannot bind ports <1024, only root can by default (on pretty much any distro I guess). Have you done something like
sysctl net.ipv4.ip_unprivileged_port_start=80to allow non-root processes to bind to port numbers >=80?I’ve read about that and I already have that in my notes as well.
It doesn’t really affect my needs because my ISP blocks incoming on those ports anyways. Also I’m choosing not to use a tunnel at the moment so I’ll be using a higher port anyways.
The last time I asked about it, a few people seemed to agree it was something to do with the firewall settings. That seems most likely since I was able to connect when I disabled my firewall. I’m not a fan of working with iptables. The language for that type of networking is gibberish to me.
I had also tried going from docker compose to rootful podman compose and ran into the same issue. Although I’m trying to work away from podman compose in the future, just taking it in steps.
Did the switch from Docker to Podman a couple of months ago. Now I host all my services (arr-stack, Forgejo, Nextcloud, Authelia, Traefik, Immich… to name a few) on my VPS and mini pc/home server with Podman.
I recently sat up headscale to connect my VPS running the Traefik Proxy to my home lab to make some of my services running on there accessible from the internet. It was quite the journey, to say the least, as networking is not my forte either.
But feel free to drop me a pm if you need some inspiration or support, maybe I can help.
Thank you for the offer. I still need a bit more more time to experiment and zero in on the issue again. Fortunately my setup is quite simple and the only bottleneck will be Caddy.
I basically run Caddy which redirects to a static generated blog, simple file server page and a Kiwix instance. I’m mostly making a self hosted reference site of materials for Linux and Scripting resources.
One day I may add a Forgeo instance but currently my entire workflow exists around rsync. I’m happy just having my single file scripts hosted as text files and don’t really need the power of git. At least not at the moment.
I’ve had immich but went to homegalley instead. Mostly because I want to keep MY directory structure in case I’m abandoning the choosen platform. Have not regretted my choice (so far … 8 months)
You can adjust the directory structure in immich using templates
I’ve not been able to make it work reliably with photos backed up using immich on my android phone, is if working for you? I read somewhere storage templates are not very robust/reliable.
Seems to be working fine for me but i don’t do anything complex, just folders by year and month
Same, using the default storage template.
I’ve been using Immich, but with my photos as external media. That lets me keep my directory structure too, but with the Immich features 🙂
Building out ansible.
Now it’s creating roles and groups, adding a few items to the hardening playbook, and I’ve been playing with tuning the output as playbooks run.
I’ve been trying to convince a VPS to run two instances of mariadb - one for local databases, one to replicate the homelab. Got mariadb@server and mariadb@replica sorted out through systemd, but now stuck on replication from mysql to mariadb. Looks like I’ll be ripping out mariadb and putting everything on mysql.
Have you checked if statement-based replication works from mysql to mariadb?
I’m hung up on unrecognized charset #255. Tried rolling everything back to utfmb3; suppose I could go all the way to Latin1. I imagine there’s a lot of depth I could learn, but dropping mariadb for mysql seems like the path of least resistance right now.
eta: got the character set sorted. Had to make a new dump, confirm that everything in the dump was utf8mb3, then re-prime the replica with that data. Wasn’t enough just to change the character sets internally.
So it works now! Good job
Interesting using systemd for that, I’d probably have chosen containers for that.
What’s the reason for replication vs. dumps? Does the client failover to the replica?
I’m not a systemd guru, but it turned out pretty easy. https://dev.mysql.com/doc/refman/5.7/en/using-systemd.html#systemd-multiple-mysql-instances Basically just make
[]sections in my.cnf thensystemd start mysqld@copyand systemd is smart enough to passcopyinto mysql.I did it slightly different, using
systemctl edit mysql@.serviceto define different default files for each instance, then[]sections in each of those files. Seems like theportoption for each has to go in a[]section, but otherwise ok.Replication because I want to put some live data, read-only, on the VPS, exposed to the world while the ‘real’ database stays safely hidden in my intranet. SSH tunnel so the replica can talk to the real database.
I actually just wrote about today’s fun experience! https://gotosocial.michaeldileo.org/@mdileo/statuses/01K7YKQ9584YBY1QTYQ8RMW7SS







