Hi everyone, I’ve been working on my homelab for a year and a half now, and I’ve tested several approaches to managing NAS and selfhosted applications. My current setup is an old desktop computer that boots into Proxmox, which has two VMs:
- TrueNAS Scale: manages storage, shares and replication.
- Debian 12 w/ docker: for all of my selfhosted applications.
The applications connect to the TrueNAS’ storage via NFS. I have two identical HDDs as a mirror, another one that has no failsafe (but it’s fine, because the data it contains is non-critical), and an external HDD that I want to use for replication, or some other use I still haven’t decided.
Now, the issue is the following. I’ve noticed that TrueNAS complains that the HDDs are Unhealthy and has complained about checksum errors. It also turns out that it can’t run S.M.A.R.T. checks, because instead of using an HBA, I’m directly passing the entire HDDs by ID to the VM. I’ve read recently that it’s discouraged to pass virtualized disks to TrueNAS, as data corruption can occur. And lately I was having trouble with a selfhosted instance of gitea, where data (apparently) got corrupted, and git was throwing errors when you tried to fetch or pull. I don’t know if this is related or not.
Now the thing is, I have a very limited budget, so I’m not keen on buying a dedicated HBA just out of a hunch. Is it really needed?
I mean, I know I could run TrueNAS directly, instead of using Proxmox, but I’ve found TrueNAS to be a pretty crappy Hypervisor (IMHO) in the past.
My main goal is to be able to manage the data that is used in selfhosted applications separately. For example, I want to be able to access Nextcloud’s files, even if the docker instance is broken. But maybe this is just an irrational fear, and I should instead backup the entire docker instances and hope for the best, or maybe I’m just misunderstanding how this works.
In any case, I have some data that I want to store and want to reliably archive, and I don’t want the docker apps to have too much control over it. That’s why I went with the current approach. It has also allowed for very granular control. But it’s also a bit more cumbersome, as everytime I want to selfhost a new app, I need to configure datasets, permissions and mounting of NFS shares.
Is there a simpler approach to all this? Or should I just buy an HBA and continue with things as they are? If so, which one should I buy (considering a very limited budget)?
I’m thankful for any advice you can give and for your time. Have a nice day!
Proxmox supports ZFS natively with management in the WebUI. So you could get rid of TrueNAS entirely and not need to deal with HBA pass-through or anything.
You also wouldn’t need NFS or have to deal with shares, as the data is available directly to Proxmox Containers via bind mounts.
Okay, if Proxmox can handle all that, I’ll be glad to ditch TrueNAS. However, I’m afraid that I won’t know how to migrate. I’ve found this reddit thread about someone who tried to do the same thing (I think) and accidentally corrupted their pools. About skipping NFS shares, that would be a big improvement for me, but I’m very unfamiliar with bind mounts. If I understand correctly, you can specify directories that live on the Proxmox Host, and they appear inside the VM, right? How does this compare to using virtual storage? Also, how can I replicate the ZFS pools to an external machine? In any case, thank you for that info!
Migration should be as simple as importing the existing ZFS stuff into the Proxmox OS. Having backups of important data is critical though.
If I understand correctly, you can specify directories that live on the Proxmox Host, and they appear inside the VM, right?
Inside a Container, yep. VMs can’t do bind mounts, and would need to use NFS to share existing data from the host to inside the VM.
How does this compare to using virtual storage?
Like a VM virtual disk? Those are exclusive to each VM and can’t be shared, so if you want multiple VMs to access the same data then NFS would be needed.
But containers with bind mounts don’t have that limitation and multiple containers can access the same data (such as media).
Also, how can I replicate the ZFS pools to an external machine?
ZFS replication would do that.
Okay, I have one of the pools that is pretty empty and has non-critical data, and I think I’ll try migrating that first, and see if it’s imported correctly by Proxmox.
About Containers, I think I’ll have to do some more research because I think I haven’t fully understood yet how they compare to VMs. Or like, when I should use the one over the other. I guess I could have a Container with a bind mount to a dataset that I want to be able to share over NFS or SMB, and handle that from whatever OS I put in the Container, right? But, I could also have a VM do that, and though it wouldn’t be able to share the data with other VMs, it can do it over NFS, can’t it? What are the advantages of doing one thing over the other?
Well, in any case, thank you for your patience, for going over each detail and taking the time to correct me where I’m wrong. I’m learning a lot, so thank you!
Edit: fixing grammar
Essentially a container shares the kernel of the host, so uses less resources to run.
VMs are useful when you need more isolation or a different kernel (or need to add kernel modules).
For most purposes containers are the easy option.
I guess I could have a Container with a bind mount to a dataset that I want to be able to share over NFS or SMB, and handle that from whatever OS I put in the Container, right?
Yep!
But, I could also have a VM do that, and though it wouldn’t be able to share the data with other VMs, it can do it over NFS, can’t it?
Also yes, just a more complex setup with more performance penalty due to using NFS to share data into the VM.
I see, okay, I’ll try out containers then. So far, I’ve been able to migrate a ZFS Pool without issues, so I’ll start migrating them all, create a container that manages NFS and see if the existing Docker VM picks up the NFS shares successfully. Thank you for going in-depth and explaining everything to me. I’ve learnt a lot!
you can install dockge in truenas and then all your docker data is not “locked” inside their application data
Yeah, actually I had that setup for a brief amount of time, but I had issues because my system has too little RAM, and TrueNAS and Docker were constantly fighting over RAM, and the system hang frequently. For some mindboggling reason, if I ran TrueNAS and a Docker VM separately on Proxmox, were I could manually specify allowed RAM, everything worked perfectly fine, even though the available RAM was still very low. If I ever go down this road again, I’ll need to buy more RAM sticks.
I run a docker host in Proxmox using ZFS datasets for the VM storage for things like my mailserver and NexcloudAIO. When I backup the docker VM, it snapshots the VM at a point in time, and backs up the snapshot to PBS. I’ve restored from that backup and it’s like the machine had just shut down as far as the data is concerned. It journals itself back to a consistent state and no data loss.
I wouldn’t run TrueNAS at all because I have no idea how that’s managing it’s storage and wouldn’t trust the result.
Wait, so if I understood correctly, you’re managing the ZFS pools directly in Proxmox, and then you have a VM that’s running docker, and using the storage that is managed by Proxmox, right? Hmm, sounds like a good solution. Is there any documentation or article that you could recommend, so that I can take a closer look? Also, how could I handle SMB shares?
Yes. So my debian docker host has some datasets attached:
mounted via fstab:
and I specify that path as the datadir for NCAIO:
Then when PBS calls a backup of that VM, all the datasets that Proxmox is managing for that backup take a snapshot, and that’s what’s backed up to PBS. Since it’s a snapshot, I can backup hourly if I want, and PBS dedups so the backups aren’t using a lot of space.
Other docker containers might have a mount that’s used as a bind mount inside the compose.yml to supply data storage.
Also, I have more than one backup job running on PBS so I have multiple backups, including on removable USB drives that I swap out (I restart the PBS server to change drives so it automounts the ZFS volumes on those removable drives and is ready for the next backup).
You could mount ZFS datasets you create in Proxmox as SMB shares in a sharing VM, and it would be handled the same.
As for documentation, I’ve never really seen any done this way but it seems to work. I’ve done restores of entire container stacks this way, as well as walked the backups to individually restore files from PBS.
If you try it and have any questions, ping me.
Wow, that’s awesome. I think that’s actually the approach I’m going to go for. This way I don’t need to buy hardware, and I don’t need to work with TrueNAS anymore.
Where you talk about “walking the backups”, do you mean that you can actually see the entire file structure of the container? I mean, I don’t know how virtual disks are stored on the dataset. Like, as far as I know, a VM virtualized disk is just a file, right? So you’d have a ZFS dataset with a single file, for example? Could you then try and navigate the files inside this VM disk file, without the VM? Or did I misunderstand, and you’re mounting the dataset, somehow, directly inside the VM? Is that like a passthrough for datasets?
In any case, thank you for sharing so much information and for offering help. I may take you up on that, as it seems that this is the approach that I feel most comfortable with.
So if I want a new container stack, I make a new Proxmox “disk” in the ZFS filesystem under the Hardware tab of the VM. This adds a “disk” to the VM when I reboot the VM (there are ways of refreshing the block devices online, but this is easier). I find the new block device and mount it in the VM at a subfolder of /stacks, which will be the new container stack location. I also add this mount point to fstab.
So now I have a mounted volume at /stacks/container-name. I put a docker-compose.yml in there and all data that the stack will use will be subfolders of that folder with bind mounts in the compose file. When I back up, that ZFS dataset that contains everything in that compose stack is snapshotted and backed up as a point-in-time. If that stack has a postgres database, it and all the data it references is internally consistent because it was snapshotted before backup. If I restore the entire folder from backup, it just thinks it had a power outage, replays it’s journals in the database, and all’s well.
So when you have a backup in PBS, from your Proxmox node you can access the backups via the filesystem browser on the left.
When you go to that backup, you can choose to do a File Restore instead of restoring the entire VM. Here I am walking the storage for my nextcloud data within the backups, and I can walk this storage for all discrete backups.
If I want to just restore a container, I will download that “partition” and transfer it to the docker VM. Down the container stack in question, blow out everything in that folder and then restore the contents of the download to the container folder. Start up the docker stack for that folder and it’s back to where it was. Alternatively, I could just restore individual files if I wanted.
You should have all your data separately stored, it shouldn’t be locked inside containers, and using a VM hosted on a device to serve the data is a little convoluted
I personally don’t like TrueNAS - I’m not a hater, it just doesn’t float my boat (but I suspect someone will rage-downvote me 😉)
So, as an alternative approach, have a look at OpenMediaVault
It’s basically a Debian based NAS designed for DIY systems, which serves the local drives but it also has docker on, so feels like it might be a better fit for you.
I tried OMV in the past, but I found TrueNAS to be more intuitive… but that’s just personal preference I guess, and I’m not opposed to using OMV. Are you suggesting, then, that I run OMV on bare metal, and use it for everything? Or should it be inside a VM? If it’s the former, how easy is it to setup docker, because I’m not that familiar with OMV (it’s been a long time since I last checked it out). Is it like installing it in Debian directly? How does it handle the storage?
I always prefer bare metal for the core NAS functionality. There’s no benefit in adding a hypervisor layer just to create an NFS / SMB / iSCSI share
OMV comes with it’s own bare metal installer, based on Debian, so it’s as stable as a rock.
If you’ve used it before, you’re probably aware that it needs it’s own drive to install on, then everything else is the bulk storage pool… I’ve used various USB / mSATA / M.2 drives over the years and found it’s a really good way to segregate things.
I stopped using OMV when - IMO - “core” functions I was using (ie syncthing) became containers, because I have no use for that level of abstraction (but it’s less work for the OMV dev to maintain addons, so fair enough)
So, you don’t have to install docker, OMV automatically handles it for you.
How much OMV’s moved on, I don’t know, but I thought it would simplify your setup.
Okay, thank you, that’s good to know. However, I don’t have two separate devices that I can use to separate the NAS functionality from the Docker functionality, that’s why I was using Proxmox in the first place. And, I’m not sure how well Docker can run in OMV. But I’ll still keep it in mind as an option, thank you!
I think you’ve misunderstood
Ok, OMV needs a separate (small) boot drive to install on (ie consider a M.2 / SSD on a USB adapter)
But, then all your (large) storage is used for the NAS.
OMV will run Docker containers, but their data would also be pointed to the large NAS storage.
| Small | Large | |--------+-----------| | OMV | Your Files| | Docker | Data, etc |You’re right, I misunderstood. I understand now, thank you for replying with detail. I’m currently still not over my “I like Hypervisors now” phase. If I go back to bare metal, I will most probably use the setup you described. Still, thank you very much, and I’m keeping this thread for future reference.
Debian supports zfs so why the extra hassle of truenas? Seems to be a lot of extra work to add a vm when you could just use zfs in Debian. Or install something like Minio in Debian and use that to manage the data in s3 style buckets, again no vm needed
Okay, no VM, understood, but I do want to use a GUI for ZFS, because I’m basically a noob.
Late to the party, but if you really want a GUI for ZFS, 45drives has a ZFS plug-in for cockpit that works quite well.
Huh, I hadn’t understood that Cockpit is another server OS that I can use. Okay, I’ll keep it in mind, thanks!
It’s not. It’s just a GUI for a server.
It’s a modular GUI, though. The base install allows basic server management: system stats, upgrades, and a couple other bits. However, you can install additional modules to expand the GUI to allow for storage management, ZFS management, container management, file sharing (NFS, SMB), a file browser, VM management, user management, and so on.
My go-to “NAS system” is just standard Debian with Cockpit and a handful of supporting modules. It gives a nice GUI like OMV, but with all the flexibility and control of a normal Debian install.
Well erm that I can’t get around… but I guess it depends on what you want to self host. I use truenas as a storage solution, standalone and it works very well for that - I have had it up and running for years and years. It also stores personal files and is not exposed to the internet apart from updates and a few other things. Then I have an old 4-node server from Supermicro which hosts the stuff I need and uses truenas to store the data. Now that is probably overkill but the data is safe and backed up well. Truenas has apps for a lot of things, and other ways to host docker containers if you have the latest scale community edition. So perhaps you could do it that way
Yeah, but I don’t have two separate machines, that’s why I’m using Proxmox. And I used TrueNAS previously, with TrueCharts, and I wasn’t happy. It was pretty unstable and finnicky for me, and hard to go back to, after running docker in a clean Debian VM, which has been rocksolid so far. Still, thanks for your suggestion
Like I said - each to their own. truecharts the apps are pretty much known to be more unstable as well, says that everywhere in the documentation. Truenas if it’s up to date, using their app catalogue with stable selected is pretty rock solid for everyone.
You’re right, their own app catalogue is stable, but I like trying out new docker apps constantly, and I feel like I’m limitted in my options if I only use their catalogue. Still, thanks for your replies.
I get that - it is never going to replace a true docker setup. However I am pretty happy with what I can achieve on my personal truenas setup, and it is NOT exposed to the web which appears to be perfectly fine.
Also I have to say that the whole idea of truenas is moving towards enterprise even if you self-host it. They’d much rather you bought a whole system than on your cobbled together old stuff. That’s why it is now community edition, a simple way of saying you are gonna get bugs and things that don’t work
