I'm back to a three-node #Proxmox cluster.
-
@yehor Do you need shared storage? #ZFS replication works very well, even with gigabit Ethernet. It can be a little hard on SSDs but even decent consumer drives should be fine (enterprise drives with PLP are first prize though). It's the option I've gone for with my three node ThinkCentre cluster.
Another option is #MooseFS, which I have heard works well, although you do need to install and manage it outside Proxmox. https://moosefs.com/
-
@yehor regarding "my network is too slow for ceph"...
My OSD is a 2TB USB Stick and my Network is 1Gbit. It takes nearly a day to rebuild. It saved my ass more than one time. Extend and migrate Storage was never that easy. Ask me in a week how the upgrade to Proxmox 9 was going.
-
@yehor Do you need shared storage? #ZFS replication works very well, even with gigabit Ethernet. It can be a little hard on SSDs but even decent consumer drives should be fine (enterprise drives with PLP are first prize though). It's the option I've gone for with my three node ThinkCentre cluster.
Another option is #MooseFS, which I have heard works well, although you do need to install and manage it outside Proxmox. https://moosefs.com/
@mjturner Well, I'm currently using replication, but my concern is that when the node fails, migrated guets could be as old as the replication period. For example, I'm replicating the Matrix instance every 15 minutes. And if the node with the Matrix server fails, HA will migrate the last replicated copy of it. And for the chat server, going back 15 minutes could not be good.
-
@yehor Do you need shared storage? #ZFS replication works very well, even with gigabit Ethernet. It can be a little hard on SSDs but even decent consumer drives should be fine (enterprise drives with PLP are first prize though). It's the option I've gone for with my three node ThinkCentre cluster.
Another option is #MooseFS, which I have heard works well, although you do need to install and manage it outside Proxmox. https://moosefs.com/
@mjturner MoosFS is interesting, thanks!
-
@mjturner Well, I'm currently using replication, but my concern is that when the node fails, migrated guets could be as old as the replication period. For example, I'm replicating the Matrix instance every 15 minutes. And if the node with the Matrix server fails, HA will migrate the last replicated copy of it. And for the chat server, going back 15 minutes could not be good.
@yehor Yes, that is always a concern. I'm still tweaking my setup, but I've seen that some people use very high replication frequencies (eg, 1 minute, even 30s). But even 1 minute lost would be bad for Matrix.
That said, does Matrix support application-level HA? For example, could you have two instances and fail over from one to the other? In some cases, if the application supports HA that's a better option that doing it via the hypervisor.
-
@yehor I use S3 solution in most cases if supported by service
-
@yehor I use S3 solution in most cases if supported by service
@alex it’s not for service itself. It’s for virtual guest machines inside Proxmox cluster for seamless migrations and high availability. S3 is not suitable for this.
-
@yehor
How slow is your network? -
-
@yehor
How slow is your network?@FritzAdalis 1 Gbps
-
@yehor
That's not all that slow, you may find it's plenty fast enough for disk I/O. If you have spinning disks you're going to need a lot of them to saturate a gig link. -
R relay@relay.infosec.exchange shared this topic
