Today's accomplishment: I have an analysis pipeline with a few parallelizable steps. I typically run my analysis in parallel on my desktop, but I've both been needing more oomph and have been running out of drive space nonstop.
My #homeserver (40 cores/80 threads with quad E4870s) is running ZFS on a RAID0 array, currently just with two 10k SAS drives. By mounting this as an NFS share on my desktop, I can put all my generated simulation data directly on that larger filesystem.
Additionally, my server is running #proxmox, and has a "science" container. I can also mount that NFS share in the science container, giving it direct access to my simulation data, and run it as a #Ray cluster head node. Now, I can do simulations on my desktop, run the single-core tasks on that machine, and farm out the parallel analysis to workers on the server, putting all 80 cores to work!
