Very nice! I didn't know about zstd. Trying it out:
Comparison: tar.gz 1.9G 11 min, tar.xz 1.2G 88 min
33 min, 1.3G result, -19 and all cores:
tar -c -I 'zstd -19 -T0' -f cleaned_files.tar.zst cleaned_files/
2 min, 1.7G result, -11 and all cores:
tar -c -I 'zstd -11 -T0' -f cleaned_files11.tar.zst cleaned_files/
Result: -11 compression on the ZSTD was smaller than the gzip compression and also more than 5 times faster. -19 compression was almost as small as the xz compression, and nearly 3x faster.
QT: https://qoto.org/@worldsendless/108522525599439326