Okay, here's webvac, which is of very limited use: https://git.freespeechextremist.com/gitweb/?p=webvac;a=blob_plain;f=README;hb=31f6b1e9fca63584c466e646bbf0af7faf855fc2

webvac serves static files using venti. It solves the "big directory" problem Pleroma has if you have a lot of uploads, it serves HEAD requests *faster* than static files on-disk, it automatically compresses/dedups, and you can use the same venti server for as many instances as you have, so you get cross-instance dedup. It has been in production on FSE for a few days, serving the hell out of files, and (given that serving uploads is an I/O-bound process) no latency increase for the most part. Almost all of this is thanks to venti, which is great software.

FSE uses venti for its backups as well, by running vac against the media directory, then using venti/copy to replicate this to a venti server offsite. After the initial backup of the full 100GB (which took most of a day), replication (three times a day) with full history takes under ten minutes, usually.

You will need: venti (for storing the blocks; a Plan 9 venti server or a P9P one running on Linux/BSD/whatever works fine), redis (for storing the map of filenames to blocks), vac/unvac from P9P ( https://github.com/9fans/plan9port , also available over IPFS: QmW7ytEymcMpw1KAsqQh73gTiP5iCQFNZeD1DxRVgMpFDA), Ruby, and the ability to tweak nginx without getting an aneurysm from the rage.

There are two main pieces: webvac-server (which serves the files) and webvac-sweep (which sweeps files into venti). webvac-sweep pulls a file into venti and then records the pathname, score, and metadata to redis. If all of that is successful, it can (optionally) delete the file. The file is now ready to be served! What FSE does is sweep everything and then remove files smaller than 4MB (after 4MB it takes about a second to retrieve the file from venti, I plan to fix the source of this problem and then serve *everything* from venti). Then we have nginx check if the file is present in the FS and if not, forward the request to webvac-server. (There is an example nginx config file for testing.)

You can browse the tree at https://git.freespeechextremist.com/gitweb/?p=webvac . You can clone it using:

$ git clone git://git.freespeechextremist.com/webvac

Or in a completely decentralized manner by grabbing a checkout from IPFS:

$ ipfs get -o webvac QmXhsk7s87NSh5fQmhwK3HNizt69iWUDn2bAYASEhpjMAF

There are installation instructions in the README at the top of the source tree and linked above.

There are more announcements coming today.

@p You made claims this was faster. Do you have benchmarks between this and the oldway to see?

@freemo I eyeballed it locally and I ran awk against the server logs. The uploads dir for FSE was 9.6MB, so just a full readdir was a 9.6MB pointer-chase across non-consecutive 4kB blocks of the disk, this is a lookup in a hash table in memory. The worst-case scenario for that was obscene, it was like ten seconds to do an 'ls' when the cache was cold. Across the board, HEAD requests are now pretty uniform and stay under 1ms of backend time.

If you need something more scientific than that, then feel free to not believe me. I'm already hacking on the next thing.

@p
i do t need anything. A proper benchmark woukd be nice to back uo your claim, but if its just a guestimate then it is what it is. I just wanted to know where the xlaim came from and if it had any weight.

@freemo

> A proper benchmark woukd be nice to back uo your claim

If I managed to make a hash table lookup in RAM slower than a pointer chase across the disk, I would publish that. This is just a property of the data structures and their respective storage media. It's an in-memory hash table versus an on-disk linked list, that'd be like devising and running a benchmark to determine that a search index is faster than a full scan of the disk. The only reason I bothered to eyeball it was to make sure I hadn't completely botched it by doing something stupid.

> I just wanted to know where the xlaim came from and if it had any weight.

Okay, the FS's block size is 4kB. dirents are not contiguous because they are appended piecemeal, so between two blocks that represent a directory, you will find thousands of blocks representing the files that have been written to disk in the interim. You can't predict it because it's a linked list. (Every clever thought you're having right now about putting an index in has to account for the physical medium and the failure modes and space overhead and performance. If you have any intuition about this, it is almost certainly tuned for RAM rather than disk, and most of the clever ideas have been tried already, and they did not result in a reliable, performant filesystem.) Here is a diagram. A pointer-chase across *random-access* memory is already bad, but even on SATA or NVMe disks, a seek is costlier.

Here is a hasty diagram, this is CS 101 stuff, dude.
filesystem_pointer_chase.png

@p

Ya know what is also CS 101, writing unit tests and benchmarks to go along side code that is written, even when it is known to be an improvement... Why? Simple, it helps us track the performance improvement and also help us tweak future modifications to the code and know when we make mistakes other than what we intended.

No one is saying your intent isnt justified, this is just how you write good code, that includes good tests not sure to prove out your current code, but more importantly as a measure for future tweaks to the code.

Good job and thanks for the hard work.

The thing is, optimisation is a tricky beast.

> that'd be like devising and running a benchmark to determine that a search index is faster than a full scan of the disk

I have seen many times where search indexes can be slower than full scans, just as sometimes hash tables can be slower than linked lists under the right circumstances.

I am not saying that applies here, I am not saying you are under any obligation to do a benchmark, I am not saying this is bad work in anyway.

All I'm saying is a benchmark would have been interesting and I dont rule out the possibility that under certain conditions it might show a slow down and in others a speedup, either of those might be marginal, and it would be interesting to see where the tradeover occurs and just how much of an improvement you get as various conditions grow.

Again not saying you need to do this to determine it was a good move. Just saying it would have been interesting to see, and the benchmarks in general useful for future diagnostics.

On the projects I run I like to create along side my unit tests extensive benchmarking. As features or fixes are added we watch the benchmarks change along side it, and it provides a similar CI tool as unit tests might. So I generally find it a worthwhile effort even if it may not be critical in knowing that the current feature set makes sense performance wise.

@freemo @p I think you both are over-estimating the level of content taught in CS 101.

@tmy

LOL true that. I work on some really advanced projects that most even advanced CS guys dont seem to be able to understand or touch (though they do use the libraries)...

I think ive gotten to the point where I dont realize how the vast majority of CS professionals really never even learned the basics, let alone the advanced stuff.

Frankly I doubt many CS noobs would understand what P is doing, let along the need for CI that includes both benchmarks and tests.

@p

@freemo @p I probably fall in that category of CS noob and yeah, it's way over my head. Something about having things make more efficient use of the hard-drive's block size.

That being said, CS 101 is dumbed-down trash that students should have the option to test out of as a required pre-req.

@tmy @freemo

> it's way over my head.

Nah, nah, trust me. Picture a linked list that you have written to disk. Then imagine that hand-wave properties of disks mean that you will want each node in the list to be 4kB or 8kB or 16kB or some arbitrary size. If it were a flat, contiguous array, then to find the Nth item, you would just seek N blocks ahead and read that block. But because it's a linked list, you have to start at the head, then read that, extract the pointer to the next node, seek there, read that, etc. That's how to make reading 9.6MB of disk took 10 seconds.

@p @tmy @freemo the solution is superior SSDs

@xair

Have you considered a 30 drive RAID0 made of SSDs ?

@tmy @p