@pkhuong I have so little experience with S3 style object stores but maybe you could answer this. Is there a reasonable way to implement a WAL on top of S3 objects if you want durability for smallish transactions? It seems like a poor fit since you don't have append semantics. You could encode a linear log by writing a new root object that contains the new log entries plus a pointer to the previous root object. But each object would be tiny, overhead massive.
@pkhuong In a high transaction rate system you can do group commit (S3 latency is probably high enough that any additional latency from group commit isn't a big issue) to combine multiple transactions into one new WAL root. Maybe that's good enough.
@pervognsen @pkhuong
Have you looked into multipart uploads?
@wolf480pl @pkhuong Thanks. I'd skimmed the docs page but it seemed to be targeting a different use case. For WAL-based durability you could use a part per commit then upon crash recovery you would complete the partial multipart upload to make the earlier parts accessible? You'd need the upload ID to recover but since that's reusable for many different commits you could register that in some overall metadata root that's only updated infrequently.
@pervognsen @wolf480pl @pkhuong Unsolicited comment; I'd feel a bit queasy about building something of that fine granularity on top of S3 (there's also some minimum size requirement for intermediate parts IIRC), but DynamoDB would seem like a fair fit if performance/parallelism/no-servers-to-manage mattered (good performance and there are some basic transactional features).
@pervognsen @christiankothe @pkhuong I think one way you'd do it with eg. Postgres is to keep the most recent WAL segment locally on DB's disk, and upload to S3 from the WAL archiving hook once the segment is complete.
That lets you recover from crashes.
But in case of losing the local disk... yeah you still lose transactions you said were committed. In most usecases people don't care. Idk what they do in scenarios where it's imoortant tho.
@pervognsen @wolf480pl @pkhuong Got it, makes sense (FWIW, EFS might also be an interesting option for a cloud-backed fully managed file system, though mounting that from outside the cloud might take some config work)
@pervognsen @christiankothe @wolf480pl @pkhuong one interesting note is that Amazon themselves didn't solve this for CloudWatch, and instead have one file per thing. Not sure what it implies, but though I mention it.
@christiankothe @wolf480pl @pkhuong This is for my own thing which uses a local disk-based WAL. I was mostly wondering if there was a reasonable way to implement an S3 adapter for the WAL storage.