@pkhuong I have so little experience with S3 style object stores but maybe you could answer this. Is there a reasonable way to implement a WAL on top of S3 objects if you want durability for smallish transactions? It seems like a poor fit since you don't have append semantics. You could encode a linear log by writing a new root object that contains the new log entries plus a pointer to the previous root object. But each object would be tiny, overhead massive.

@pkhuong In a high transaction rate system you can do group commit (S3 latency is probably high enough that any additional latency from group commit isn't a big issue) to combine multiple transactions into one new WAL root. Maybe that's good enough.

@pervognsen @pkhuong
Have you looked into multipart uploads?

@wolf480pl @pkhuong Thanks. I'd skimmed the docs page but it seemed to be targeting a different use case. For WAL-based durability you could use a part per commit then upon crash recovery you would complete the partial multipart upload to make the earlier parts accessible? You'd need the upload ID to recover but since that's reusable for many different commits you could register that in some overall metadata root that's only updated infrequently.

@pervognsen @wolf480pl @pkhuong Unsolicited comment; I'd feel a bit queasy about building something of that fine granularity on top of S3 (there's also some minimum size requirement for intermediate parts IIRC), but DynamoDB would seem like a fair fit if performance/parallelism/no-servers-to-manage mattered (good performance and there are some basic transactional features).