Paul Ganssle @pganssle@qoto.org

I think this actually boils down to this bug: https://trac.ffmpeg.org/ticket/7037

If I had more time to play with OSS and more experience in this domain I might jump on it. Probably in the meantime I'll try to cobble together a script to mostly-automatically carry over the HDR metadata in transcodes.

Resurrecting this because I finally got around to trying it: unfortunately, this didn't work well for me. I get a lot more opportunity to tweak the color palette, but I haven't gotten one that turns out quite right yet, which makes me think this approach would need about as much tweaking as manually figuring out the parameters to copy over HDR metadata.

Additionally, Tone-mapping requires using `zscale` to make the inputs linear, which requires compiling with `--enable-libzimg`, which Arch linux and most docker containers I've found don't do. I managed to create a docker container that installs `ffmpeg-full` from the AUR, but that was a whole complicated affair, unfortunately, and takes forever.

@benoit Plot twist: HR is full of tax protesters, and they send you a bunch of literature on how you never needed to pay taxes in the first place.

https://en.wikipedia.org/wiki/Tax_protester

@timorl If it's an open instance you could also make an account there and try following someone here.

@freemo I mean, the crowd did kill two of those people... One deliberately, the other accidentally.

@freemo Looks like one of those deaths was trampled by other rioters, one had a heart attack and another had a stroke.

Hard to blame the police for any deaths except the woman who was shot.

@freemo Yeah, should have been multiprocessing.RLock or whatever specific re-entrant lock you are using. I probably shouldn't have even namespaced it 😛

@freemo Hmm, weird. If there is no solution that allows you to have more than 500 RLocks, maybe you can get a larger array by locking "blocks" of the array for your atomic operations. Then you can have 500 locks spread across 20,000 elements.

Python is always pass by reference, so you can do something like this:

locks = [threading.RLock() for _ in range(500)]
locks = locks * int(math.ceil(len(array) / 500))
random.shuffle(locks)
locks = locks[:len(array)]

Each element is randomly assigned a block of locks, so even if the access pattern is non-random the lock distribution will be. At any given time you have 32 processes contending for 500 locks, which seems like good odds.

@freemo Though TBH I'd kinda love to have a problem that admits an embarrassingly parallel solution as an excuse to write something significant in Rust to test out that "fearless concurrency".

Closest I've come is this: https://gitlab.com/pganssle/metadata-backup

There's a big queue that theoretically could be read in parallel, but I think the file system access ends up blocking, because adding multithreading into the mix doesn't seem to have meaningfully sped anything up.

@freemo Weird. If you are doing so much in parallel and it's a big part of your operation (and you think it's worth it to explore this further) it might make sense to try out using Cython or Numba with a function that releases the GIL, then use multithreading instead of multiprocessing.

Running hundreds of processes and serializing / serializing your data probably creates a ton of overhead.

@freemo And yeah, I am far from thinking Python has the best, most ergonomic parallelism story, but I do think Python scales better than many people give it credit for. 🙂

@freemo Nice! Glad to hear it!

@yuvipanda@mastodon.technology Oh, didn't realize he's on mastodon.technology, so he's never going to see my response (exactly the reason I left that instance).

@yuvipanda@mastodon.technology I pull the pypa manylinux docker image from quay.io, I think that is the official location.

Not anything especially performance sensitive, but I haven't noticed a problem yet.

@freemo I don't know of any offhand, sorry. Julien Danjou has an article on it that you've probably seen: https://julien.danjou.info/atomic-lock-free-counters-in-python/

I'm mildly surprised I don't have an easy answer for this honestly. Seems like something that should at least be in toolz or boltons or something.

I admittedly don't do a ton of concurrency stuff that would need this (in Python), though, so there may be something obvious I've missed.

@freemo I'm not sure the GIL-ectomy had come to fruition in time for the deadline, and the 2→3 migration was actually not so bad compared to how it could have been if Python 3 had *also* made extensive changes to the C API.

From what I can tell, the 2 to 3 migration nearly killed Python as a language, and might well have actually killed it if it had been any worse. Hard to Monday morning quarterback on this.