It turns out that Python pickles don't care about excess data at the end of the file, and it is well-known that ZIP files can be cleanly appended to other data. I have used this knowledge to create a system that stores pickled metadata about Python zipimport packages as a preamble to the ZIP file.

I am now gradually building an overly-complicated runtime environment based on this concept. The current problem is how to build the bundle that builds bundles.

Part of what I'm building here is a dynamic linker for Python, despite the fact that Python already has a good dynamic linker.

@alexbuzzbee @alexbuzzbee Ah. I tend to use the preamble to a ZIP file to store a Python script to import itself using zipimporter, so I can have a single-file .pyw with an if __name__ == "__main__": and Windows double-clickability. But that pickling idea is good!

You know about sys.path_hooks, right?

@wizzwizz4 I know that I could do something that uses the Python interpreter's existing tools for doing this kind of stuff, but instead I am building an entire second layer of (very unnecessary) import functionality. All the benefits of the system I'm building could be gotten more easily by using Python as intended and adding a few small modules that tie into importlib and sys, but I'm doing this for the over-engineering, not for the practical benefits.

@wizzwizz4 I'm sure there are plenty of reasonable things to do using the pickle+ZIP combination, but this is not one of them.

@alexbuzzbee @wizzwizz4 Were you aware of https://pyoxidizer.readthedocs.io/ before you decided to start building this?

(It's a bundler similar in concept to tools like py2exe, but with different architectural goals, and it does support embedding data.)

I ask because, even if you're not trying to put *everything* in a single file, it could have something useful to learn from.

(And, being a bundler, it sort of goes in the opposite direction of well-known Python plugin frameworks like YAPSY.)

@ssokolow @wizzwizz4 My project is kind of a combination of a multi-application host, a plugin system, a bundler, and some tools for doing everything with asyncio.

It's very weird and scatter-brained, but the basic idea originated as "what if there was an operating system written in Python" and then took too much inspiration from MS .NET and developed from there without actually using most of the tools that are built into Python.