Packaging Python

Opining on the standard distribution methods of interpreted programing languages

In building a linux distro from scratch, one of the things I'm avoiding as much as possible is scripting languages. Tools are written in C or C++ near exclusively for both performance and relocatability. Scripting languages require a shebang line, starting with #!, to give the interpreter, which is given as either an absolute path, or as an executable found in the path through the use of the standard /usr/bin/env utility. Not having a /usr directory at all discourages using env to find the script interpreter, and the isolated application packaging I'm pushing for makes discourages hardcoding absolute paths to the same interpreter that would be bundled with the application rather than set as an expected part of the system.

The regular distribution of python includes both dynamic binary modules in lib-dynlib, as well as a bunch of python scripts that comprise the main libraries. You wouldn't be expected to modify those unless you're a developer, and your core installation isn't the place you'd imagine as the right place to be doing python development. These make up almost 2500 files throughout ${LIBDIR}/python${version}, including both source and bytecode-compiled versions of python libraries. As part of the regular python library path ther is ${LIBDIR}/python${version}.zip, which is a compressed version of the python standard libraries as a single file. The dynamic libraries can also be built into the libpython${version}.so to avoid extra dynamic modules, and save some space as well. After careful modification of Modules/Setup.local and a set of post-build commands, this is reduced to 2 files: ${LIBDIR}/python${version}/os.pyc and ${LIBDIR}/python${version}.zip. os.pyc is used to validate the python directory, and ${LIBDIR}/python${version}/lib-dynlib is kept as an empty directory to avoid similar warnings when running the python interpreter.

To generate the .pyc files from the .py files, you can run "python -m compileall -b ${DIR}", where the -b flag will compile the source files to be loadable directly, instead of existing in a __pyshared__ folder versioned by the python that compiled them. From here, the source is no longer necessary and can be discarded in the installed system.

In addition to this, python applications can themselves be installed as zip files. You can zip up a python program with all of its dependencies, with the entry point as __main__.pyc. With a shebang, it'll load itself and run as a single file. The single important python program necessary for building a lot of programs is meson, which I've packaged this way and performs as well as it does packaged in the standard manner while also not polluting the filesystem with a bunch of extra source files that won't be modified or even looked at by an end user.