Lumiera
The new emerging NLE for GNU/Linux
State Draft
Date Di 11 Jan 2011 17:00:55 CET
Proposed by Ichthyostega <prg@ichthyostega.de>

Lumiera should be a freely relocatable application bundle. Relying only on the relative folder structure within this bundle, the application will be fully functional at any location, provided that the external library dependencies are resolvable using the standard mechanisms of the platform. The setup mechanism must be obvious, self-explanatory and must not rely on compiled in magic or buildsystem trickery. Yet packaging into a FSH conforming installation location should be supported by the same mechanisms.

Description

Lumiera is planned to become a large professional application bundle, relying on several external resources for proper operation. An installed Lumiera application will be more like Gimp, Blender, OpenOffice or Eclipse, not like bash, autotools or emcas.

Besides that, it can be expected that Lumiera frequently will be used in a project or studio like setup, where the application isn’t installed, but just unZIPped / unTARed and used as-is. Thus, it should be sufficient to unpack the application bundle and point it to the session file and maybe the media storage.

The Eclipse platform can serve as a model for the setup of an modern application of that style: It can be just unpacked, and when looking into the folder structure, the meaning of the parts is obvious, and the basic bootstrap is controlled by two short text based INI files. While Lumiera presumably won’t get that heavyweight and is clearly not intended to become a general business application platform like OSGi — the underlying principles can serve as a point of reference for modern development standards.

This leads to the following conclusions:

  • we need to define a standard folder layout for the bundle

  • the application must not rely on any compiled-in absolute paths

  • the executable should fetch the directly accompanying shared modules

  • all other lib dependencies should be handled by the system mechanisms

  • the bootstrap configuration likewise must not be compiled-in

  • this configuration must be human readable and clearly exhibit its purpose

  • the same system must be able to adapt to a FSH conforming installation layout

Judging from our current planning and the existing codebase, Lumiera is on a good way in that direction, yet some cleanup needs to be done, especially removing convenience shortcuts from the early days of development and catching up with the repair of some traits of sloppyness here and there.

Library resolution

In former days, it was common habit to compile-in a hard wired absolute RPATH. This can be considered obsolete practice; for example, the Debian policy forbids doing so. This is the result from numerous maintainability problems in the past. On the other hand, the GNU linker and other modern linkers support a relative resolution of shared modules directly accompanying an specific executable. The Debian policy allows this, if and only if these shared modules are installed with the same binary package and only used by this specific executable(s). Together, this is exactly what we need to solve our requirement.

Thus, the build process enables the new-style DT-tags in the Elf binary and sets the DT_RUNPATH with an value relative to $ORIGIN, which resolves to the path of the currently executing binary. Moreover, it is sufficient to set this on the initial executable only, because this creates a common searchpath for all lib resolution events in the scope of that loaded executable. Besides that, we need to care that our private libraries have a unique SONAME, in this case all starting with the prefix liblumiera*. Note moreover that this new-style DT_RUNPATH indeed can be overridden by an LD_LIBRARY_PATH in the environment, should there be the need for very special experiments.

Bootstrap location

Thus, a single relative library folder becomes the only hard wired start configuration. In our case, the folder $ORIGIN/modules was chosen. The root of the package then holds all the binaries depending on these common internal libraries, that is the lumiera executable and any accompanying special tools. As usual with such large application bundles, these get only symlinked into the /usr/bin folder on installation.

For sake of clarity, after starting the executable, the same location is used to load the bootstrap configuration. This configuration in turn defines all further locations like the extended configuration, project templates, plugin search path, the GUI module to load, the search path for icons and GUI resources, project templates and similar basics.

Relative paths and the location of the executable

According to the general requirements outlined above, most paths should be given in a relative fashion. Unfortunately there is no portable solution for self-discovering the currently running executable. But at least there is a solution for all current major platforms. Under Linux, this information can be retrieved from the kernel through the /proc filesystem.

Again for sake of clarity, the same token $ORIGIN was chosen to denote this executable location (note: this is not the current working directory). Moreover, due to the folder layout detailed above, this coincides with the root of the application bundle, thus making for a self-explanatory convention. Besides $ORIGIN, these search paths later on likely will contain locations below the user’s home directory, e.g. ~/.lumiera/themes

Tasks

  • identify what impedes such a modern setup procedure (✔ done)

  • rectify the folder structure created in the build target directory (✔ done)

  • build the executables in a way to allow relative resolution of the internal shared modules (✔ done)

  • replace the compiled-in path definitions for plugin loading by a configurable bootstrap (✔)

  • add an working library implementation for a config loader (✔ done)

  • add a mechanism for establishing the path of the current execubable.
    This is non-portable (✔ done)

  • wire the prepared API in the GUI to use this working config loader for resolving GUI resources (✔ done)

  • try to extract the path search code from the existing config loader, or build a new solution based on standard libraries (✔ done)

  • introduce an output root directory into the buildsystem, allowing for package builds (✔)

  • define a Debian packaging as proof-of-concept (✔ done)

Discussion

Pros

  • self-contained

  • self-explanatory

  • based on best practices

  • conforming with FSH and Debian policy

Cons

  • requires work

  • raises the bar at the implementation side

  • requires an bootstrap sequence to be explicitly performed on application startup

  • breaks with some beloved habits of the Unix community

Alternatives

I can think of two alternatives

  1. dealing with all those problems later

  2. not making an concept, rather sticking to UNIX habits

The first alternative is indeed worth considering, because we’re settling some things to be really implemented way later, which bears some dangers. But, on the other hand, it is a common practice known from extreme programming to deliver early and regularly, which effectively means to set up the deploy path of an application really early in the development cycle. The rationale is that — according to general experience — the deployment always turns up some very specific problems and constraints, which can be a serious threat when discovered late in the development process.

The second alternative isn’t really applicable IMHO. The original UNIX philosophy breeds on an academic setup and really excels with small nifty commandline utils combined by pipes, each specialised to do a single thing very well. These utils are more like the objects within our implementation. The concept of large application software bundles and desktop software was always a bit alien within the classic UNIX environment.

Rationale

This RfC can be seen as an commitment to an professional approach and as clarification: Traditionally, the Unix community hailed a lot of black magic practices like compiled-in installation paths, macro magic, sed and awk trickery, inline code compiled on-the-fly, relying on very specific and un-obvious behaviour of some build script, configuration via environment variables and a lot of similar idioms. These practices might be adequate in a quickly moving Research & Development setup, but turned out to be not so helpful when it comes to industrial strength development, as they are known to lead to maintenance problems.

Comments

State → Draft

There is now a complete implementation of this concept on my “proc” branch.
Moreover, I was able to define an initial Debian packaging for Lumiera on top of that implementation.

During that work, I had opportunity to visit various areas of the existing codebase, which reminded me of several small issues, which seem to become unhealthy when lying around unfixed for such a long time. Probably I’ll start a clean-up initiative and try to bring these points to discussion separately.

So 13 Feb 2011 20:04:00 CET Ichthyostega <prg@ichthyostega.de>