Welcome to TiddlyWiki created by Jeremy Ruston, Copyright © 2007 UnaMesa Association
PageTemplate
|>|SiteTitle - SiteSubtitle|
|>|MainMenu|
|DefaultTiddlers<<br>><<br>><<br>>ViewTemplate<<br>><<br>>EditTemplate|SideBarOptions|
|~|OptionsPanel|
|~|SideBarTabs|
|~|AdvancedOptions|
|~|<<tiddler Configuration.SideBarTabs>>|
''StyleSheet:'' StyleSheetColors - StyleSheetLayout - StyleSheetPrint
ColorPalette
SiteUrl
//pattern of collaboration for loosely coupled entities, to be used for various purposes within Proc...//
Expecting Advice and giving Advice — this collaboration ranges somewhere between messaging and dynamic properties, but cross-cutting the primary, often hierarchical relation of dependencies. Always happening at a certain //point of advice,// which creates a distinct, static nature different of being just a convention, on the other hand, Advice is deliberately kept optional and received synchronously, albeit possibly within an continuation.
!Specification
''Definition'': Advice is an optional, mediated collaboration between entities taking on the roles of advisor and advised, thereby passing a custom piece of advice data, managed by the advice support system. The possibility of advice is created by both of the collaborators entering the system, where the advised entity exposes a point of advice, while the advising entity provides an actual advice value.
[>img[Entities for Advice collaboration|uml/fig141445.png]]
!!Collaborators
* the ''advised'' entity
* the ''advisor''
* ''point of advice''
* ''advice system''
* the ''binding''
* the ''advice''
Usually, the ''advised'' entity opens the collaboration by requesting an advice. The ''advice'' itself is a piece of data of a custom type, which needs to be //copyable.// Obviously, both the advised and the advisor need to share knowledge about the meaning of this advice data. (in a more elaborate version we might allow the advisor to provide a subclass of the advice interface type). The actual advice collaboration happens at a ''point of advice'', which needs to be derived first. To this end, two prerequisites are to be fulfilled (without fixed sequence): The advised puts up an ''advice request'' by specifying his ''binding'', which is a pattern for matching. An entity about to give advice attaches a possible ''advice provision'', combined with an advisor binding, which similarly is a pattern. The ''advice system'' as mediator resolves both sides, by matching (which in the most general case could be an unification). This process creates an ''advice point solution'' — allowing the advisor to fed the piece of advice into a kind of communication channel (»advice channel«), causing the advice data to be placed into the point of advice. After passing a certain (implementation defined) break point, the advice leaves the influence of the advisor and gets exposed to the advised entities. Especially, this involves copying the advice data into a location managed by the advice system. In the standard case, the advised entity picks up the advice synchronously (and non-blocking). Of course, there could be a status flag to find out if there is new advice. Moreover, typically the advice data type is default constructible and thus there is always a basic form of advice available, thereby completely decoupling the advised entity from the timings related to this collaboration.
!!extensions
In a more elaborate scheme, the advised entity could provide a signal to be invoked either in the thread context of the advisor (being still blocked in the advice providing call), or in a completely separate thread. A third solution would be to allow the advised entity to block until receiving new advice. Both of these more elaborate schemes would also allow to create an advice queue — thereby developing the advice collaboration into a kind of messaging system. Following this route seems questionable though.
→ AdviceSituations
→ AdviceRequirements
→ AdviceImplementation
[<img[Advice solution|uml/fig141573.png]]
The advice system is //templated on the advice type// — so basically any collaboration is limited to a distinct advice type. But currently (as of 5/2010), this typed context is kept on the interface level, while the implementation is built on top of a single lookup table (which might create contention problems in the future and thus may be changed without further notice). The advice system is a system wide singleton service, but it is never addressed directly by the participants. Rather, instances of ~AdviceProvision and ~AdviceRequest act as point of access. But these aren't completely symmetric; while the ~AdviceRequest is owned by the advised entity, the ~AdviceProvision is a value object, a uniform holder used to introduce new advice into the system. ~AdviceProvision is copied into an internal buffer and managed by the advice system, as is the actual advice item, which is copied alongside.
In order to find matches and provide advice solutions, the advice system maintains an index data structure called ''~Binding-Index''. The actual binding predicates are represented by value objects stored within this index table. The matching process is triggered whenever a new possibility for an advice solution enters the system, which could be a new request, a new provision or a change in the specified bindings. A successful match causes a pointer to be set within the ~AdviceRequest, pointing to the ~AdviceProvision acting as solution. Thus, when a solution exists, the advised entity can access the advice value object by dereferencing this pointer. A new advice solution just results in setting a different pointer, which is atomic and doesn't need to be protected by locking. But note, omitting the locking means there is no memory barrier; thus the advised entity might not see any changed advice solution, until the corresponding thread(s) refresh their CPU cache. This might or might not be acceptable, depending on the context, and thus is configurable as policy. Similarly, the handling of default advice is configurable. Usually, advice is a default constructible value object. In this case, when there isn't any advice solution (yet), a pseudo solution holding the default constructed advice value is used to satisfy any advice access by the client (advised entity). The same can be used when the actual ~AdviceProvision gets //retracted.// As an alternative, when this default solution approach doesn't work, we can provide a policy either to throw or to wait blocking — but this alternative policy is similarly implemented with an //null object// (a placeholder ~AdviceProvision). Anyway, this implementation technique causes the advice system to collect some advice provisions, bindings and advice objects over time. It should use a pooling custom allocator in the final version. As the number of advisors is expected to be rather small, the storage occupied by these elements, which is effectively blocked until application exit, isn't considered a problem.
!organising the advice solution
This is the tricky part of the whole advice system implementation. A naive implementation will quickly degenerate in performance, as costs are of order ~AdviceProvisions * ~AdviceRequests * (average number of binding terms). But contrary to the standard solutions for rules based systems (either forward or backward chaining), in this case here always complete binding sets are to be matched, which allows to reduce the effort.
!!!solution mechanics
The binding patterns are organised by //predicate symbol and the lists are normalised.// A simple normalisation could be lexicographic ordering of the predicate symbols. Then the resulting representation can be //hashed.// When all predicates are constant, a match can be detected by hashtable lookup, otherwise, in case some of the predicates contain variable arguments ({{red{planned extension}}}), the lookup is followed by an unification. For this to work, we'll have to include the arity into the predicate symbols used in the first matching stage. Moreover, we'll create a //matching closure// (functor object), internally holding the arguments for unification. This approach allows for //actual interpretation of the arguments.// It is conceivable that in special cases we'll get multiple instances of the same predicate, just with different arguments. The unification of these terms needs to consider each possible pairwise combination (cartesian product) — but working out the details of the implementation can safely be deferred until we'll actually hit such a special situation, thanks to the implementation by a functor.
Fortunately, the calculation of this normalised patterns can be separated completely from the actual matching. Indeed, we don't even need to store the binding patterns at all within the binding index — storing the hash value is sufficient (and in case of patterns with arguments we'll attach the matching closure functor). Yet still we need to store a marker for each successful match, together with back-links, in order to handle changing and retracting of advice.
!!!storage and registrations
We have to provide dedicated storage for the actual advice provisions and for the index entries. Mostly, these objects to be managed are attached through a single link — and moreover the advice system is considered performance critical, so it doesn't make sense to implement the management of these entries by smart-ptr. This rules out ~TypedAllocationManager and prompts to write a dedicated storage frontend, later to be backed by Lumiera's mpool facility.
* both the advice provision and the advice requests attach to the advice system after fulfilling some prerequisites; they need to detach automatically on destruction.
* in case of the provision, there is a cascaded relation: the externally maintained provision creates an internal provision record, which in turn attaches an index entry.
* both in case of the provision and the request, the relation of the index bears some multiplicity:
** a multitude of advice provisions can attach with the same binding (which could be the same binding pattern terms, but different variable arguments). Each of them could create a separate advice solution (at least when variable arguments are involved); it would be desirable to establish a defined LIFO order for any search for possibly matching advice.
** a multitude of advice requests can attach with the same binding, and each of them needs to be visited in case a match is detected.
* in both cases, any of these entries could be removed any time on de-registration of the corresponding external entity
* we need to track existing advice solutions, because we need to be able to overwrite with new advice and to remove all solutions bound to a given pattern about to leave the system. One provision could create a large number of solutions, while each registration always holds onto exactly one solution (which could be a default/placeholder solution though)
!!!!subtle variations in semantics
While generally advice has value semantics and there is no ownership or distinguishable identity, the actual implementation technique creates the possibility for some subtle semantic variations. At the time of this writing (5/2010) no external point of reference was available to decide upon the correct implementation variant. These variations get visible when advice is //retracted.// Ideally, a new advisor would re-attach to an existing provision and supersede the contained advice information with new data. Thus, after a chain of such new provisions all attaching with the identical binding, when finally the advice gets retracted, any advice provisions would be gone and we'd fall back onto the default solution. Thus, "retracting" would mean to void any advice given with this binding.
But there is another conceivable variation of semantics, which yields some benefits implementation-wise: Advice can be provided as "I don't care what was said, but here is new information". In this case, the mechanism of resolving and finding a match would be responsible to pick the latest addition, while the provisions would just be dumped into the system. In this case, "retracting" would mean just to cancel //one specific// piece of information and might cause in an older advice solution to be uncovered and revived. The default (empty) solution would be used in this case only after retracting all advice provisions.
!!!!implementation variants with respect to attachment and memory management
Aside from the index, handling of the advice provisions turns out to be tricky.
* management by ref-count was ruled out due to contention and locality considerations
* the most straight forward implementation would be for the ~AdviceProvision within the advisor to keep kind of an "inofficial" link to "its" provision, allowing to modify and retract it during the lifetime of the advisor. When going away without retracting (the default behaviour), the provision, as added into the system would remain there as a dangling entry. It is still reachable via the index, but not maintained in any further way. If memory usage turns out to be a problem, we'd need to enqueue these entries for clean-up.
* but as this simple solution contradicts the general advice semantics in a subtle way (see previous paragraph), we could insist on really re-capturing and retracting previous advice automatically on each new advice provision or modification. In this case, due to the requirement of thread safety, each addition, binding modification, placing of new advice or retraction would require to do an index search to find an existing provision with equivalent binding (same binding definition, not just a matching binding pattern). As a later provision could stomp upon an existing provision without the original advisor noticing this, we can't use the internal references anymore; we really need to search each time and also need a global lock during the modification transaction.
* an attempt to reduce this considerable overhead would be to use an back-link from the provision as added to the system to the original source (the ~AdviceProvision owned by the advisor). On modification, this original source would be notified and thus detached. Of course this is tricky to implement correctly, and also requires locking.
The decision for the initial implementation is to use the first variant and just accept the slightly imprecise semantics.
When copying a Provision, the hidden link to existing advice data is //not shared.//
!!!!de-allocation of advice data
It is desirable that the dtors of each piece of advice data be called eventually. But ensuring this reliably is tricky, because advice
data may be of various types and is added to the system to remain available, even after the original {{{advice::Provision}}} went out of scope. Moreover, the implementation decision was //not// to employ a vtable for the advice collaborators and data holders, so we're bound to invoke the dtor with the correct specific type.
There are some special cases when de-allocation happens while the original provision is still alive (new advice, changed binding, retracting). But in any other case, responsibility for de-allocation has to be taken by the ~AdviceSystem, which unfortunately can't handle the specific type information. Thus the original provision needs to provide a deleter function, and there is no way to avoid storing a function pointer to this deleter within the ~AdviceSystem, together with the advice data holder.
It seems reasonable to create this deleter right away and not to share the link to advice data, when copying a provision, to keep responsibilities straight. {{red{Question: does this even work?? }}} to be verified: does the address of the advice data buffer really determine alone what is found as "existing" provision?
!!!lifecycle considerations
Behind the scenes, hidden within the {{{advice.cpp}}} implementation file, the ~AdviceSystem is maintained as singleton. According to a general lifecycle policy within Lumiera, no significant logic is allowed to execute in the shutdown phase of the application, once the {{{main()}}} has exited. Thus, any advice related operations might throw {{{error::Logic}}} after that point. The {{{~AdviceSystem()}}} also is a good place to free any buffers holding incorporated advice data, after having freed the index datastructure referring to these buffer storage, of course.
!!!!handling of default advice
Basically, the behaviour when requesting non-existing advice may be configured by policy. But the default policy is to return ref to a default constructed instance of the advice type in that case. Just the (implementation related) problem is that we return advice by {{{const&}}}, not by value, so we're bound to create and manage this piece of default advice during the lifetime of the ~AdviceSystem. The way the ~AdviceSystem is accessed (only through the frontend of {{{advice::Request}}} and {{{advice::Provision}}} objects, in conjunction with the desire to control this behaviour by policy, creates a tricky implementation situation.
* regarding the lifecycle (and also from the logical viewpoint) it would be desirable to handle this "default" or "no solution" case similar to accessing an existing solution. But unfortunately doing so would require a fully typed context; thus basically on inserting a new request, when returning from the index search without a dedicated solution, we'd need to fabricate a fallback solution to insert it into the provision index, while still holding the index lock. At that point it is not determined if we ever need that fallback solution. Alternatively we could consider to fabricate this fallback solution on first unsuccessful advice fetch. But this seems sill worse, as it turns an (possibly even lock free) ptr access into an index operation. Having a very cheap advice access seems like an asset.
* on the other hand, using some separate kind of singleton bundle just for these default advice data (e.g. a templated version of //Meyer's Singleton...//), the fallback solution can be settled independent from the ~AdviceSystem, right in the {{{advice.hpp}}} and using static memory, but the downside is now the fallback solution might be destroyed prior to shutdown of the ~AdviceSystem, as it lives in another compilation unit.
Thus the second approach looks favourable, but we should //note the fact that it is hard to secure this possible access to an already destroyed solution,// unless we decline using the advice feature after the end of {{{main()}}}. Such a policy seems to be reasonable anyway, as the current implementation also has difficulties to prevent accessing an already destroyed {{{advice::Provision}}}, being incorporated in the ~AdviceSystem, but accessed through a direct pointer in the {{{advice::Request}}}.
!!!locking and exception safety
The advice system is (hopefully) written such as not to be corrupted in case an exception is thrown. Adding new requests, setting advice data on a provision and any binding change might fail due to exhausted memory. The advice system remains operational in this case, but the usual reaction would be //subsystem shutdown,// because the Advice facility typically is used in a very low-level manner, assuming it //just works.// As far as I can see, the other mutation operations can't throw.
The individual operations on the interface objects are //deliberately not thread-safe.// The general assumption is that {{{advice::Request}}} and {{{advice::Provision}}} will be used in a safe environment and not be accessed or modified concurrently. An notable exception to this rule is accessing Advice: as this just includes checking and dereferentiating a pointer, it might be done concurrently. But note, //the advice system does nothing to ensure visibility of the solution within a separate thread.// If this thread still has the old pointer value in his local cache, it won't pick up the new solution. In case the old solution got retracted, this even might cause access to already released objects. You have been warned. So it's probably a good idea to ensure a read barrier happens somewhere in the enclosing usage context prior to picking up a possibly changed advice solution concurrently.
''Note'': the underlying operations on the embedded global {{{advice::Index}}} obviously need to be protected by locking the whole index table on each mutation, which also ensures a memory barrier and thus propagates changed solutions. While this settles the problem for the moment, we might be forced into a more fine grained locking due to contention prolems later on...
!!!index datastructure
It is clear by now that the implementation datastrucutre has to serve as a kind of //reference count.// Within this datastructure, any constructed advice solution needs to be reflected somehow, to prevent us from discarding an advice provision still accessible. Allowing lock-free access to the advice solution (planned feature) adds an special twist, because in this case we can't even tell for sure if an overwritten old solution is actually gone (or if its still referred from some thread's cached memeory). This could be addressed with an transactional approach (which might be good anyway) — but I tend to leave this special concern asside for now.
To start with, any advice matching and solution will //always happen within matching buckets of a hash based pattern organisation.// The binding index thus should build on two hashtables (one for the requests and one for the provisions), but using specifically crafted datastructures as buckets. The individual entries within these bucket structures in both cases will be comprised of a binding matcher (to determine if an match actually happens) and a back-link to the registered entitiy (provision or request). Given the special pattern of the advice solutions, existing solutions could be tracked within the entries at the request side.
* Advice provisions are expected to happen only in small numbers; they will be searched stack-like, starting from the newes provisions, until a match is found.
* Each advised entity basically creates an advice request, so there could be a larger number of request entries. In the typical search triggered from the provision side, each request entry will be visited and checked for match, which, if successful, causes a pointer to be set within the ~AdviceRequest object (located outside the realm of the advice system). While — obviously — multiple requests with similar binding match could be folded into a sub-list, we need actual timing measurements to determine the weight of these two calculation steps of matching and storing, which together comprise the handling of an advice solution.
The above considerations don't fully solve the question how to represent an computed solution within the index datastructure, candidates being to use the index within the provision list, or a direct pointer to the provision or even just to re-use the pointer stored into the ~AdviceRequest. My decision is to do the latter. Besides solutions found by matching, we need //fallback solutions// holding a default constructed piece of advice of the requested type. As these defaults aren't correlated at all to the involved bindings, but only to the advice type as such, it seems reasonable to keep them completely apart, like e.g. placing them into static memory managed by the ~AdviceProvision template instantiations.
!!!interactions to be served by the index
[>img[Advice solution|draw/AdviceBindingIndex1.png]]
;add request
:check existing provisions starting from top until match; use default solution in case no match is found; publish solution into the new request; finally attach the new request entry
;remove request
:just remove the request entry
;modify request
:handle as if newly added
;add provision
:push new provision entry on top; traverse all request entries and check for match with this new provision entry, publish new solution for each match
;retract provision
:remove the provision entry; traverse all request entries to find those using this provision as advice solution, treat these as if they where newly added requests
;modify provision
:add a new (copy of the) provision, followed by retracting the old one; actually these two traversals of all requests can be combined, thus treating a request which used the old provision but doesn't match the new one is treated like a new request
<<<
__Invariant__: each request has a valid soultion pointer set (maybe pointing to a default solution). Whenever such a solution points to a registered provision, there is a match between the index entries and this is the top-most possible match to any provision entry for this request entry
<<<
Clearly, retracting advice (and consequently also the modification) is expensive. After finishing these operations, the old/retracted provision can be discarded (or put asside in case of non-locking advice access). Other operations don't cause de-allocation, as provisions remain within the system, even if the original advising entity is gone.
From analysing a number of intended AdviceSituations, some requirements for an Advice collaboration and implementation can be extracted.
* the piece of advice is //not shared// between advisor and the advised entities; rather, it is copied into storage managed by the advice system
* the piece of advice can only be exposed {{{const}}}, as any created advice point solution might be shared
* the actual mode of advice needs to be configurable by policy — signals (callback functors) might be used on both sides transparently
* the client side (the advised entity) specifies initially, if a default answer is acceptable. If not, retrieving advice might block or fail
* on both sides, the collaboration is initiated specifying an advice binding, which is an conjunction of predicates, --optionally dynamic--^^no!^^
* there is a tension between matching performance and flexibility. The top level should be entirely static (advice type)
* the analysed usage situations provide no common denominator on the preferences regarding the match implementation.
* some cases require just a match out of a small number of tokens, while generally we might get even a double dispatch
* later, possible and partial solutions could be cached, similar to the rete algorithm. Dispatching a solution should work lock-free
* advice can be replaced by new advice, which causes all matching advice solutions to behave as being overwritten.
* when locking is left out, we can't give any guarantee as to when a given advice gets visible to the advised entity
* throughput doesn't seem to be an issue, but picking up existing advice should be as fast as possible
* we expect a small number of advisors collaborating with and a larger number of advised entities.
!!questions
;when does the advice collaboration actually happen?
:when there is both a client (advised) and a server (advisor) and their advice bindings match
;can there be multiple matches?
:within the system as a whole there can be multiple solutions
:but the individual partners never see more than one connection
:each point of advice has exactly one binding and can establish one advice channel
;but when an attempt is made to transfer more information?
:both sides don't behave symmetrically, and thus the consequences are different
:on the client side, advice is just //available.// When there is newer one, the previous advice is overwritten
:the server side doesn't //contain// advice — rather, it is placed into the system. After that, the advisor can go away
:thus, if an advisor places new advice into an existing advice provision, this effectively initiates a new collaboration
:if the new advice reaches the same destination, it overwrites; but it may as well reach a different destination this time
;can just one advice provision create multiplicity?
:yes, because of the matching process there could be multiple solutions. But neither the client nor the server is aware of that.
;can advice be changed?
:No. When inserted into the system, the advisor looses any direct connection to the piece of advice (it is copied)
:But an advisor can put up another piece of advice into the same advice provision, thereby effectively overwriting at the destination
;if advice is copied, what about ownership and identity?
:advice has //value semantics.// Thus it has no distinguishable identity beyond the binding used to attach it
:a provision does not "own" advice. It is a piece of information, and the latest information is what counts
;can the binding be modified dynamically?
:this is treated as if retracting the existing point of advice and opening a new one.
;what drives the matching?
:whenever a new point of advice is opened, search for a matching solution happens.
:thus, the actual collaboration can be initiated from both sides
:when a match happens, the corresponding advice point solution gets added into the system
;what about the lifetime of such a solution?
:it is tied to the //referral// — but there is an asymmetry between server and client
:referral is bound to the server sided / client sided point of advice being still in existence
:but the server sided point of advice is copied into the system, while the client sided is owned by the client
:thus, when an advisor goes away without explicitly //retracting// the advice, any actual solution remains valid
:on the client side there is an asymmetry: actually, a new advice request can be opened, with an exactly identical binding
:in this case, existing connections will be re-used. But any differences in the binding will require searching a new solution
;is the search for an advice point solution exhaustive?
:from the server side, when a new advice provision / binding is put up, //any// possible advice channel will be searched
:contrary to this, at the client side, the first match found wins and will establish an advice channel.
!decisions
After considering the implementation possibilities, some not completely determined requirements can be narrowed down.
* we //do// support the //retracting of advice.//
* there is always an implicit //default advice solution.//
* advice //is not an messaging system// — no advice queue
* signals (continuations) are acceptable as a extension to be provided later
* retracting advice means to retreat a specific solution. This might or might not uncover earlier solutions (undefined behaviour)
* we don't support any kind of dynamic re-evaluation of the binding match (this means not supporting the placement use case)
* the binding pattern is //interpreted strictly as a conjunction of logic predicates// — no partial match, but arguments are allowed
* we prepare for a later extension to //full unification of arguments,// and provide a way of accessing the created bindings as //advice parameters.//
Combining all these requirements and properties provides the foundation for the → AdviceImplementation
[[Advice]] is a pattern extracted from several otherwise unrelated constellations
!Proxy media in the engine
Without rebuilding the engine network, we need the ability to reconfigure some parts to adapt to low resolution place-holder media temporarily. The collaboration required to make this happen seems to ''cross-cut'' the normal processing logic. Indeed, the nature of the adjustments is highly context dependent — not every processing node needs to be adjusted. There is a dangerous interference with the ongoing render processes, prompting for the possibility to pick up this information synchronously.
* the addressing and delivery of the advice is based on a mix of static (type) and dynamic information
* it is concievable that the actual matching may even include a token present in the direct invocation context (but this possibility was ruled out by later decision)
* the attempt to recieve and pick up advice needs to be failsafe
* locking should be avoided by design
!Dependency injection for testing
While inversion of control is a guiding principle on all levels, the design of the Lumiera application deliberately stays just below the level of employing a dependency injection container. Instead, common services are accessible //by type// and the builder pattern is used more explicitly at places. Interestingly, the impact on writing unit tests was by far not so serious as one might expect, based on the usual reasoning of D.I. proponents. But there remain some situations, where sharing a common test fixture would come in handy
* here the test depending on a fixture puts up a hard requirement for the actual advice to be there.
* thus, the advice support system can be used to communicate a need for advice
* but it seems unreasonable to extend it actually to transmitt a control flow
!properties of placement
The placement concept plays a fundamental role within Lumiera's HighLevelModel. Besides just being a way of sticking objects together and defining the common properties of //temporal position and output destination,// we try to push this approach to enable a more general, open and generic use. "Placement" is understood as locating within a grid comprised of various degrees of freedom — where locating in a specific way might create additional dimensions to be included into the placement. The standard example is an output connection creating additional adjustable parameters controlling the way the connected object is embedded into a given presentation space (consider e.g. a sound object, which — just by connection, gains the ability of being //panned// by azimuth, elevation and distance)
* in this case, obviously the colaboration is n:m, while each partner preferrably should only see a single advice link.
* advice is used here to negotiate a direct colaboration, which is then handed off to another facility (wiring a control connection)
* the possibility of an advice colaboration in this case is initiated rather from the side of the advisor
* deriving an advice point solution includes some kind of negotioation or active re-evaluation
* the possible adivsors have to be queried according to their placement scope relations
* this queriying might even trigger a resolution process within the advising placement.
__Note__: after detailed analysis, this use case was deemed beyond the scope of the [[Advice]] core concept and idea.
//As a use case, it was dropped.// But we retain some of the properties discovered by considering this scenario, especially the n:m relation, the symmetry in terms of opening the collaboration, and the possibility to have a specially implemented predicate in the binding pattern.
→ AdviceRequirements
Memory management facility for the low-level model (render nodes network). The model is organised into temporal segments, which are considered to be structurally constant and uniform. The objects within each segment are strongly interconnected, and thus each segment is being built in a single build process and is replaced or released as a whole. __~AllocationCluster__ implements memory management to support this usage pattern. He owns a number of object families of various types.[>img[draw/AllocationCluster.png]]
* [[processing nodes|ProcNode]] — probably with several subclasses (?)
* [[wiring descriptors|WiringDescriptor]]
* the input/output descriptor arrays used by the latter
To Each of those families we can expect an initially undetermined (but rather large) number of individual objects, which can be expected to be allocated within a short timespan and which are to be released cleanly on destruction of the AllocationCluster.
''Problem of calling the dtors''
Even if the low-level memory manager(s) may use raw storage, we require that the allocated object's destructors be called. This means keeping track at least of the number of objects allocated (without wasting too much memory for bookkeeping). Besides, as the objects are expected to be interconnected, it may be dangerous to destroy a given family of objects while another family of objects may rely on the former in its destructor. //If we happen do get into this situation,// we need to define a priority order on the types and assure the destruction sequence is respected.
→ see MemoryManagement
Asset management is a subsystem on its own. Assets are "things" that can be loaded into a session, like Media, Clips, Effects, Transitions. It is the "bookkeeping view", while the Objects in the Session relate to the "manipulation and process view". Some Assets can be //loaded// and a collection of Assets is saved with each Session. Besides, there is a collection of basic Assets always available by default.
The Assets are important reference points holding the information needed to access external resources. For example, an Clip asset can reference a Media asset, which in turn holds the external filename from which to get the media stream. For Effects, the situation is similar. Assets thus serve two quite distinct purposes. One is to load, list, group search and browse them, and to provide an entry point to create new or get at existing MObject in the Session, while the other purpose is to provide attribute and property information to the inner parts of the engine, while at the same time isolating and decoupling them from environmental details.
We can distinguish several different Kinds of Assets, each one with specific properties. While all these Kinds of Assets implement the basic Asset interface, they in turn are the __key abstractions__ of the asset management view. Mostly, their interfaces will be used directly, because they are quite different in behaviour. Thus it is common to see asset related operations being templated on the Asset Kind.
→ see also [[Creating and registering Assets|AssetCreation]]
[img[Asset Classess|uml/fig130309.png]]
!Media Asset
Some piece of Media Data accessible at some external Location and able to be processed by Lumiera. A Media File on Harddisk can be considered as the most basic form of Media Asset, with some important derived flavours, like a Placeholder for a currently unavailable Source, or Media available in different Resolutions or Formats.
* __outward interface operations__ include querying properties, creating an Clip MObject, controlling processing policy (low res proxy placeholders, interlacing and other generic pre- and postprocessing)
* __inward interface operations__ include querying filename, codec, offset and any other information necessary for creating a source render node, getting additional processing policy decisions (handling of interlacing, aspect ratio).
→ MediaAsset
!Processing Asset
Some software component able to work on media data in the Lumiera Render engine Framework. This includes all sorts of loadable effects, as well as some of the standard, internal facilities (Mask, Projector). Note that Processing Assets typically provide some attachment Point or means of communication with GUI facilities.
* __outward interface operations__ include getting name and description, investigating the media types the processor is able to handle, cause the underlying module to be acutally loaded...
* __inward interface operations__ include resolving the actual processing function.
→ ProcAsset
!Structural Asset
Some of the building blocks providing the framework for the objects placed into the current Session. Notable examples are [[processing pipes|Pipe]] within the high-level-model, Viewer attachment points, Tracks, Sequences, Timelines etc.
* __outward interface operations__ include...
* __inward interface operations__ include...
→ StructAsset {{red{still a bit vague...}}}
!Meta Asset
Any resources related to the //reflective recurse of the application on itself,// including parametrisation and customisation aspects and similar metadata, are categorised and tracked apart of the primary entities. Examples being types, scales and quantisation grids, decision rules, control data stores (automation data), annotations attached to labels, inventory entities etc.
* __outward interface operations__ include...
* __inward interface operations__ include...
→ MetaAsset {{red{just emerging as of 12/2010}}}
!!!!still to be worked out..
is how to implement the relationship between [[MObject]]s and Assets. Do we use direct pointers, or do we prefer an ID + central registry approach? And how to handle the removal of an Asset.
→ see also [[analysis of mem management|ManagementAssetRelation]]
→ see also [[Creating Objects|ObjectCreation]], especially [[Assets|AssetCreation]]
//9/07: currently implementing it as follows: use a refcounting-ptr from Clip-~MObject to asset::Media while maintaining a dependency network between Asset objects. We'll see if this approach is viable//
Assets are created by a Factories returning smart pointers; the Asset creation is bound to specific use cases and //only available// for these specific situations. There is no generic Asset Factory.
For every Asset we generate a __Ident tuple__ and a long ID (hash) derived from this Ident tuple. The constructor of the abstract base class {{{Asset}}} takes care of this step and automatically registeres the new Asset object with the AssetManager. Typically, the factory methods for concrete Asset classes provide some shortcuts providing sensible default values for some of the Ident tuple data fields. They may take additional parameters — for example the factory method for creating {{{asset::Media}}} takes a filename (and may at some point in the future aply "magic" based on examination of the file → LoadingMedia)
Generally speaking, assets can be seen as the statical part or view of the session and model. They form a global scope and are tied to the [[model root|ModelRootMO]] — which means, they're going to be serialised and de-serialised alongside with this model root scope. Especially the de-serialisation triggers (re)-creation of all assets associated with the session to be loaded.
{{red{TODO:}}} //there will be a special factory mechanism for this case, details pending definition as of 2/2010 //
The Asset Manager provides an Interface to an internal Database holding all Assets in the current Session and System state. It may be a real Database at some point (and for the moment it's a Hashtable). Each [[Asset]] is registered automatically with the Asset Manager; it can be queried either by it's //identification tuple// or by it's unique ID.
Conceptually, assets belong to the [[global or root scope|ModelRootMO]] of the session data model. A mechanism for serialising and de-serialising all assets alongside with the session is planned as of 2/2010
Conceptually, Assets and ~MObjects represent different views onto the same entities. Assets focus on bookkeeping of the contents, while the media objects allow manipulation and EditingOperations. Usually, on the implementation side, such closely linked dual views require careful consideration.
!redundancy
Obviously there is the danger of getting each entity twice, as Asset and as ~MObject. While such dual entities could be OK in conjunction with much specialised processing, in the case of Lumiera's Proc-Layer most of the functionality is shifted to naming schemes, configuration and generic processing, leaving the actual objects almost empty and deprived of distinguishing properties. Thus, starting out from the required concepts, an attempt was made to join, reduce and straighten the design.
* type and channel configuration is concentrated to MediaAsset
* the accounting of structural elements in the model is done through StructAsset
* the object instance handling is done in a generic fashion by using placements and object references
* clips and labels appear as ~MObjects solely; on the asset side there is just an generic [[id tracking mechanism|TypedID]].
* tracks are completely deprived of processing functionality and become lightweight containers, also used as clip bins.
* timelines and sequences are implemented as façade to equivalent structures within the model
* this leaves us only with effects requiring both an object and asset implementation
[<img[Fundamental object relations used in the session|uml/fig138885.png]]
Placing an MObject relatively to another object such that it should be handled as //attached// to the latter results in several design and implementation challenges. Actually, such an attachment creates a cluster of objects. The typical use case is that of an effect attached to a clip or processing pipe.
* attachment is not a globally fixed relation between objects, rather, it typically exists only for some limited time span (e.g. the duration of the basic clip the effect is attached to)
* the order of attachment is important and the attached placement may create a fork in the signal flow, so we need a way for specifying reproducibly how the resulting wiring should be
* when building, we access the information in reversed direction: we have the target object and need to query for all attachments
The first step towards an solution is to isolate the problem; obviously we don't need to store the objects differently, we just need //information about attached objects// for some quite isolated tasks (namely for creating a GUI representation and for combining attached objects into a [[Pipe]] when building). Resorting to a query (function call) interface should turn the rest of the problem into an implementation detail. Thus
* for an __attachment head__ (= {{{Placement<MObject>}}} to which other objects have been attached) get the ordered list of attachments
* for an __attached placement__ (member of the cluster) get the placement of the corresponding attachment head
* retrieve and break the attachment when //deleting.//
!!Implementation notes
Attachment is managed within the participating placements, mostly by special [[locating pins|LocatingPin]]. Attachment doesn't necessarily nail down an attached object to a specific position, rather the behaviour depends on the type of the object and the locating pins actually involved, especially on their order and priority. For example, if an {{{Placement<Effect>}}} doesn't contain any locating pin defining a temporal position, then the attachment will result in the placement inheriting the temporal placement of the //attachment head// (i.e. the clip this effect has been attached to). But, if on the contrary the effect in question //does// have an additional locating pin, for example relative to another object or even to a fixed time position, this one will "win" and determine the start position of the effect — it may even move the effect out of the time interval covered by the clip, in which case the attachment has no effect on the clip's processing pipe.
The attachment relation is hierarchical and has a clearly defined //active// and //passive// side: The attachment head is the parent node in a tree, but plays the role of the passive partner, to which the child nodes attach. But note, this does not mean we are limited to a single attachment head. Actually, each placement has a list of locating pins and thus can attach to several other placements. For example, a transition attaches to at least two local pipes (clips). {{red{TODO: unresolved design problem; seems to contradict the PlacementScope}}}
!!!!Relation to memory management
Attachment on itself does //not// keep an object alive. Rather, it's implemented by an opaque ID entry (→ PlacementRef), which can be resolved by the PlacementIndex. The existence of attachments should be taken into account when deleting an object, preferably removing any dangling attachments to prevent an exception to be thrown later on. On the other hand, contrary to the elements of the HighLevelModel, processing nodes in the render engine never depend on placements — they always refer directly to the MObject instance or even the underlying asset. In the case of MObject instances, the pointer from within the engine will //share ownership// with the placement (remember: both are derived from {{{boost::shared_ptr}}}).
Automation is treated as a function over time. It is always tied to a specific Parameter (which can thus be variable over the course of the timeline). All details //how// this function is defined are completely abstracted away. The Parameter uses a ParamProvider to get the value for a given Time (point). Typically, this will use linear or bezier interpolation over a set of keyframes internally. Parameters can be configured to have different value ranges and distribution types (on-off, stepped, continuous, bounded)
[img[how to implement Automation|uml/fig129669.png]]
While generally automation is treated as a function over time, defining and providing such a function requires some //Automation Data.// The actual layout and meaning of this data is deemed an implementation detail of the [[parameter provider|ParamProvider]] used, but nevertheless an automation data set has object characteristics within the session (high-level-model), allowing it to be attached, moved and [[placed|Placement]] by the user.
Starting out from the concepts of Objects, Placement to Tracks, render Pipes and connection properties (→ see [[here|TrackPipeSequence]]) within the session, we can identify the elementary operations occuring within the Builder. Overall, the Builder is organized as application of //visiting tools// to a collection of objects, so finally we have to consider some object kind appearing in the working function of the given builder tool, which holds at this moment some //context//. The job now is to organize this context such as to create a predictable build process from this //event driven// approach.
→see also: BuilderPrimitives for the elementary situations used to cary out the building operations
!Builder working Situations
# any ''Clip'' (which at this point has been reduced already to a part of a simple elementary media stream → see [[Fixture]])
## yields a source reading node
## which needs to be augmented by the underlying media's [[processing pattern|ProcPatt]]
##* thus inserting codec(s) and source transformations
##* effectively this is an application of effects
## at this point we have to process (and maybe generate on-the-fly) the [[source port of this clip|ClipSourcePort]]
##* the output of the source reading and preprocessing defined thus far is delivered as input to this port, which is done by a ~WiringRequest (see below)
##* as every port, it is the entry point to a [[processing pipe|Pipe]], thus the source port has a processing pattern, typically inserting the camera (transformation effect) at this point
## followed by the application of effects
##* separately for every effect chain rooted (placed) directly onto the clip
##* and regarding the chaining order
## next we have to assess the [[pipes|Pipe]] to which the clip has been placed
## producing a [[wiring request|WiringRequest]] for every pair {{{(chainEndpoint, pipe)}}}
# [>img[draw/Proc.builder1.png]] attaching an ''Effect'' is actually always an //insertion operation// which is done by //prepending// to the previously built nodes. Effects may be placed as attached to clips and pipes, which causes them to be included in the processing chain at the given location. Effects may as well be placed at an absolute time, which means they are to be applied to every clip that happens to be at this time — but this usecase will be reolved when creating the Fixture, causing the effect to be attached to the clips in question. The same holds true for Effects put on tracks.
# treating an ''wiring request'' means
## detecting possible and impossible connections
## deriving additional possible "placement dimensions" generated by executing such an connection (e.g. connecting a mono source to a spatial sound system bus creates panning possibilities)
##* deriving parameter sources for this additional degrees of freedom
##* fire off insertion of the necessary effects to satisfy this connection request and implement the additional "placement dimensions" (pan, layer order, overlay mode, MIDI channel selection...)
# processing the effects and further placements ''attached to a Pipe'' is handled identical to the processing done with all attachments to individual clips.
# ''Transitions'' are to be handled differently according to their placement (→ more on [[Transitions|TransitionsHandling]])
#* when placed normally to two (or N) clips, they are inserted at the exit node of the clip's complete effect chain.
#* otherwise, when placed to the source port(s) or when placed to some other pipes they are inserted at the exit side of those pipe's effect chains. (Note: this puts additional requirements on the transition processor, so not every transition can be placed this way)
After consuming all input objects and satisfying all wiring requests, the result is a set of [[exit nodes|ExitNode]] ready for pulling data. We call the network reachable from such an exit node a [[Processor]], together all processors of all segments and output data types comprise the render engine.
!!!dependencies
Pipes need to be there first, as everything else will be plugged (placed) to a pipe at some point. But, on the other hand, for the model as such, pipes are optional: We could create sequences with ~MObjects without configuring pipes (but won't be able then to build any render processor of course). Similarily, there is no direct relation between tracks and pipes. Each sequence is comprised of at least one root track, but this has no implications regarding any output pipe.
Effects can be attached only to already existing pipelines, starting out at some pipes entry port or the source port of some clip. Besides that, all further parts can be built in any order and independent of each other. This is made possible by using [[wiring requests|WiringRequest]], which can be resolved later on. So, as long as we start out with the tracks (to resolve any pipe they are placed to), and further, if we manage to get any effect placed to some clip-MO //after// setting up and treating the clip, we are fine and can do the building quasi event driven.
!!!building and resolving
Building the network for the individual objects thus creates a queue of wiring requests. Some of them may be immediately resolvable, but detecting this correctly can be nontrivial, and so it seems better to group all wiring requests based on the pipe and treat them groupwise. Because — in the most general case — connecting includes the use of transforming and joining nodes, which can create additional wiring requests (e.g. for automation parameter data connections). Finally, if the network is complete, we could perform [[optimisations|RenderNetworkOptimisation]]
/***
|Name|BetterTimelineMacro|
|Created by|SaqImtiaz|
|Location|http://tw.lewcid.org/#BetterTimelineMacro|
|Version|0.5 beta|
|Requires|~TW2.x|
!!!Description:
A replacement for the core timeline macro that offers more features:
*list tiddlers with only specfic tag
*exclude tiddlers with a particular tag
*limit entries to any number of days, for example one week
*specify a start date for the timeline, only tiddlers after that date will be listed.
!!!Installation:
Copy the contents of this tiddler to your TW, tag with systemConfig, save and reload your TW.
Edit the ViewTemplate to add the fullscreen command to the toolbar.
!!!Syntax:
{{{<<timeline better:true>>}}}
''the param better:true enables the advanced features, without it you will get the old timeline behaviour.''
additonal params:
(use only the ones you want)
{{{<<timeline better:true onlyTag:Tag1 excludeTag:Tag2 sortBy:modified/created firstDay:YYYYMMDD maxDays:7 maxEntries:30>>}}}
''explanation of syntax:''
onlyTag: only tiddlers with this tag will be listed. Default is to list all tiddlers.
excludeTag: tiddlers with this tag will not be listed.
sortBy: sort tiddlers by date modified or date created. Possible values are modified or created.
firstDay: useful for starting timeline from a specific date. Example: 20060701 for 1st of July, 2006
maxDays: limits timeline to include only tiddlers from the specified number of days. If you use a value of 7 for example, only tiddlers from the last 7 days will be listed.
maxEntries: limit the total number of entries in the timeline.
!!!History:
*28-07-06: ver 0.5 beta, first release
!!!Code
***/
//{{{
// Return the tiddlers as a sorted array
TiddlyWiki.prototype.getTiddlers = function(field,excludeTag,includeTag)
{
var results = [];
this.forEachTiddler(function(title,tiddler)
{
if(excludeTag == undefined || tiddler.tags.find(excludeTag) == null)
if(includeTag == undefined || tiddler.tags.find(includeTag)!=null)
results.push(tiddler);
});
if(field)
results.sort(function (a,b) {if(a[field] == b[field]) return(0); else return (a[field] < b[field]) ? -1 : +1; });
return results;
}
//this function by Udo
function getParam(params, name, defaultValue)
{
if (!params)
return defaultValue;
var p = params[0][name];
return p ? p[0] : defaultValue;
}
window.old_timeline_handler= config.macros.timeline.handler;
config.macros.timeline.handler = function(place,macroName,params,wikifier,paramString,tiddler)
{
var args = paramString.parseParams("list",null,true);
var betterMode = getParam(args, "better", "false");
if (betterMode == 'true')
{
var sortBy = getParam(args,"sortBy","modified");
var excludeTag = getParam(args,"excludeTag",undefined);
var includeTag = getParam(args,"onlyTag",undefined);
var tiddlers = store.getTiddlers(sortBy,excludeTag,includeTag);
var firstDayParam = getParam(args,"firstDay",undefined);
var firstDay = (firstDayParam!=undefined)? firstDayParam: "00010101";
var lastDay = "";
var field= sortBy;
var maxDaysParam = getParam(args,"maxDays",undefined);
var maxDays = (maxDaysParam!=undefined)? maxDaysParam*24*60*60*1000: (new Date()).getTime() ;
var maxEntries = getParam(args,"maxEntries",undefined);
var last = (maxEntries!=undefined) ? tiddlers.length-Math.min(tiddlers.length,parseInt(maxEntries)) : 0;
for(var t=tiddlers.length-1; t>=last; t--)
{
var tiddler = tiddlers[t];
var theDay = tiddler[field].convertToLocalYYYYMMDDHHMM().substr(0,8);
if ((theDay>=firstDay)&& (tiddler[field].getTime()> (new Date()).getTime() - maxDays))
{
if(theDay != lastDay)
{
var theDateList = document.createElement("ul");
place.appendChild(theDateList);
createTiddlyElement(theDateList,"li",null,"listTitle",tiddler[field].formatString(this.dateFormat));
lastDay = theDay;
}
var theDateListItem = createTiddlyElement(theDateList,"li",null,"listLink",null);
theDateListItem.appendChild(createTiddlyLink(place,tiddler.title,true));
}
}
}
else
{
window.old_timeline_handler.apply(this,arguments);
}
}
//}}}
Binding-~MObjects are used to associate two entities within the high-level model.
More specifically, such a binding serves
* to outfit any top-level [[Timeline]] with real content, which is contained within a [[Sequence]]
* to build a VirtualClip, that is to link a complete sequence into another sequence, where it appears like a new virtual media or clip.
!Properties of a Binding
Binding is a relation entity, maintaining a link between parts of the session. Actually this link is achieved somewhat indirect: The binding itself is an MObject, but it points to a [[sequence asset|Sequence]]. Moreover, in case of the (top-level) timelines, there is a timeline asset acting as a frontend for the ~BindingMO.
* the binding exposes special functions needed to implement the timeline {{red{planned as of 11/10}}}
* similarly, the binding exposes functions allowing to wrap up the bound sequence as VirtualMedia (when acting as VirtualClip).
* the Binding holds an OutputMapping -- allowing to specify, resolve and remember [[output designations|OutputDesignation]]
Note: there are other binding-like entities within the model, which are deliberately not subsumed below this specification, but rather implemented stand alone.
→ see also SessionInterface
!Implementation
On the implementation side, we use a special kind of MObject, acting as an anchor and providing an unique identity. Like any ~MObject, actually a placement establishes the connection and the scope, and typically constitutes a nested scope (e.g. the scope of all objects //within// the sequence to be bound into a timeline)
Binding can be considered an implementation object, rarely to be created directly. Yet it is part of the high-level model
{{red{WIP 11/10}}}: it is likely that -- in case of creating a VirtualClip -- BindingMO will be hooked up behind another façade asset, acting as ''virtual media''
!!!channel / output mapping {{red{WIP 11/10}}}
The Binding-~MObject stores an OutputMapping. Basically this, together with OutputDesignation, implements the mapping behaviour
* during the build process, output designation(s) will be retrieved for each pipe.
* indirect and relative designations are to resolved; the relative ones are forwarded to the next enclosing binding
* in any case, the result is an direct WiringPlug, which can then be matched up and wired
* but in case of implementing a virtual clip, in addition to the direct wiring...
** a relative output designation (the N^^th^^ channel of this kind) is carried over to the target scope to be re-resolved there.
** any output designation specification yields a summation pipe at the binding, i.e. a position corresponding to the global pipes when using the same sequence as timeline.
** The output of these summation pipes is treated like a media channels
** but for each of those channels, an OutputDesignation is //carried over// into the target (virtual clip)
*** now, if a distinct output designation can be determined at this placement of the virtual clip, it will be used for the further routing
*** otherwise, we try to re-evaluate the original output designation carried over. <br/>Thus, routing will be done as if the original output designation was given at this placement of the virtual clip.
There is some flexibility in the HighLevelModel, allowing to attach the same [[Sequence]] onto multiple [[timelines|Timeline]] or even into a [[meta-clip|VirtualClip]]. Thus, while there is always an containment relation which can be used to define the current PlacementScope, we can't always establish an unique path from any given location up to the model root. In the most general case, we have to deal with a DAG, not a tree.
!solution idea
Transform the DAG into a tree by //focussing// on the current situation and context. Have a state containing the //current path.// → QueryFocus
Incidentally, this problem is quite similar to file system navigation involving ''symlinks''.
* under which circumstances shall discovery follow symlinks?
* where do you get by {{{cd ..}}} — especially when you went down following a symlink?
This leads us to building our solution here to match a similar behaviour pattern, according to the principle of least surprise. That is, disovery shall follow the special [[bindings|BindingMO]] only when requested explicitly, but the current "shell" (QueryFocus) should maintain a virtual/effective path to root scope.
!!detail decisions
!!!!how to represent scoping
We us a 2-layer approach: initially, containment is implemented through a registration in the PlacementIndex. Building on that, scope is layered on top as an abstraction, which uses the registered containment relation, but takes the current access path into account at the critical points (where a [[binding|BindingMO]] comes into play)
!!!!the basic containment tree
each Placement (with the exception of the root) gets registered as contained in yet another Placement. This registration is entirely a tree, and thus differs from the real scope nesting at the Sequence level: The scopes constituting Sequences and Timelines are registered as siblings, immediately below the root. This has some consequences
# Sequences as well as Timelines can be discovered as contents of the model root
# ScopePath digresses at Sequence level from the basic containment tree
!!!!locating a placement
constituting the effective logical position of a placement poses sort-of a chicken or egg problem: We need already a logical position to start with. In practice, this is done by recurring on the QueryFocus, which is a stack-like state and automatically follows the access or query operations on the session. //Locating a placement// is done by //navigating the current query focus.//
!!!!navigating a scope path location
As the current query focus stack top always holds a ScopePath, the ''navigating operation'' on ScopePath is the key for managing this logical view onto the "current" location.
* first, try to locate the new scope in the same sequence as the current scope, resulting in a common path prefix
* in case the new scope belongs to a different sequence, this sequence might be connected to the current one as a meta-clip, again resulting in a common prefix
* otherwise use the first possible binding according to the ordering of timelines as a prefix
* use the basic containment path as a fallback if no binding exists
!!{{red{WIP 9/10}}}Deficiencies
To buy us some time, analysing and implementing the gory details of scope path navigation and meta-clips was skipped for now.
Please note the shortcomings and logical contradictions in the solution currently in code:
* the {{{ScopePath::navigate()}}}-function was chosen as the location to implement the translation logic. //But actually this translation logic is missing.//
* so basically we're just using the raw paths of the basic containment tree; more specifically, the BindingMO (=Timeline) isn't part of the derived ScopePath
* this will result in problems even way before we implement meta-clips (because the Timeline is assumed to provide output routing information) to the Placements
* QueryFocus, with the help of ScopeLocator exposes the query services of the PlacementIndex. So actually it's up to the client code to pick the right functions. This might get confusing
* The current design rather places the implementation according to the roles of the involved entities, which causes some ping-pong on the implementation level. Especially the ScopeLocator singleton can be accessed multiple times. This is the usual clarity vs. performance tradeoff. Scope resolution is assumed rather to be //not performance critical.//
All rendering, transformations and output of media data requires using ''data buffers'' -- but the actual layout and handling of these buffers is closely related to the actual implementation of these operations. As we're relying heavily on external libraries and plug-ins for performing these operations, there is no hope getting away with just one single {{{Buffer}}} data type definition. Thus, we need to confine ourselves to a common denominator of basic operations regarding data buffers and abstract the access to these operations through a BufferProvider entity. Beyond these basic operations, mostly we just need to assure that //a buffer exists as an distinguishable element// -- which in practice boils down to pushing around {{{void*}}} variables.
Obviously, overloading a pointer with semantic meaning isn't exactly a brilliant idea -- and the usual answer is to embed this pointer into a smart handle, which also yields the nice side-effect of explaining this design to the reader. Thus a buffer handle
* can only be obtained from a BufferProvider
* can be used to identify a buffer
* can be dereferenced
* can be copied
!design quest: buffer type information
To perform anything useful with such a buffer handle, the client code needs some additional information, which can be generalised into a //type information:// Either, the client needs to know the size and kind of data to expect in the buffer, maybe just assume to get a specific buffer with suitably dimensions, or the client needs to know which buffer provider to contact for any management operations on that buffer (handle). And, at some point there needs to be a mechanism to verify the validity of a handle. But all of this doesn't mean that it's necessary to encode or embedd this information directly into the handle -- it might also be stored into a registration table (which has the downside of creating contention), or it might just be attached implicitly to the invocation context.
Just linking this type information to the context is certainly the most elegant solution, but also by far the most difficult to achieve -- not to mention the implicit dependency on a very specific invocation situation. So for now (9/2011) it seems best to stick to the simple and explicit implementation, just keeping that structural optimisation in mind. And the link to this buffer type information should be made explicit within the definition anyway, even if we choose to employ another design tradeoff later.
* thus the conclusion is: we introduce a ''descriptor object'', which will be stored within the handle
* each BufferProvider exposes a ''descriptor prototype''; it can be specialised and used by to [[organise implementation details|BufferMetadata]]
!sanity checks
there are only limited sanity checks, and they can be expected to be optimised away for production builds.
Basically the client is responsible for sane buffer access.
Buffers are used to hold the media data for processing and output. Within the Lumiera RenderEngine and [[Player]] subsystem, we use some common concepts to handle the access and allocation of working buffers. Yet this doesn't imply having only one central authority in charge of every buffer -- such an approach wouldn't be possible (due to collaboration with external systems) and wouldn't be desirable either. Rather, there are some common basic usage //patterns// -- and there are some core interfaces used throughout the organisation of the rendering process.
Mostly, the //client code,// i.e. code in need of using buffers, can access some BufferProvider, thereby delegating the actual buffer management. This binds the client to adhere to kind of a //buffer access protocol,// comprised of the ''announcing'', ''locking'', optionally ''attaching'' and finally the ''releasing'' steps. Here, the actual buffer management within the provider is a question of implementation and will be configured during build-up of the scope in question.
!usage situations
;rendering
:any calculations and transformations of media data typically require an input- and output buffer. To a large extent, these operations will be performed by specialised libraries, resulting in a call to some plain-C function receiving pointers to the required working buffers. Our invocation code has the liability to prepare and provide those pointers, relying on a BufferProvider in turn.
;output
:most any of the existing libraries for handling external output require the client to adhere to some specific protocol. Often, this involves some kind of callback invoked at the external library's discretion, thus forcing our engine to prepare data within an intermediary buffer. Alternatively, the output system might provide some mechanism to gain limited direct access to the output buffers, and such an access can again be exposed to our internal client code through the BufferProvider abstraction.
!primary implementations
;memory pool
:in all those situations, where we just need a working buffer for some time, we can rely on our internal custom memory allocator.
:{{red{~Not-Yet-Implemented as of 9/11}}} -- as a fallback we just rely on heap allocations through the language runtime
;frame cache
:whenever a calculated result may be of further interest, beyond the immediate need triggering the calculation, it might be eligible for caching.
:The Lumiera ''frame cache'' is a special BufferProvider, maintaining a larger pool of buffers which can be pinned and kept around for some time,
:accomodating limited resources and current demand for fresh result buffers.
the generic BufferProvider implementation exposes a service to attach and maintain additional metadata with individual buffers. Using this service is not mandatory -- a concrete buffer provider implementation may chose to maintain more specific metadata right on the implementation level, especially if more elaborate management is necessary within the implementation anyway (e.g. the frame index). We can expect most buffer provider implementations to utilise at least the generic buffer type id service though.
!buffer types and descriptors
Client code accesses buffer through [[smart buffer handles|BuffHandle]], including some kind of buffer type information, encoded into a type ID within the ''buffer descriptor''. These descriptors are used like prototypes, relating the type-~IDs hierarchically. Obviously, the most fundamental distinction is the BufferProvider in charge for that specific buffer. Below that, the next mandatory level of distinction is the ''buffer size''. In some cases, additional distinctions can be necessary. Each BufferProvider exposes a service to yield unique type ~IDs governed by such a hierarchical scheme.
!state and metadata for individual buffers
Beyond that, it can be necessary to associate at least a state flag with //individual buffers.// Doing so requires the buffer to be in //locked state,// otherwise it wouldn't be distinguishable as an separate entity (a client is able to access the buffer memory address only after "locking" this buffer). Especially when using a buffer provider in conjunction with an OutputSlot, these states and transitions are crucial for performing an orderly handover of generated data from the producer (render engine) to the consumer (external output sink).
__Note__: while the API to access this service is uniform, conceptually there is a difference between just using the (shared) type information and associating individual metadata, like the buffer state. Type-~IDs, once allocated, will never be discarded (within the lifetime of an Lumiera application instance -- buffer associations aren't persistent). To the contrary, individual metadata //will be discarded,// when releasing the corresponding buffer. According to the ''prototype pattern'', individual metadata is treated as a one-way-off specialisation.
It turns out that -- throughout the render engine implementation -- we never need direct access to the buffers holding actual media data. Buffers are just some entity to be //managed,// i.e. "allocated", "locked" and "released"; the //actual meaning of these operations can be left to the implementation.// The code within the render engine just pushes around ''smart-prt like handles''. These [[buffer handles|BuffHandle]] act as a front-end, being created by and linked to a buffer provider implementation. There is no need to manage the lifecycle of buffers automatically, because the use of buffers is embedded into the render calculation cycle, which follows a rather strict protocol anyway. Relying on the [[capabilities of the scheduler|SchedulerRequirements]], the sequence of individual jobs in the engine ensures...
* that the availability of a buffer was ensured prior to planning a job ("buffer allocation")
* that a buffer handle was obtained ("locked") prior to any operation requiring a buffer
* that buffers are marked as free ("released") after doing the actual calculations.
!operations
While BufferProvider is an interface meant to be backed by various different kinds of buffer and memory management approaches, there is a common set of operations to be supported by any of them
;announcing
:client code may announce beforehand that it expects to get a certain amount of buffers. Usually this causes some allocations to happen right away, or it might trigger similar mechanisms to ensure availability; the BufferProvider will then return the actual number of buffers guaranteed to be available. This announcing step is optional an can happen any time before or even after using the buffers and it can be repeated with different values to adjust to changing requirements. Thus the announced amount of buffers always denotes //additional buffers,// on top of what is actively used at the moment. This safety margin of available buffers usually is accounted separately for each distinct kind of buffer (buffer type). There is no tracking as to which specific client requested buffers, beyond the buffer type.
;locking
:this operation actually makes a buffer available for a specific client and returns a [[buffer handle|BuffHandle]]. The corresponding buffer is marked as used and can't be locked again unless released. If necessary, at that point the BufferProvider might allocate memory to accommodate (especially when the buffers weren't announced beforehand). The locking may fail and raise an exception. You may expect failure to be unlikely when buffers have been //announced beforehand.// To support additional sanity checks, the client may provide a token-ID with the lock-operation. This token may be retrieved later and it may be used to ensure the buffer is actually locked for //this token.//
;attaching
:optionally the client may attach an object to a locked buffer. This object is placement-constructed into the buffer and will be destroyed automatically when releasing the buffer. Alternatively, the client may provide a pair of constructor- / destructor-functors, to be invoked in a similar way. This allows e.g. to install descriptor structures within the buffer, as required by an external library.
;releasing
:buffers need to be released explicitly by the client code. This renders the corresponding BuffHandle invalid, (optionally) invokes a destructor function of an attached object and maybe reclaims the buffer memory
!!type metadata service
In addition to the basic operations, clients may associate BufferMetadata with individual buffers;
in the basic form, this means just maintaining a type tag describing the kind of buffer, while optionally this service might be extended to e.g. associating a state flag.
__see also__
→ OutputSlot relying on a buffer provider to deal with frame output buffers
→ more about BufferManagement within the RenderEngine and [[Player]] subsystem
→ RenderMechanics for details on the buffer management within the node invocation for a single render step
The invocation of individual [[render nodes|ProcNode]] uses an ''buffer table'' internal helper data structure to encapsulate technical details of the allocation, use, re-use and feeing of data buffers for the media calculations. Here, the management of the physical data buffers is delegated through a BufferProvider, which typically is implemented relying on the ''frame cache'' in the backend. Yet some partially quite involved technical details need to be settled for each invocation: We need input buffers, maybe provided as external input, while in other cases to be filled by a recursive call. We need storage to prepare the (possibly automated) parameters, and finally we need a set of output buffers. All of these buffers and parameters need to be rearranged for invoking the (external) processing function, followed by releasing the input buffers and commiting the output buffers to be used as result.
Because there are several flavours of node wiring, the building blocks comprising such a node invocation will be combined depending on the circumstances. Performing all these various steps is indeed the core concern of the render node -- with the help of BufferTable to deal with the repetitive, tedious and technical details.
!requirements
The layout of the buffer table will be planned beforehand for each invocation, allongside with planning the individual invocation jobs for the scheduler. At that point, a generic JobTicket for the whole timeline segment is available, describing the necessary operations in an abstract way, as determined by the preceeding planning phase. Jobs are prepared chunk wise, some time in advance (but not all jobs of at once). Jobs will be executed concurrently. Thus, buffer tables need to be created repeatedly and placed into a memory block accessed and owned exclusively by the individual job.
* within the buffer table, we need an working area for the output handles, the input handles and the parameter descriptors
* actually, these can be seen as pools holding handle objects which might even be re-used, especially for a chain of effects calculated in-place.
* each of these pools is characterised by a common //buffer type,// represented as buffer descriptor
* we need some way to integrate with the StateProxy, because some of the buffers need to be marked especially, e.g. as result
* there should be convenience functions to release all pending buffers, forwarding the release operation to the individual handles
//Building the fixture is actually at the core of the [[builder's operation|Builder]]//
{{red{WIP as of 11/10}}} → see also the [[planning page|PlanningBuildFixture]]
;Resolving the DAG[>img[Steps towards creating a Segmentation|draw/SegmentationSteps1.png]]
Because of the possibility of binding a Sequence multiple times, and maybe even nested as virtual clip, the [[high-level model|HighLevelModel]] actually constitutes a DAG, not a tree. This leds to quite some tricky problems, which we try to resolve by //rectifying the DAG into N virtual trees.// (→ BindingScopeProblem)
Relying on this transformation, each Timeline spans a sub-tree virtually separated from all other timelines; the BuildProcess is driven by [[visiting|VisitorUse]] all the //tangible// objects within this subtree. In the example shown to the right, Sequence-β is both bound as VirtualClip into Sequence-α, as well as bound independently as top-level sequence into Timeline-2. Thus it will be visited twice, but the QueryFocus mechanism ensures that each visitation »sees« the proper context.
;Explicit Placements
Each tangible object placement (relevant for rendering), which is encountered during that visitation, gets //resolved// into an [[explicit placement|ExplicitPlacement]]. If we see [[Placement]] as a positioning within a multi dimensional configuration space, then the resolution into an explicit placement is like the creation of an ''orthogonal base'': Within the explicit placement, each LocatingPin corresponds exactly to one degree of freedom and can be considered independently from all other locating pins. This resolution step removes any fancy dynamic behaviour and all scoping and indirect references. Indeed, an explicit placement is a mere //value object;// it isn't part of the session core (PlacementIndex), isn't typed and can't be referred indirectly.
;Segmentation of Time axis
This simple and explicit positioning thus allows to arrange all objects as time intervals on a single axis. Any change and especially any overlap is likely to create a different wiring configuration. Thus, for each such configuration change, we fork off a new //segment// and //copy over// all partially touched placements. The resulting seamless sequence of non-overlapping time intervals provides the backbone of the datastructure called [[Fixture]].
;Building the Network
From this backbone, the actual [[building mechanism|BuilderMechanics]] proceeds as a ongoing visitation and resolution, resulting in the gowth of a network of [[render nodes|ProcNode]] starting out from the source reading nodes and proceeding up through the local pipes, the transitions and the global pipes. When this build process is exhausted, besides the actual network, the result is a //residuum of nodes not connected any further.// Any of these [[exit nodes|ExitNode]] can be associated to a ~Pipe-ID in the high-level model. Within each segment, there should be one exit node per pipe-ID at max. These are the [[model ports|ModelPort]] resulting from the build process, keyed by their corresponding ~Pipe-ID.
→ see [[Structure of the Fixture|Fixture]]
All decisions on //how // the RenderProcess has to be carried out are concentrated in this rather complicated Builder Subsystem. The benefit of this approach is, besides decoupling of subsystems, to keep the actual performance-intensive video processing code as simple and transparent as possible. The price, in terms of increased complexity — to pay in the Builder — can be handled by making the Build Process generic to a large degree. Using a Design By Contract approach we can decompose the various decisions into small decision modules without having to trace the actual workings of the Build Process as a whole.
[>img[Outline of the Build Process|uml/fig129413.png]]
The building itself will be broken down into several small tool application steps. Each of these steps has to be mapped to the MObjects found on the [[Timeline]]. Remember: the idea is that the so called "[[Fixture]]" contains only [[ExplicitPlacement]]s which in turn link to MObjects like Clips, Effects and [[Automation]]. So it is sufficient to traverse this list and map the build tools to the elements. Each of these build tools has its own state, which serves to build up the resulting Render Engine. So far I see two steps to be necessary:
* find the "Segments", i.e. the locations where the overall configuration changes
* for each segment: generate a ProcNode for each found MObject and wire them accordingly
Note, //we still have to work out how exactly building, rendering and playback work// together with the backend-design. The build process as such doesn't overly depend on these decisions. It is easy to reconfigure this process. For example, it would be possible as well to build for each frame separately (as Cinelerra2 does), or to build one segment covering the whole timeline (and handle everything via [[Automation]]
→see also: [[Builder Overview|Builder]]
→see also: BasicBuildingOperations
→see also: BuilderStructures
→see also: BuilderMechanics
→see also: PlanningBuildFixture
→see also: PlanningSegementationTool
→see also: PlanningNodeCreatorTool
[img[Colaborations in the Build Process|uml/fig128517.png]]
Actually setting up and wiring a [[processing node|ProcNode]] involves several issues and is carried out at the lowest level of the build process.
It is closely related to → [[the way nodes are operated|NodeOperationProtocol]] and the → [[mechanics of the render process|RenderMechanics]]
!!!object creation
The Nodes are small polymorphic objects, carrying configuration data, but no state. They are [[specially allocated|ManagementRenderNodes]], and the object creation is accessible by means of the NodeFactory solely. They //must not be deallocated manually.// The decision of what concrete node type to create depends on the actual build situation and is worked out by the combination of [[mould|BuilderMould]] and [[processing pattern|ProcPatt]] at the current OperationPoint, issuing a call to one of NodeFactory's {{{operator()}}}
!!!node, plugin and processing function
Its a good idea to distinguish clearly between those concepts. A plugin is a piece of (possibly external) code we use to carry out operations. We have to //discover its properties and capabilities.// We don't have to discover anything regarding nodes, because we (Lumiera builder and renderengine) are creating, configuring and wiring them to fit the specific purpose. Both are to be distinguished from processing functions, which do the actual calculations on the media data. Every node typically encompasses at least one processing function, which may be an internal function in the node object, a library function from Lumiera or GAVL, or external code loaded from a plugin.
!!!node interfaces
As a consequence of this distinctions, in conjunction with a processing node, we have to deal with three different interfaces
* the __build interface__ is used by the builder to set up and wire the nodes. It can be full blown C++ (including templates)
* the __operation interface__ is used to run the calculations, which happens in cooperation of Proc-Layer and Backend. So a function-style interface is preferable.
* the __inward interface__ is accessed by the processing function in the course of the calculations to get at the necessary context, including in/out buffers and param values.
!!!wiring data connections
A node //knows its predecessors, but not its successors.// When being //pulled// in operation, it can expect to get a frame provider for accessing the in/out buffer locations (some processing functions may be "in-place capable", but that's only a special case of the former). At this point, the ''pull principle'' comes into play: the node may request input frames from the frame provider, passing its predecessors as a ''continuation''.
With regard to the build process, the wiring of data connections translates into providing the node with its predecessors and preconfiguring the possible continuations. While in the common case, a node has just one input/output and pulls from its predecessor a frame for the same timeline position, the general case can be more contrived. A node may process N buffers in parallel and may require several different time positions for it's input, even at a differing framerate. So the actual source specification is (predNode,time,frameType). The objective of the wiring done in the build process is to factor out the parts known in advance, while in the render process only the variable part need to be filled in. Or to put it differently: wiring builds a higher order function (time)->(continuation), where continuation can be invoked to get the desired input frame.
!!!wiring control conections
In many cases, the parameter values provided by these connections aren't frame based data, rather, the processing function needs a call interface to get the current value (value for a given time), which is provided by the parameter object. Here, the wiring needs to link to the suitable parameter instance, which is located within the high-level model (!). As an additional complication, calculating the actual parameter value may require a context data frame (typically for caching purposes to speed up the interpolation). While these parameter context data frames are completely opaque for the render node, they have to be passed in and out similar to the state needed by the node itself, and the wiring has to prepare for accessing these frames too.
The Builder takes some MObject/[[Placement]] information (called Timeline) and generates out of this a Render Engine configuration able to render this Objects. It does all decisions and retrieves the current configuration of all objects and plugins, so the Render Engine can just process them stright forward.
The Builder is the central part of the [[Builder Pattern|http://en.wikipedia.org/wiki/Builder_pattern]]
<br/>
As the builder [[has to create a render node network|BuilderModelRelation]] implementing most of the features and wiring possible with the various MObject kinds and placement types, it is a rather complicated piece of software. In order to keep it manageable, it is broken down into several specialized sub components:
* clients access builder functionality via the BuilderFacade
* the [[Proc-Layer-Controller|Controller]] initiates the BuildProcess and does the overall coordination of scheduling edit operations, rebuilding the fixture and triggering the Builder
* to carry out the building, we use several primary tools (SegmentationTool, NodeCreatorTool,...), together with a BuilderToolKit to be supplied by the [[tool factory|BuilderToolFactory]]
* //operating the Builder// can be viewed at from two different angles, either emphasizing the [[basic building operations|BasicBuildingOperations]] employed to assemble the render node network, or focussing rather at the [[mechanics|BuilderMechanics]] of cooperating parts while processing.
* besides, we can identify a small set of elementary situations we call [[builder primitives|BuilderPrimitives]], to be covered by the mentioned BuilderToolKit; by virtue of [[processing patterns|ProcPatt]] they form an [[interface to the rule based configuration|BuilderRulesInterface]].
* the actual building (i.e. the application of tools to the timeline) is done by the [[Assembler|BuilderAssembler]], which is basically a collection of functions (but has a small amount of global configuration state)
* any non-trivial wiring of render nodes, tracks, pipes and [[automation|Automation]] is done by the services of the [[connection manager|ConManager]]
The cooperation of several components creates a context of operation for the primary builder working tool, the [[node creator|PlanningNodeCreatorTool]]:
* the BuilderToolFactory acts as the "builder for the builder tools", i.e. we can assume to be able to retrive all needed primary tools and elementary tools from this factory, completely configured and ready to use.
* the [[Assembler|BuilderAssembler]] has the ability to consume objects from the high level model and feed them to the node creator (which translates into a dispatch of individual operations suited to the objects to be treated). This involves some sort of scheduling or ordering of the operaions, which is the only means to direct the overall process such as to create a sensible and usable result. //This is an fundamental design decision:// the actual working tools have no hard wired knowledge of the "right process", which makes the whole Builder highly configurable ("open").
* the [[connection manager|ConManager]] on the contrary is a passive service provider. Fed with [[wiring requests|WiringRequest]], he can determine if a desired connection is possible, and what steps to take to implement it; the latter recursively creates further building requests to satisfy by the assembler, and possibly new wiring requests.
!!pattern of operation
The working pattern of this builder mechanics can be described as triggering, enqueuing, priorizing, recursing and exhausting. Without the priorizing part, it would be a depth-first call graph without any context state, forcing us to have all cross reference information available at every node or element to be treated. We prefer to avoid this overhead by ordering the operations into several phases and within these phases into correlated entities with the help of a ''weighting function'' and scheduling with a ''priority queue''
!!call chain
After preparing the tools with the context state of this build process, the assembler drives the visitation process in the right order. The functions embedded within the visitor (NodeCreatorTool) for treating specific kinds of objects in turn use the toolkit (=the fully configured tool factory) to get the mould(s) for the individual steps they need to carry out. This involves preparing the mould (with the high-level object currently in-the-works, a suitable processing pattern and additional references), followed by operating the mould. The latter "plays" the processing pattern in the context of the mould, which, especially with the help of the operation point, carries out the actual building and/or connecting step. While doing so, the node factory will be invoked, which in turn invokes the wiring factory and thus pre-determines the node's prospective mode of operation when later called for rendering.
[>img[Builder creating the Model|uml/fig132868.png]]
The [[Builder]] uses different kinds of tools for creating a network of render nodes from a given high-level model. When breaking down this (necessarily complex) process into small manageable chunks, we arrive at [[elementary building situations|BuilderPrimitives]]. For each of these there is a specialized tool. We denote these tools as "moulds" because they are a rather passive holder for the objects to be attached and wired up. They are shaped according to the basic form the connections have to follow for each of these basic situations:
* attaching an effect to a pipe
* combining pipes via a transition
* starting out a pipe from a source reader
* general connections from the exit node of a pipe to the port of another pipe
In all those cases, the active part is provided by [[processing patterns|ProcPatt]] — sort of micro programs executed within the context of a given mould: the processing pattern defines the steps to take (in the standard/basic case this is just "attach"), while the mould holds and provides the location where these steps will operate. Actually, this location is represented as a OperationPoint, provided by the mould and abstracting the details of making multi-channel connections.
While assembling and building up the render engines node network, a small number of primitive building situations is encountered repeatedly. The BuilderToolKit provides a "[[mould|BuilderMould]]" for each of these situations, typically involving parametrisation and the application of a [[processing pattern|ProcPatt]].
The ''Lifecycle'' of such a mould starts out by arming it with the object references involved into the next building step. After conducting this building step, the resulting render nodes can be found — depending on the situation — attached either to the same mould, or to another kind of mould, but in any case ready to be included in the next building step. Thus, //effectively// the moulds are //used to handle the nodes being built,// due to the fact that the low-level model (nodes to be built) and the high-level model (objects directing what is to be built) are //never connected directly.//
!List of elementary building situations
!!!inserting an Effect or Plugin
[>img[draw/builder-primitives1.png]]
The __~PipeMould__ is used to chain up the effects attached to a clip (=local pipe) or global pipe (=bus)
* participating: a Pipe and an Effect
* point of reference: current exit node of the pipe
* result: Effect appended at the pipe's exit node
* returns: ~PipeMould holding onto the new exit node
@@clear(right):display(block):@@
!!!attaching a transition
[>img[draw/builder-primitives2.png]]
After having completed N pipe's node chains, a __~CombiningMould__ can be used to join them into a [[transition|TransitionsHandling]]
* participating: N pipe's exit nodes, transition
* point of reference: N exit nodes corresponding to (completed) pipes
* result: transition has been attached with the pipe's exit nodes, new wiring requests created attached to the transition's exit node(s)
* returns: ~WiringMould, connected with the created wiring request
Using this mould implicitly "closes" the involved pipes, which means that we give up any reference to the exit node and can't build any further effect attached to this pipes. Generally speaking, "exit node" isn't a special kind of node, rather it's a node we are currently holding on. Similarly, there is nothing directly correlated to a pipe within the render nodes network after we are done with building the part of the network corresponding to the pipe; the latter serves rather as a blueprint for building, but isn't an entity in the resulting low-level model.
Actually, there is {{red{planned}}} a more general (and complicated) kind of transition, which can be inserted into N data connections without joining them together into one single output, as the standard transitions do. The ~CombiningMould can handle this case too by just returning N wiring moulds as a result.
@@clear(right):display(block):@@
!!!building a source connection
[>img[draw/builder-primitives3.png]]
The __~SourceChainMould__ is used as a starting point for any further building, as it results in a local pipe (=clip) rooted at the clip source port. This reflects the fact that the source readers (=media access points) are the //leaf nodes// in the node graph we are about to build.
* participating: source port of a clip, media access point, [[processing pattern|ProcPatt]]
* point of reference: //none//
* result: processing pattern has been //executed//, resulting in a chain of nodes from the source reader to the clip source port
* returns: ~PipeMould holding onto the new exit node (of a yet-empty pipe)
@@clear(right):display(block):@@
!!!wiring a general connection
Any wiring (outside the chain of effects within a pipe) is always done from exit nodes to the port of another pipe, requiring an [[wiring request|WiringRequest]] already checked and deemed resolvable. Within the __~WiringMould__ the actual wiring is conducted, possibly adding a summation node (called "overlayer" in case of video) and typically a fader element (the specific setup to be used is subject to configuration by processing patterns)
* participating: already verified connection request, providing a Pipe and an exit node; a processing pattern and a Placement
* points of reference: exit node and (optionally) starting point of a pipe's chain (in case there are already other connections)
* result: summation node prepended to the port of the pipe, processing pattern has been //executed// for building the connection from the exit node to the pipe's port, ParamProvider has been setup in [[accordance|PlacementDerivedDimension]] to the Placement.
* returns: ~PipeMould holding onto the destination pipe's exit node, ~WiringMould holding onto the port side of the same pipe, i.e. the destination where further connections will insert summation nodes. {{red{TODO how to handle the //empty//-case?}}}
[>img[draw/builder-primitives4.png]]
@@clear(right):display(block):@@
* the MObjects implement //Buildable//
* each Buildable can "receive" a Tool object and apply it
* the different Tool objects are iterated/mapped onto the list of MObjects in the [[Timeline]]
* __Rationale__
** the MObject class hierarchy is rather fixed (it is unlikely the we will be adding much new MObject subclasses)
** so this design makes it easy to add new Tool subclasses, and within each Tool subclass, all operations on the different MObject classes are grouped together, so it is easy to see what is going on.
** a given Tool instance can carry state while being iterated, so we don't need any global (or object-global) variables to hold the result of the build process
This programming technique is often referred to as [["double dispatch" or "visitor"|VisitorUse]]. We use a specialized library implementation of this pattern — heavily inspired by the [[Loki library|http://loki-lib.sourceforge.net/]]. We use this approach not only for the builder, but also for carrying out operations on the objects in the session in a typesafe manner.
It is the low level foundation of the actual [[building operations|BasicBuildingOperations]] necessary to create render nodes starting from the given high level model.
[img[Entities cooperating in the Builder|uml/fig129285.png]]
!Colaborations
While building, the application of such a visiting tool (especially the [[NodeCreatorTool|PlanningNodeCreatorTool]]) is embedded into an execution context formed by the BuilderToolFactory providing our BuilderToolKit, the [[Assembler|BuilderAssembler]] and [[connection manager|ConManager]]. The colaboration of these parts can be seen as the [[mechanics of the builder|BuilderMechanics]] — sort of the //outward view//, contrary to the //invard aspects// visible when focussing on how the nodes are put together.
[img[Colaborations in the Build Process|uml/fig128517.png]]
Besides the primary working tool within the builder (namely the [[Node Creator Tool|PlanningNodeCreatorTool]]), on a lower level, we encounter several [[elementary building situations|BuilderPrimitives]] — and for each of these elementary situations we can retrieve a suitable "fitting tool" or [[mould|BuilderMould]]. The palette of these moulds is called the ''tool kit'' of the builder. It is subject to configuration by rules.
!!addressing a mould
All mould instances are owned and managed by the [[tool factory|BuilderToolFactory]], and can be referred to by their type (PipeMould, CombiningMould, SourceChainMould, WiringMould) and a concrete object instance (of suitable type). The returned mould (instance) acts as a handle to stick together the given object instance (from the high-level model) with the corresponding point in the low-level node network under construction. As consequence of this approach, the tool factory instance holds a snapshot of the current building state, including all the active spots in the build process. As the latter is driven by objects from the high-level model appearing (in a sensible order → see BuilderMechanics) within the NodeCreatorTool, new moulds will be created and fitted as necessary, and existing moulds will be exhausted when finished, until the render node network is complete.
!!configuring a mould
As each mould kind is different, it has a {{{prepare(...)}}} function with suitably typed parameters. The rest is intended to be self-configuring (for example, a ~CombiningMould will detect the actual kind of Transition and select the internal mode of operation), so that it's sufficient to just call {{{operate()}}}
!!sequence of operations
When {{{operate()}}} doesn't throw, the result is a list of //successor moulds// — you shouldn't use the original mould after triggering its operation, because it may have been retracted as a result and reused for another purpose by the tool factory. It is not necessary to store these resulting moulds either (as they can be retrieved as described above), but they can be used right away for the next building step if applicable. In the state they are returned from a successful building step (mould operation = execution of a contained [[processing pattern|ProcPatt]]), they are usually already holding a reference to the part of the network just created and need to be configured only with the next high-level object (effect, placement, pipe, processing pattern or similar, depending on the concrete situation) in order to carry out the next step.
!!single connection step
at the lowest level within the builder there is the step of building a //connection.// This step is executed by the processing pattern with the help of the mould. Actually, making such a connection is more complicated, because in the standard case it will connect N media streams simultaneously (N=2 for stereo sound or 3D video, N=6 for 5.1 Surround, N=9 for 2nd order Ambisonics). These details are encapsulated within the OperationPoint, which is provided by the mould and exhibits a common interface for the processing pattern to express the connecting operation.
→see also: BuilderPrimitives for the elementary working situations corresponding to each of these [[builder moulds|BuilderMould]]
''Bus-~MObjects'' create a scope and act as attachment point for building up [[global pipes|GlobalPipe]] within each timeline. While [[Sequence]] is a frontend -- actually implemented by attaching a root-[[Track]] object -- for //each global pipe// a BusMO is attached as child scope of the [[binding object|BindingMO]], which in turn actualy implements either a timeline or a [[meta-clip|VirtualClip]].
* each global pipe corresponds to a bus object, which thus refers to the respective ~Pipe-ID
* bus objects may be nested, forming a //subgroup//
* the placement of a bus holds a WiringClaim, denoting that this bus //claims to be the corresponding pipe.//
* by default, a timeline is outfitted with one video and one sound master bus
Calculation stream is an organisational unit used at the interface level of the Lumiera engine.
Representing a //stream of calculations,// delivering generated data within //timing constraints,// it is used
*by the [[play process(es)|PlayProcess]] to define and control properties of the output generation
*at the engine backbone to feed the [[Scheduler]] with individual [[render jobs|RenderJob]] to implement this stream of calculations
Calculation stream objects are stateless, constant chunks of definition -- any altering of playback or rendering parameters just causes the respective descriptors to be superseeded. The presence of a CalcStream (being alive within the denoted time span) implies that using any of the associated jobs, dispatcher tables, node and wiring descriptors is safe
!lifecycle
Calculation stream descriptors can be default constructed, representing a //void calculation.// You can't do anything with these.
Any really interesting calculation stream needs to be retrieved from the EngineFaçade. Additionally, an existing calculation stream can be chained up or superseded, yielding a new CalcStream based on the parameters of the existing one, possibly with some alterations.
!purpose
When a calculation stream is retrieved from the EngineFaçade it is already registered and attached there and represents an ongoing activity. Under the hood, several further collaborators will hold a copy of that calculation stream descriptor. While, as such, a CalcStream has no explicit state, at any time it //represents a current state.// In case the running time span of that stream is limited, it becomes superseded automatically, just by the passing of time.
Each calculation stream refers a relevant [[frame dispatcher table|FrameDispatcher]]. Thus, for the engine (interface level), the calculation stream allows to produce the individual [[render jobs|RenderJob]] to enqueue with the [[Scheduler]]. This translation step is what links and relates nominal time with running wall clock time, thereby obeying the [[timing constraints|Timings]] given while initially defining the calculation stream.
Additionally, each calculation stream knows how to access a //render environment closure,// allowing to re-schedule and re-adjust the setup of this stream. Basically, this closure is comprised of several functors (callbacks), which could be invoked to perform management tasks later on. Amongst others, this allows the calculation stream to redefine, supersede or "cancel itself", without the need to access a central registration table at the engine interface level.
Background: #fefefd
Foreground: #000
PrimaryPale: #8fb
PrimaryLight: #4dc9a7
PrimaryMid: #16877a
PrimaryDark: #0f3f56
SecondaryPale: #ffc
SecondaryLight: #fe8
SecondaryMid: #db4
SecondaryDark: #841
TertiaryPale: #eef
TertiaryLight: #ccd
TertiaryMid: #99a
TertiaryDark: #667
Error: #f88
Within Proc-Layer, a Command is the abstract representation of a single operation or a compound of operations mutating the HighLevelModel.
Thus, each command is a ''Functor'' and a ''Closure'' ([[command pattern|http://en.wikipedia.org/wiki/Command_pattern]]), allowing commands to be treated uniformly, enqueued in a [[dispatcher|ProcDispatcher]], logged to the SessionStorage and registered with the UndoManager.
Commands are //defined// using a [[fluent API|http://en.wikipedia.org/wiki/Fluent_interface]], just by providing apropriate functions. Additionally, the Closure necessary for executing a command is built by binding to a set of concrete parameters. After reaching this point, the state of the internal representation could be serialised by plain-C function calls, which is important for integration with the SessionStorage.
→ see CommandDefinition
→ see CommandHandling
→ see CommandLifecycle
→ see CommandUsage
Commands can be identified and accessed //by name// — consequently there needs to be an internal command registry, including a link to the actual implementing function, thus allowing to re-establish the connection between command and implementing functions when de-serialising a persisted command. To create a command, we need to provide the following information
* operation function actually implementing the command
* function to [[undo|UndoManager]] the effect of the command
* function to capture state to be used by UNDO.
* a set of actual parameters to bind into these functions (closure).
!Command definition object
The process of creating a command by providing these building blocks is governed by a ~CommandDef helper object. According to the [[fluent definition style|http://en.wikipedia.org/wiki/Fluent_interface]], the user is expected to invoke a chain of definition functions, finally leading to the internal registration of the completed command object, which then might be dispatched or persisted. For example
{{{
CommandDefinition ("test.command1")
.operation (command1::operate) // provide the function to be executed as command
.captureUndo (command1::capture) // provide the function capturing Undo state
.undoOperation (command1::undoIt) // provide the function which might undo the command
.bind (obj, val1,val2) // bind to the actual command parameters (stores command internally)
.executeSync(); // convenience call, forwarding the Command to dispatch.
}}}
!Operation parameters
While generally there is //no limitation// on the number and type of parameters, the set of implementing functions and the {{{bind(...)}}} call are required to match. Inconsistencies will be detected by the compiler. In addition to taking the //same parameters as the command operation,// the {{{captureUndo()}}} function is required to return (by value) a //memento// type, which, in case of invoking the {{{undo()}}}-function, will be provided as additional parameter. To summarise:
|!Function|>|!ret(params)|
| operation| void |(P1,..PN)|
| captureUndo| MEM |(P1,..PN)|
| undoOperation| void |(P1,..PN,MEM)|
| bind| void |(P1,..PN)|
Usually, parameters should be passed //by value// — with the exception of target object(s), which are typically bound as MObjectRef, causing them to be resolved at commad execution time (late binding).
Organising any ''mutating'' operations executable by the user (via GUI) by means of the [[command pattern|http://en.wikipedia.org/wiki/Command_pattern]] can be considered //state of the art// today. First of all, it allows to discern the specific implementation operations to be called on one or several objects within the HighLevelModel from the operation requested by the user, the latter being rather a concept. A command can be labeled clearly, executed under controlled circumstances, allowing transactional behaviour.
!Defining commands
[>img[Structure of Commands|uml/fig134021.png]] Basically, a command could contain arbitrary operations, but we'll assume that it causes a well defined mutation within the HighLevelModel, which can be ''undone''. Thus, when defining (→[[syntax|CommandDefinition]]) a command, we need to specify not only a function doing the mutation, but also another function which might be called later to reverse the effect of the action. Besides, the action operates on a number of ''target'' objects and additionally may require a set of ''parameter'' values. These are to be stored explicitly within the command object, thus creating a ''closure'' — the operation //must not// rely on other hidden parameters (maybe with the exception of very generic singleton system services?).
Theoretically, defining the "undo" operation might utilise two different approaches:
* specifying an //inverse operation,// known to cancel out the effect of the command
* capturing a //state memento,// which can later be played back to restore the state found prior to executing the command.
While obviously the first solution is much simpler to implement on behalf of the command framework, the second solution has distinct advantages, especially in the context of an editing application: there might be rounding or calculation errors, the inverse might be difficult to define correctly, the effect of the operation might depend on circumstances, be random, or might even trigger a resolution operation to yield the final result. Thus the decision is within Lumiera Proc-Layer to //favour state capturing// — but in a modified, semi-manual and not completely exclusive way.
!Undo state
While the usual »Memento« implementation might automatically capture the whole model (resulting in a lot of data to be stored and some uncertainty about the scope of the model to be captured), in Lumiera we rely instead on the client code to provide a ''capture function'' and a ''playback function'' alongside with the actual operation. To help with this task, we provide a set of standard handlers for common situations. This way, operations might capture very specific information, might provide an "intelligent undo" to restore a given semantic instead of just a fixed value — and moreover the client code is free actually to employ the "inverse operation" model in special cases where it just makes more sense than capturing state.
!Handling of commands
A command may be [[defined|CommandDefinition]] completely from scratch, or it might just serve as a CommandPrototype with specific targets and parameters. The command could then be serialised and later be recovered and re-bound with the parameters, but usually it will be handed over to the ProcDispatcher, pending execution. When ''invoking'', the handling sequence is to [[log the command|SessionStorage]], then call the ''undo capture function'', followed from calling the actual ''operation function''. After success, the logging and [[undo registration|UndoManager]] is completed. In any case, finally the ''result signal'' (a functor previously stored within the command) is emitted. {{red{10/09 WIP: not clear if we acutally implement this concept}}}
By design, commands are single-serving value objects; executing an operation repeatedly requires creating a collection of command objects, one for each invocation. While nothing prevents you from invoking the command operation functor several times, each invocation will overwrite the undo state captrued by the previous invocation. Thus, each command instance should bes seen as the promise (or later the trace) of a single operation execution. In a similar vein, the undo capturing should be defined as to be self sufficient, so that invoking just the undo functor of a single command performes any necessary steps to restore the situation found before invoking the corresponding mutation functor — of course only //with respect to the topic covered by this command.// So, while commands provide a lot of flexibility and allow to do a multitude of things, certainly there is an intended CommandLifecycle.
→ command [[definition|CommandDefinition]] and [[-lifecycle|CommandLifecycle]]
→ more on possible [[command usage scenarios|CommandUsage]]
→ more details regarding [[command implementation|CommandImpl]]
Commands are separated in a handle (the {{{control::Command}}}-object), to be used by the client code, and an implementation level, which is managed transparently behind the stages. Client code is assumed to build a CommandDefinition at some point, and from then on to access the command ''by ID'', yielding the command handle.
Binding of arguments, invocation and UNDO all are accessible through this frontend.
!Infrastructure
To support this handling scheme, some infrastructure is in place:
* a command registry maintains the ID ↔ Command relation.
* indirectly, through a custom alloctaor, the registry is also involved into allocation of the command implementation frame
* this implementation frame combines
** an operation mutation and an undo mutation
** a closure, implemented through an argument holder
** an undo state capturing mechanism, based on a capturing function provided on definition
* performing the actual execution is delegated to a handling pattern object, accessed by name.
[<img[Structure of Commands|uml/fig135173.png]]
While generally the command framework was designed to be flexible and allow a lot of different use cases, execution paths and to serve various goals, there is an ''intended lifecycle'' — commands are expected to go through several distinct states.
The handling of a command starts out with a ''command ID'' provided by the client code. Command ~IDs are unique (human readable) identifiers and should be organised in a hierarchical fashion. When provided with an ID, the CommandRegistry tries to fetch an existing command definition. In case this fails, we enter the [[command definition stage|CommandDefinition]], which includes specifying functions to implement the operation, state capturing and UNDO. When all of this information is available, the entity is called a ''command definition''. Conceptually, it is comparable to a //class// or //meta object.//
By ''binding'' to specific operation arguments, the definition is //armed up// and becomes a real ''command''. This is similar to creating an instance from a class. Behind the scenes, storage is allocated to hold the argument values and any state captured to create the ability to UNDO the command's effect later on.
A command is operated or executed by passing it to an ''execution pattern'' — there is a multitude of possible execution patterns to choose from, depending on the situation.
{{red{WIP... details of ~ProcDispatcher not specified yet}}}
When a command has been executed (and maybe undone), it's best to leave it alone, because the UndoManager might hold a reference. Anyway, a ''clone of the command'' could be created, maybe bound with different arguments and treated separately from the original command.
!State predicates
* fetching an non-existent command raises an ~LUMIERA_ERROR_INVALID_COMMAND
* a command definition becomes //valid// ({{{bool true}}}) when all necessary functions are specified. Technically this coincides with the creation of a CommandImpl frame behind the scenes, which also causes the Command (frontend/handle object) to evaluate to {{{true}}} in bool context from then on.
* when, in addition to the above, the command arguments are bound, it becomes //executable.//
* after the (fist) execution, the command gets also //undo-able.//
State predicates are accessible through the Command (frontend); additionally there are static query functions in class {{{Command}}}
//for now (7/09) I'll use this page to collect ideas how commands might be used...//
* use a command for getting a log entry and an undo possibility automatically
* might define, bind and then execute a command at once
* might define it and bind it to a standard set of parameters, to be used as a prototype later.
* might just create the definition, leaving the argument binding to the actual call site
* execute it and check the success/failure result
* just enqueue it, without caring for successful execution
* place it into a command sequence bundle
* repeat the execution
!!!a command definition....
* can be created from scratch, by ID
* can be re-accessed, by ID
* can't be modified once defined (this is to prevent duplicate definitions with the same ID)
* but can be dropped, which doesn't influence already existing dependent command instances
* usually will be the starting point for creating an actual command by //binding//
!!!a command instance....
* normally emerges from a definition by binding arguments
* the first such binding will create a named command registration
* but subsequent accesses by ID or new bindings create an anonymous clone
* which then in turn might then be registered explicitly with a new ID
* anonymous command instances are managed by referral and ref-counting
!!!an execution pattern....
* can only be defined in code by a class definition, not at runtime
* subclasses the ~HandlingPattern interface and uses an predefined ID (enum).
* a singleton instance is created on demand, triggered by referring the pattern's ID
* is conceptually //stateless// — of course there can be common configuration values
* is always invoked providing a concrete command instance to execute
* is configured into the command instance, to implement the command's invocation
* returns a duck-typed //result// object
The Connection Manager is a service for wiring connections and for querying information and deriving decisions regarding various aspects of data streams and the possibility of connections. The purpose of the Connection Manager is to isolate the [[Builder]], which is client of this information and decision services, from the often convoluted details of type information and organizing a connection.
!control connections
my intention was that it would be sufficient for the builder to pass an connection request, and the Connection Manager will handle the details of establishing a control/parameter link.
{{red{TODO: handling of parameter values, automation and control connections still need to be designed}}}
!data connections
Connecting data streams of differing type involves a StreamConversion. Mostly, this aspect is covered by the [[stream type system|StreamType]]. The intended implementation will rely partially on [[rules|ConfigRules]] to define automated conversions, while other parts need to be provided by hard wired logic. Thus, regarding data connections, the ConManager can be seen as a specialized Facade and will delegate to the → [[stream type manager|STypeManager]]
* retrieve information about capabilities of a stream type given by ID
* decide if a connection is possible
* retrieve a //strategy// for implementing a connection
This index refers to the conceptual, more abstract and formally specified aspects of the Proc-Layer and Lumiera in general.
More often than not, these emerge from immediate solutions, being percieved as especially expressive, when taken on, yielding guidance by themselves. Some others, [[Placements|Placement]] and [[Advice]] to mention here, immediately substantiate the original vision.
Configuration Queries are requests to the system to "create or retrieve an object with //this and that // capabilities". They are resolved by a rule based system ({{red{planned feature}}}) and the user can extend the used rules for each Session. Syntactically, they are stated in ''prolog'' syntax as a conjunction (=logical and) of ''predicates'', for example {{{stream(mpeg), pipe(myPipe)}}}. Queries are typed to the kind of expected result object: {{{Query<Pipe> ("stream(mpeg)")}}} requests a pipe excepting/delivering mpeg stream data — and it depends on the current configuration what "mpeg" means. If there is any stream data producing component in the system, which advertises to deliver {{{stream(mpeg)}}}, and a pipe can be configured or connected with this component, then the [[defaults manager|DefaultsManagement]] will create/deliver a [[Pipe|PipeHandling]] object configured accordingly.
→ [[Configuration Rules system|ConfigRules]]
→ [[accessing and integrating configuration queries|ConfigQueryIntegration]]
* planning to embed a YAP Prolog engine
* currently just integrated by a table driven mock
* the baseline is a bit more clear by now (4/08)
→ see also ConfigRules
→ see also DefaultsManagement
!Use cases
[<img[when to run config queries|uml/fig131717.png]]
The key idea is that there is a Rule Base — partly contained in the session (building on a stock of standard rules supplied with the application). Now, whenever there is the need to get a new object, for adding it to the session or for use associated with another object — then instead of creating it by a direct hard wired ctor call, we issue a ConfigQuery requesting an object of the given type with some //capabilities// defined by predicates. The same holds true when loading an existing session: some objects won't be loaded back blindly, rather they will be re-created by issuing the config queries again. Especially an important use case is (re)creating a [[processing pattern|ProcPatt]] to guide the wiring of a given media's processing pipeline.
At various places, instead of requiring a fixed set of capabilities, it is possible to request a "default configured" object instead, specifying just those capabilities we really need to be configured in a specific way. This is done by using the [[Defaults Manager|DefaultsManagement]] accessible on the [[Session]] interface. Such a default object query may either retrieve an already existing object instance, run further config queries, and finally result in the invocation of a factory for creating new objects — guarded by rules to suit current necessities.
@@clear(left):display(block):@@
!Components and Relations
[>img[participating classes|uml/fig131461.png]]
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
Access point is the interface {{{ConfigRules}}}, which allowes to resolve a ConfigQuery resulting in an object with properties configured such as to fulfill the query. This whole subsystem employes quite some generic programming, because actually we don't deal with "objects", but rather with similar instantiations of the same functionality for a collection of different object types. For the purpose of resolving these queries, the actual kind of object is not so much of importance, but on the caller side, of course we want to deal with the result of the queries in a typesafe manner.
Examples for //participating object kinds// are [[pipes|Pipe]], [[processing patterns|ProcPatt]], effect instances, [[tags|Tag]], [[labels|Label]], [[automation data sets|AutomationData]],...
@@clear(right):display(block):@@
For this to work, we need each of the //participating object types// to provide the implementation of a generic interface {{{TypeHandler}}}, which allows to access actual C/C++ implementations for the predicates usable on objects of this type within the Prolog rules. The implementation has to make sure that, alongside with each config query, there are additional //type constraints// to be regarded. For example, if the client code runs a {{{Query<Pipe>}}}, an additional //type guard// (implemented by a predicate {{{type(pipe)}}} has to be inserted, so only rules and facts in accordance with this type will be used for resolution.
!when querying for a [["default"|DefaultsManagement]] object
[<img[colaboration when issuing a defaults query|uml/fig131845.png]]
@@clear(left):display(block):@@
Many features can be implemented by specifically configuring and wiring some unspecific components. Rather than tie the client code in need of some given feature to these configuration internals, in Lumiera the client can //query // for some kind of object providing the //needed capabilities. // Right from start (summer 2007), Ichthyo had the intention to implement such a feature using sort of a ''declarative database'', e.g. by embedding a Prolog system. By adding rules to the basic session configuration, users should be able to customize the semi-automatic part of Lumiera's behaviour to great extent.
[[Configuration Queries|ConfigQuery]] are used at various places, when creating and adding new objects, as well when building or optimizing the render engine node network.
* Creating a [[pipe|PipeHandling]] queries for a default pipe or a pipe with a certain stream type
* Adding a new [[track|TrackHandling]] queries for some default placement configuration, e.g. the pipe it will be plugged to.
* when processing a [[wiring request|WiringRequest]], connection possibilities have to be evaluated.
* actually building such a connection may create additional degrees of freedom, like panning for sound or layering for video.
!anatomy of a Configuration Query
The query is given as a number of logic predicates, which are required to be true. Syntactically, it is a string in prolog syntax, e.g. {{{stream(mpeg)}}}, where "stream" is the //predicate, // meaning here "the stream type is...?" and "mpeg" is a //term // denoting an actual property, object, thing, number etc — the actual kind of stream in the given example. Multible comma separated predicates are combined with logical "and". Terms may be //variable // at start, which is denoted syntactically by starting them with a uppercase letter. But any variable term need to be //bound // to some known value while computing the solution to the query, otherwise the query fails. A failed query is treated as a local failure, which may cause some operation being aborted or just some other possibility being chosen.
Queries are represented by instantiations of the {{{Query<TYPE>}}} template, because their actual meaning is "retrieve or create an object of TYPE, configured such that...!". At the C++ side, this ensures type safety and fosters programming against interfaces, while being implemented rule-wise by silently prepending the query with the predicate "{{{object(TYPE)}}}"
!!!querying for default
A special kind of configuration query includes the predicate {{{default(X)}}}, which is kind-of an existential quantisation of the variable {{{X}}}. Using this predicate states that a suitable //default-configured// object exists somehow. This behaviour could be used as an fallback within a config query, allowing us always to return a solution. The latter can be crucial when it comes to integrating the query subsystem into an existing piece of implementation logic, which requires us to give some guarantees.
→ see DefaultsManagement on details how to access these captured defaults
But note, the exact relation of configuration queries and query-for-default still needs to be worked out, especially when to use which flavour. {{red{WIP 10/2010 ⇒ see Ticket [[#705|http://issues.lumiera.org/ticket/705]]}}}
!executing a Configuration Query
Actually posing such an configuration query, for example to the [[Defaults Manager in the Sessison|DefaultsManagement]], may trigger several actions: First it is checked against internal object registries (depending on the target object type), which may cause the delivery of an already existing object (as reference, clone, or smart pointer). Otherwise, the system tries to figure out an viable configuration for a newly created object instance, possibly by issuing recursive queries. In the most general case this may silently impose additional decisions onto the //execution context // of the query — by default the session.
!Implementation
At start and for debugging/testing, there is an ''dummy'' implementation using a map with predefined queries and answers. But for the real system, the idea is to embed a ''YAP Prolog'' engine to run the queries. This includes the task of defining and loading a set of custom predicates, so the rule system can interact with the object oriented execution environment, for example by transforming some capability predicate into virtual calls to a corresponding object interface. We need a way for objects to declare some capability predicates, together with a functor that can be executed on an object instance (and further parameters) in the cause of the evaluation of some configuration query. Type safety and diagnostics play an important role here, because effectively the rule base is a pool of code open for arbitray additions from the user session.
→ [[considerations for a Prolog based implementation|QueryImplProlog]]
→ [[accessing and integrating configuration queries|ConfigQueryIntegration]]
→ see {{{src/common/query/mockconfigrules.cpp}}} for the table with the hard wired (mock) answers
!!!fake implementation guidelines
The fake implementation should follow the general pattern planned for the Prolog implementation. That is, find, query, create, query. Thus, such a config query can be considered existentially quantised. The interplay with the factories is tricky. Because factories might issue config queries themselves, while a factory invocation driven from within this rule pattern //must not// re-invoke the factory. Besides, a factory invocation should create, not fetch an existing solution (?)
{{red{WARN}}} there is an interference with the (planned) Undo function: a totally correct implementation of "Undo" would need to remember and restore the internal state of the query system (similar to backtracking). But, more generally, such an correct implementation is not achievable, because we are never able to capture and control the whole state of a real world system doing such advanced things like video and sound processing. Seemingly we have to accept that after undoing an action, there is no guarantee we can re-do it the same way as it was first time.
Here, in the context of the Render Engine, the Controller component is responsible for triggering and coordinating the build process and for activating the backend and the Render Engine configuration created by the Builder to carry out the actual rendering. There is another Controller in the backend, the ~PlaybackController, which is in charge of the playback/rendering/display state of the application, and consequently will call this (Proc-Layer) Controller to get the necessary Render Engine.
!Facade
This is an very important external Interface, because it links together all three Layers of our current architecture. It can be used by the backend to initiate [[Render Processes (=StateProxy)|StateProxy]] and it will probably be used by the Dispatcher for GUI actions as well...
The question is where to put all the state-like information [[associated with the current session|SessionOverview]]. Because this is certainly "global", but may depend on the session or need to be configured differently when loading another session. At the moment (9/07) Ichthyo considers the following solution:
* represent all configuration as [[Asset]]s
* find a way {{red{TODO}}} how to reload the contents of the [[AssetManager]].
* completely hide the Session object behind a ''~PImpl'' smart pointer, so the session object can be switched when reloading.
* the [[Fixture]] acts as isolation layer, and all objects refered from the Fixture are refcounting smart pointers. So, even when the session gets switched, the old objects remain valid as long as needed.
A ''frame of data'' is the central low-level abstraction when dealing with media data and media processing.
As detailed in the [[definition|DefaultsManagement]], {{{default(Obj)}}} is sort of a Joker along the lines "give me a suitable Object and I don't care for further details". Actually, default objects are implemented by the {{{mobject::session::DefsManager}}}, which remembers and keeps track of anything labeled as "default". This defaults manager is a singleton and can be accessed via the [[Session]] interface, meaning that the memory track regarding defaults is part of the session state. Accessing an object via the query for an default actually //tagges// this object (storing a weak ref in the ~DefsManager). Alongside with each object successfully queried via "default", the degree of constriction is remembered, i.e. the number of additional conditions contained in the query. This enables us to search for default objects starting with the most unspecific.
!Skeleton
# ''search'': using the predicate {{{default(X)}}} enumerates existing objects of suitable type
#* candidates are delivered starting with the least constrained default
#* the argument is unified
#** if the rest of the query succeeds we've found our //default object// and are happy.
#** otherwise, if all enumerated solutions are exhausted without success, we enter
# ''default creation'': try to get an object fulfilling the conditions and remember this situation
#* we issue an ConfigQuery with the query terms //minus// the {{{default(X)}}} predicate
#* it depends on the circumstances how this query is handled. Typically the query resolution first searches existing objects and then creates a new instance to match the required capabilities. Usually, this process succeeds, but there can be configurations leading to failure.
#** failing the ~ConfigQuery is considered an (non-critical) exception (throws), as defaults queries are supposed to succeed
#** otherwise, the newly created object is remembered (tagged) as new default, together with the degree of constriction
!!!Implementation details
Taken precisely, the "degree of constriction" yields only a partial ordering — but as the "default"-predicate is sort of a existential quantification anyway, its sole purpose is to avoid polluting the session with unnecessary default objects, and we don't need to care for absolute precision. A suitable approximation is to count the number of predicates terms in the query and use a (sorted) set (separate for each Type) to store weak refs to the the objects tagged as "default"
{{red{WARN}}} there is an interference with the (planned) Undo function. This is a general problem of the config queries; just ignoring this issue seems reasonable.
!!!Problems with the (preliminary) mock implementation
As we don't have a Prolog interpreter on board yet, we utilize a mock store with preconfigured answers. (see MockConfigQuery). As this preliminary solution is lacking the ability to create new objects, we need to resort to some trickery here (please look away). The overall logic is quite broken, not to say outright idiotic, because the system isn't capable to do any real resolution — if we ignore this fact, the rest of the algorithm can be implemented, tested and used right now.
For several components and properties there is an implicit default value or configuration; it is stored alongside with the session. The intention is that defaults never create an error, instead, they are to be extended silently on demand. Objects configured according to these defaults can be retrieved at the [[Session]] interface by a set of overloaded functions {{{Session::current->default(Query<TYPE> ("query string"))}}}, where the //query string // defines a capability query similar to what is employed for pipes, stream types, codecs etc. This query mechanism is implemented by [[configuration rules|ConfigRules]]
!!!!what is denoted by {{{default}}}?
{{{default(Obj)}}} is a predicate expressing that the object {{{Obj}}} can be considered the default setup under the given conditions. Using the //default// can be considered as a shortcut for actually finding a exact and unique solution. The latter would require to specify all sorts of detailed properties up to the point where only one single object can satisfy all conditions. On the other hand, leaving some properties unspecified would yield a set of solutions (and the user code issuing the query had to provide means for selecting one soltution from this set). Just falling back on the //default// means that the user code actually doesn't care for any additional properties (as long as the properties he //does// care for are satisfied). Nothing is said specifically on //how// this default gets configured; actually there can be rules //somewhere,// and, additionally, anything encountered once while asking for a default can be re-used as default under similar circumstances.
→ [[implementing defaults|DefaultsImplementation]]
Along the way of working out various [[implementation details|ImplementationDetails]], decisions need to be made on how to understand the different facilities and entities and how to tackle some of the problems. This page is mainly a collection of keywords, summaries and links to further the discussion. And the various decisions should allways be read as proposals to solve some problem at hand...
''Everything is an object'' — yes of course, that's a //no-brainer,// todays. Rather, important is to note what is not "an object", meaning it can't be arranged arbitrarily
* we have one and only one global [[Session]] which directly contains a collection of multiple [[Timelines|Timeline]] and is associated with a globally managed collection of [[assets|Asset]]. We don't utilise scoped variables here (no "mandantisation"); if a media has been //opened,// it is just plain //globally known// as asset.
* the [[knowledge base|ConfigRules]] is just available globally. Obviously, the session gets a chance to install rules into this knowledge base, but we don't stress ownership here.
* we have a [[Fixture]] which acts as isolation layer towards the render engine and is (re)built automatically.
The high-level view of the tangible entities within the session is unified into a ''single tree'' -- with the notable exception of [[external outputs|OutputManagement]] and the [[assets|Asset]], which are understood as representing a //bookkeeping view// and kept separate from the //things to be manipulated// (MObjects).
We ''separate'' processing (rendering) and configuration (building). The [[Builder]] creates a network of [[render nodes|ProcNode]], to be processed by //pulling data // from some [[Pipe]]
''Objects are [[placed|Placement]] rather'' than assembled, connected, wired, attached. This is more of a rule-based approach and gives us one central metaphor and abstraction, allowing us to treat everything in an uniform manner. You can place it as you like, and the builder tries to make sense out of it, silently disabling what doesn't make sense.
An [[Sequence]] is just a collection of configured and placed objects (and has no additional, fixed structure). [[Tracks|Track]] form a mere organisational grid, they are grouping devices not first-class entities (a track doesn't "have" a pipe or "is" a video track and the like; it can be configured to behave in such manner by using placements though). [[Pipes|Pipe]] are hooks for making connections and are the only facility to build processing chains. We have global pipes, and each clip is built around a lokal [[source port|ClipSourcePort]] — and that's all. No special "media viewer" and "arranger", no special role for media sources, no commitment to some fixed media stream types (video and audio). All of this is sort of pushed down to be configuration, represented as asset of some kind. For example, we have [[processing pattern|ProcPatt]] assets to represent the way of building the source network for reading from some media file (including codecs treated like effect plugin nodes)
Actual ''media data and handling'' is abstracted rigorously. Media is conceived as being stream-like data of distinct StreamType. When it comes to more low-level media handling, we build on the DataFrame abstraction. Media processing isn't the focus of Lumiera; we organise the processing but otherwise ''rely on media handling libraries.'' In a similar vein, multiplicity is understood as type variation. Consequently, we don't build an audio and video "section" and we don't even have audio tracks and video tracks. Lumiera uses tracks and clips, and clips build on media, but we're able to deal with multichannel mixed-typed media.
Lumiera is not a connection manager, it is not an audio-visual real time instrument, and it doesn't aim at running presentations. It's an ''environment for assembling and building up'' something (an edit, a session, a piece of media work). This decision is visible at various levels and contexts, like a reserved attitude towards hardware acceleration (it //will// be supported, but reliable proxy editing has a higher priority), or the decision, not to incorporate system level ports and connections directly into the session model (they are mapped to [[output designations|OutputDesignation]] rather)
''State'' is rigorously ''externalized'' and operations are to be ''scheduled'', to simplify locking and error handling. State is either treated similar to media stream data (as addressable and cacheable data frame), or is represented as "parameter" to be served by some [[parameter provider|ParamProvider]]. Consequently, [[Automation]] is just another kind of parameter, i.e. a function — how this function is calculated is an encapsulated implementation detail (we don't have "bezier automation", and then maybe a "linear automation", a "mask automation" and yet another way to handle transitions)
Deliberately there is an limitaion on the flexibility of what can be added to the system via Plugins. We allow configuration and parametrisation to be extended and we allow processing and data handling to be extended, but we disallow extensions to the fundamental structure of the system by plugins. They may provide new implementations for already known subsystems, but they can't introduce new subsystems not envisioned in the general design of the application.
* thus fixed assortments include: the possbile kinds of MObject and [[Asset]], the possible automation parameter data types, the supported kinds of plugin systems
* while plugins may extend: the supported kinds of media, the [[media handling libraries|MediaImplLib]] used to deal with those media, the session storage backends, the behaviour of placments
This __proc-Layer__ and ~Render-Engine implementation started out as a design-draft by [[Ichthyo|mailto:Ichthyostega@web.de]] in summer 2007. The key idea of this design-draft is to use the [[Builder Pattern|http://en.wikipedia.org/wiki/Builder_pattern]] for the Render Engine, thus separating completely the //building// of the Render Pipeline from //running,// i.e. doing the actual Render. The Nodes in this Pipeline should process Video/Audio and do nothing else. No more decisions, tests and conditional operations when running the Pipeline. Move all of this out into the configuration of the pipeline, which is done by the Builder.
!Why doesn't the current Cinelerra-2 Design succeed?
The design of Cinelerra-2 basically follows a similar design, but __fails because of two reasons__
# too much differentiation is put into the class hierarchy instead of configuring Instances differently.<br>This causes overly heavy use of virtual functions and -- in order to ameliorate this -- falling back to hard wired branching
# far too much coupling and back-coupling to the internals of the »EDL«, forcing a overly rigid structure on the latter
!Try to learn from the Problems of the current Cinelerra-2 codebase
* build up an [[Node Abstraction|ProcNode]] powerful enough to express //all necessary Operations// without the need to recure on the actual object type
* need to redesign the internals of the Session in a far more open manner. Everything is an [[M-Object|MObjects]] which is [[placed|Placement]] in some manner
* strive at a StrongSeparation between Session and Render Engine, encapsulate and allways implement against interfaces
!Design Goals
As always, the main goal is //to cut down complexity// by the usual approach to separate into small manageable chunks.
To achieve this, here we try to separate ''Configuration'' from ''Processing''. Further, in Configuration we try to separate the ''high level view'' (users view when editing) from the ''low level view'' (the actual configuration effective for the calculations). Finally, we try to factor out and encapsulate ''State'' in order to make State explicit.
The main tool used to implement this separation is the [[Builder Pattern|http://en.wikipedia.org/wiki/Builder_pattern]]. Here especially we move all decisions and parametrization into the BuildProcess. The Nodes in the render pipeline should process Video/Audio and do nothing else. All decisions, tests and conditional operations are factored out of the render process and handled as configuration of the pipeline, which is done by the Builder. The actual color model and number of ports is configured in by a pre-built wiring descriptor. All Nodes are of equal footing with each other, able to be connected freely within the limitations of the necessary input and output. OpenGL and renderfarm support can be configured in as an alternate implementation of some operations together with an alternate signal flow (usable only if the whole Pipeline can be built up to support this changed signal flow), thus factoring out all the complexities of managing the data flow between core and hardware accelerated rendering. We introduce separate control data connections for the [[automation data|Automation]], separating the case of true multi-channel-effects from the case where one node just gets remote controlled by another node (or the case of two nodes using the same automation data).
Another pertinent theme is to make the basic building blocks simpler, while on the other hand gaining much more flexibility for combining these building blocks. For example we try to unfold any "internal-multi" effects into separate instances (e.g. the possibility of having an arbitrary number of single masks at any point of the pipeline instead of having one special masking facility encompassing multiple sub-masks. Similarly, we treat the Objects in the Session in a more uniform manner and gain the possibility to [[place|Placement]] them in various ways.
//Currently (5/2011) this page is used to collect and build up a coherent design for the player subsystem of Lumiera..//
!Starting point
The intention is to start out with the design of the PlayerDummy and to //transform// it into the full player subsystem.
* the ~DisplayService in that dummy player design moves down into Proc-Layer and becomes the OutputManager
* likewise, the ~DisplayerSlot is transformed into the interface OutputSlot, with various implementations to be registered with the OutputManager
* the core idea of having a play controler act as the frontend and handle to an PlayProcess is retained.
!!Idea
Introduce a new kind of MObject, a GeneratorMO, to represent what the current dummy player does: generate some kind of test data. As a first step, we might retro-fit the existing dummy player to this structure, and then expand it to have a minimal render engine setup for tests and diagnostics. That is, there will be some kind of GeneratorNode to produce test data.
!!Stages of transition
Reworking the dummy player into the [[Player subsystem|PlayService]] is a larger undertaking, and best broken up into several //stages.//
# just moving the existing entities into the final location (typically down in the layer hierarchy)
# introduce a GeneratorMO and a GeneratorNode, while providing a shortcut to get the GeneratorNode without requiring the (not yet implemented) Builder.
# rework the ~ProcessImpl and the ~TickService to become the [[calculation stream|CalcStream]] and use the real Engine Interface, but still with an mock implementation hooked up behind
# extend and elaborate the handling of output, build the foundation of the real OutputManager
# switch to the real Scheduler and Engine implementation
!!!!new entities to build {{red{#805}}}
* the ~DisplayService needs to be generalised into an OutputManager interface
* we need to introduce a new kind of StructAsset, the ViewerAsset.
* then, of course, the PlayService interface needs to be created
* as a temporary soltion, a ''~DummyPlayConnection'' will be created hard-wired, within the ~DummyPlayer service. This dumy container is a test setup, explicitly creating a OutputManager implementation, a GeneratorMO, a GeneratorNode (thus omitting the not-yet-implemented Builder) and the corresponding ModelPort. This way, the ~DummyPlayer service can be changed to use the real PlayService for creating the dummy generated output data. It will then pass back the resulting PlayController to the existing GUI setup.
The Method of pushing frames to the Viewer widget is changed fundamentally during that transition: while previously, in the PlayerDummy design study, the GUI opened a Display-façade interface -- now rather the GUI has to register an OutputSlot implementation with the global OutputManager. Such a registration requires an explicit deregistration on shutdown, and this can be made to block, until no further data will be delivered. Using this approach we hope to circumvent problems with occasional crashes on closing the application, due to dispatching frame output signals in the event thread while shutdown is already in progress.
LayerSeparationInterface provided by the GUI.
Access point especially for the playback. A render- or playback process uses the DisplayFacade to push media data up to the GUI for display within a viewer widget of full-screen display. This can be thought off as a callback mechanism. In order to use the DisplayFacade, client code needs a DisplayerSlot (handle), which needs to be set up by the UI first and will be provided when starting the render or playback process.
A service within the GUI to manage output of frames generated by the lower layers of the application.
*providing the actual implementation of the DisplayFacade
*creating and maintaining of [[displayer slots|DisplayerSlot]], connected to viewer widgets or similar
An output port wired up to some display facility or viewer widget within the UI. For the client code, each slot is represented by a handle, which can be used to lock into this slot for an continuous output process. Managed by the DisplayService
''EDL'' is a short-hand for __E__dit __D__ecision __L__ist. The use of this term can be confusing; for the usual meaning see the definition in [[Wikipedia|http://en.wikipedia.org/wiki/Edit_decision_list]]
Cinelerra uses this term in a related manner but with a somewhat shifted focus: In Cinelerra the EDL is comprised of the whole set of clips and other media objects arranged onto the tracks by the user. It is the result of the user's //editing efforts.//
In this usage, the EDL in most cases will be almost synonymous to »the session«, just the latter emphasizes more the state aspect. While the Lumiera project started out using the same terminology, later on, when support for multiple "containers" within the session and for [[meta-clips|VirtualClip]] was determined to be of much importance, the new term »[[Sequence]]« was preferred.
These are the tools provided to any client of the Proc layer for handling and manipulating the entities in the Session. When defining such operations, //the goal should be to arrive at some uniformity in the way things are done.// Ideally, when writing client code, one should be able to guess how to achieve some desired result.
!guiding principle
The approach taken to define any operation is based primarily on the ''~OO-way of doing things'': entities operate themselfs. You don't advise some manager, session or other »god class« to manipulate them. And, secondly, the scope of each operation will be as large as possible, but not larger. This often means performing quite some elementary operations — sometimes a convenience shortcut provided by the higher levels of the application may come in handy — and basically this gives rise to several different paths of doing the same thing, all of which need to be equivalent.
!main tasks
* you ''create a clip'' either from a source media or from another clip (copy, maybe clone?). The new clip always reflects the full size (and other properties) of the source used for creating.
* you can request a clip to ''resize'', which always relates to its current dimensions.
* you can ''place or attach'' the clip to some point or other entity by creating a [[Placement]] from the clip. (→ [[handling of Placements|PlacementHandling]])
* you can ''adjust'' a placement relative to some other placement, meta object (i.e. selection), label or media, and this may cause the placement to resize or even effectively remove the clip.
* you can perform ''adjustments'' on a Sequence as a whole.
All these operations propagate to directly dependant objects and may schedule global reconfigurations.
!state, locking, policies
While performing any manipulative task, the state of the object is considered inconsistent, but it is guaranteed to be consistent after any such operation, irrespective of the result. There is no automatic atomicity and, consequently each propagation is considered a separate operation.
!!parallelism
At the level of fine grained operations on individual entities, there is ''no support for parallelism''. Individual entities don't lock themselves. Tasks perform locking and are to be scheduled. Thus, we need an isolation layer towards all inherently multithreaded parts of the system, like the GUI or the renderengine.
This has some obvious and some subtle consequences. Of course, manipulating //tasks// will be queued and scheduled somewhere. But as we rely on object identity and reference semantics for the entities in the session, some value changes are visible to operations going on in parallel threads (e.g. rendering) and each operation on the session entities //has to be aware of this.//
Consequently, sometimes there needs to be done sort of a ''read-copy-update'', i.e. self-replacement by copy followed by manipulation of the new copy, while ongoing processes use the unaltered original object until they receive some sort of reset.
!!undo
Basically, each elementary operation has to record the information necessary to be undone. It does so by registering a Memento with some central UndoManager facility. This Memento object contains a functor pre-bound with the needed parameter values. (Besides, the UndoManager is free to implement a second level of security by taking independent state snapshots).
{{red{to be defined in more detail later...}}}
We have to deal with effects on various different levels. One thing is to decide if an effect is applicable, another question is to find out about variable and automatable parameters, find a module which can be used as an effect GUI and finally to access a processing function which can be executed on a buffer filled with suitable data.
The effect asset (which is a [[processing asset|ProcAsset]]) provides an unified interface for loading external processing modules and querying their properties. As usual with assets, this is the "bookkeeping view" to start with, but we can create a //processor// from such an asset, i.e. an instance of the processing facility which can be used and wired into the model. Such a processor is an MObject and can be placed into the session (high level model); moreover it holds a concrete wiring for the various control parameters and it has an active link to the effect GUI module. As every ~MObject, it could be placed multiple times to affect several pipes, but all those different Placements would behave as being linked together (with respect to the control parameter sources and the GUI)
When building the low-level model, the actual processing code is resolved and a processing function is installed into the processing node representing the effect. This step includes checking of actual [[media type information|StreamType]], which may result in selecting a specifically tailored implementation function. The parameter wiring on the other hand is //shared// between the effect ~MObject, the corresponding GUI module and all low-level processing nodes. Actually, parameters are more of a reference than actually being values: they provide a {{{getValue()}}} function, which also works with automation. This way, any parameter or automation changes are reflected immediately into the produced output.
Initially, only the parameter (descriptors) are present on the effect ~MObject, while the actual [[parameter providers|ParamProvider]] are created or wired (by the ConManager) on demand.
The primary interface used by the upper application layers to interact with the render engine, to create and manage ongoing [[calculation streams|CalcStream]].
Below this facade, there is a thin adaptadion and forwarding layer, mainly talking to
* the individual [[Calculation Streams|CalcStream]] created for each PlayProcess.
* the FrameDispatcher, which translates such streams into a series of RenderJob entries
* the [[Scheduler]], which is responsible to perform these jobs in a timely fashion
!Quality of Service
Within the Facade, there is the definition of the {{{EngineService::Quality}}} tag, alongside with several pre-defined quality settings.
Actually this interface is a strategy, allowing to define quite specific quality levels, in case we need that. Clients can usually just use
these ~QoS-tags like enum values (they are copyable), without caring for the engine implementation related details.
A general identification scheme, ombining a human readable symbolic name, unique within a //specifically typed context,// and machine readable hash ID (LUID). ~Entry-IDs allow for asset-like position accounting and for type safe binding between configuration rules and model obects. They allow for creating an entry with symbolic id and distinct type, combined with an derived hash value, without the overhead in storage and instance management imposed by using a full-fledged Asset.
Similar to an Asset, an identification tuple is available (generated on the fly), as is an unique LUID and total ordering. The type information is attached as template parameter, but included into the hash calculation. All instantiations of the EntryID template share a common baseclass, usable for type erased common registration.
~Entry-IDs as such do not provide any automatic registration or ensure uniqueness of any kind, but they can be used to that end. Especially, they're used within the TypedID registration framework for addressing individual entries {{red{planned as of 12/2010}}}
→ TypedLookup
→ TypeHandler
→ MetaAsset
The »Timeline« is a sequence of ~MObjects -- here clips -- together with an ExplicitPlacement, locating each clip at a given time and track. (Note: I simplified the time format and wrote frame numbers to make it more clear)
[img[Example1: Objects in the Session/Fixture|uml/fig128773.png]]
----
After beeing processed by the Builder, we get the following Render Engine configuration
{{red{note: please take this only as a "big picture", the implementation details got a lot more complicated as of 6/08}}}
[img[Example1: generated Render Engine|uml/fig129029.png]]
{{red{TODO: seemingly this example is slightly outdated, as the implementation of placements is now indirect via LocatingPin objects}}}
This Example showes the //high level// Sequence as well. This needs to be transformed into a Fixture by some facility still to be designed. Basically, each [[Placement]] needs to be queried for this to get the corresponding ExplicitPlacement. The difficult part is to handle possible Placement constraints, e.g. one clip can't be placed at a timespan covered by another clip on the same track. In the current Cinelerra2, all of this is done directly by the GUI actions.
The »Timeline« is a sequence of ~MObjects -- note: using the same Object instances -- but now with the calculated ExplicitPlacement, locating the clip at a given time and track. The effect is located absolutely in time as well, but because it is the same Instance, it has the pointer to the ~RelativePlacement, wich basically attaches the effect to the clip. This structure may look complicated, but is easy to process if we go "backward" and just rely on the information contained in the ExplicitPlacement.
[img[Example2: Clip with Effect and generated Fixture for this Sequence|uml/fig128901.png]]
----
After beeing processed by the Builder, we get a Render Engine configuration.<br>
It has to be segmented at least at every point with changes in the configuration, but some variations are possible, e.g. we could create a Render Engine for every Frame (as Cinelerra2 does) or we could optimize out some configurations (for example the effect extended beyond the end of the clip)
{{red{note: as of 6/08 this can be taken only as the "big picture". Implementation will differ in details, and is more complicated than showed here}}}
[img[Example2: generated Render Engine|uml/fig129157.png]]
!MObject assembly
To make the intended use of the classes more clear, consider the following two example Object graphs:
* a video clip and a audio clip placed (explicitly) on two tracks →[[Example1]]
* a video clip placed relatively, with an attached HUE effect →[[Example2]]
a special ProcNode which is used to pull the finished output of one Render Pipeline (Tree or Graph). This term is already used in the Cinelerra2 codebase. I am unsure at the moment if it is a distinct subclass or rahter a specially configured ProcNode (a general design rule tells us to err in favour of the latter if in doubt).
The render nodes network is always built separate for each [[timeline segment|Segmentation]], which is //constant in wiring configuration.// Thus, while exit node(s) are per segment, the corresponding exit nodes of consecutive segments together belong to a ModelPort, which in turn corresponds to a global pipe (master bus not connected any further). These relations guide the possible configuration for an exit node: It may still provide multiple channels -- but all those channels are bound to belong to a single logical stream -- same StreamPrototype, always handled as bundle, connected and routed in one step. For example, when there is an 5.1 Audio master bus with a single fader, then "5.1 Audio" would be a prototype and these 6 channels will always be handled together; in such a case it makes perfectly sense to access these 6 audio channels through a single exit node, which is keyed (identified) by the same [[Pipe]]-ID as the corresponding ModelPort and the corresponding global pipe ("5.1 Audio master bus")
A special kind (subclass) of [[Placement]]. As such it is always linked to a //Subject//, i.e. a MObject. But contrary to the (standard) placements, which may exhibit all kinds of fancy dynamic and scope dependent behaviour, within an explicit placement all properties are resolved and materialised. While the (standard) placement may contain an arbitrary list of LocatingPin objects, the resolution into an explicit placement performs a kind of »orthogonalisation«: each remaining LocatingPin defines exactly one degree of freedom independent of all others. Most notably, the explicit placement always specifies a absolute time and [[output designation|OutputDesignation]] for for locating the Subject. Explicit placements are ''immutable''.
!!Implementation considerations
Explicit placements are just created and never mutated, but copying and storage might become a problem.
It would thus be desirable to have a fixed-sized allocation, able to hold the placement body as well as the (fixed) locating pins inline.
We use Factories
* for centralizing [[memory management|MemoryManagement]]
* to support polymorphism (of course...)
!Requirements
* request the actual placement/allocation from the backend
* allways hand out a smart-pointer
* encapsulate / make configurable the smart-pointer type
* install a callback into the smart-pointer for destroying the resource.
* redirect the destroying request to the backend
!Implementation Questions
* how much genericity? (Ichthyo is rather inclined not to overdo this one. Often it is preferable to have repeated implementations follow a well known pattern, if this leads to short and simple implementations, while the complete general solution will be much more contrived).
* how to specify the actual type needed?
* how to implement the cases where a subtype needs to be selected (abstract factory pattern). Embody this into the Factory, pass it in as a Strategy or treat the Factory just as a simple service taking an explicit type-ID and providing the new object?
!!chosen design
My main goal is to have an easy-to-use interface for the implementer of individual classes using this factory mechanism. The intended use should mimic the standard use of operator new, and at the same time there should be a possibility to configure »real« Factory behaviour in.
For this reason I make Factory a Functor, so it can be incorporated as a member into another class, while still looking like a function call to the client code. The users of this factory template typically either parametrize it with existing smart-pointer types, or they may chose to create a specialized Factory derived from this Factory template. After typically hiding this configuration behind a typedef, the user adds a static field to the class intended to use the Factory and initializes this field with the concrete Factory (this may pass internal ~IDs to the constructor of the Factory and from there on to the underlying Allocator).
{{{
#include "common/factory.hpp"
class Product
{
int wow_;
public:
typedef lumiera::factory::RefcntFactory<Product> Factory;
static Factory create;
Product() : wow_(42) {} ;
};
/** storage for a static Factory instance
* for creating refcounting Ptrs to Product objects
*/
Product::Factory Product::create; // <----note this is a ctor call
}}}
Now, the clients of {{{class Product}}} can create ref-counting pointers to Product-objects by doing a fully qualified {{{create()}}} functor call:
{{{
std::tr1::shared_ptr<Product> huii = Product::create (); // <----will invoke the default Product() ctor
std::tr1::shared_ptr<Product> pfuii = huii;
}}}
Some further details
* the product class (or base class) cares for using an custom allocator via an overloaded {{{operator new(size_t)}}} if applicable.
* thus, when Functor or any derived class issues a new XX(), our custom Allocator gains control
* the Functor-behaviour relies on a custom {{{operator()}}} which can be overridden in derived classes to take various parameter lists.
* additionally, such a Factory class derived from Functor can do specific magic and e.g. create some subclass
* and, as the created smart-pointer is a template template parameter, such a custom Functor can create all sorts of Proxies, wrappers and the like
* a special case of this factory use is the [[Singleton]] factory, which is used a lot within the Proy-Layer code
a specially configured view -- joining together high-level and low-level model.
The Fixture acts as //isolation layer// between the two models, and as //backbone to attach the render nodes.//
* all MObjects have their position, length and configuration set up ready for rendering.
* any nested sequences (or other kinds of indirections) have been resolved.
* every MObject is attached by an ExplicitPlacement, which declares a fixed position (Time, [[Pipe|OutputDesignation]])
* these ~ExplicitPlacements are contained immediately within the Fixture, ordered by time
* besides, there is a collection of all effective, possibly externally visible [[model ports|ModelPortRegistry]]
As the builder and thus render engine //only consults the fixture,// while all editing operations finally propagate to the fixture as well, we get an isolation layer between the high level part of the Proc layer (editing, object manipulation) and the render engine. [[Creating the Fixture|BuildFixture]] is an important first step and sideeffect of running the [[Builder]] when createing the [[render engine network|LowLevelModel]].
''Note'': all of the especially managed storage of the LowLevelModel is hooked up behind the Fixture
→ FixtureStorage
→ FixtureDatastructure
!{{red{WIP}}} Structure of the fixture
[<img[Structure of the Fixture|draw/Fixture1.png]]
The fixture is like a grid, where one dimension is given by the [[model ports|ModelPortRegistry]], and the other dimension extends in time. Within the time dimension there is a grouping into [[segments|Segmentation]] of constant structure.
;Model Ports
:The model ports share a single uniform and global name space: actually they're keyed by ~Pipe-ID
:Model ports are derived as a result of the build process, as the //residuum// of all nodes not connected any further
:Each port belongs to a specific Timeline and is associated with the [[Segmentation]] of that timeline.
;Segmentation
:The segmentation partitiones the time axis of a single timeline into segments of constant (wiring) configuration
:Together, the segments form a seamless sequence of time intervals. They contain a copy of each (explicit) placement of a visible object touching that time interval. Besides that, segments are the top level grouping device of the render engine node graph; they are always built and discarded at once.
:Segments (and even a different Segmentation) may be //hot swapped// into an ongoing render.
;Exit Nodes
:Each segment holds an ExitNode for each relevant ModelPort of the corresponding timeline.
:Thus the exit nodes are keyed by ~Pipe-ID as well (and consequently have a distinct [[stream type|StreamType]]) -- each model port coresponds to {{{<number_of_segments>}}} separate exit nodes, but of course an exit node may be //mute.//
Generally speaking, the datastructure to implement the ''Fixture'' (→ see a more general description [[here|Fixture]]) is comprised of a ModelPortRegistry and a set of [[segmentations|Segmentation]] per Timeline.
This page focusses on the actual data structure and usage details on that level. See also → [[storage|FixtureStorage]] considerations.
!transactional switch
A key point to note is the fact that the fixture is frequently [[re-built|BuildFixture]] by the [[Builder]], while render processes may be going on in parallel. Thus, when a build process is finished, a transactional ''commit'' happens to ''hot swap'' the new parts of the model. This is complemented by a clean-up of tainted render processes; finally, storage can be reclaimed.
To support this usage pattern, the Fixture implementation makes use of the [[PImpl pattern|http://c2.com/cgi/wiki?PimplIdiom]]
!Collecting usage scenarios {{red{WIP 12/10}}}
* ModelPort access
** get the model port for a given ~Pipe-ID
** enumerate the model ports for a Timeline
* rendering frame dispatch
** get or create the frame dispatcher table
** dispatch a single frame to yield the corresponding ExitNode
* (re)building
** create a new implementation transaction
** create a new segmentation
** establish what segments actually need to be rebuilt
** dispatch a newly built segment into the transaction
** schedule superseded segments and tainted process for clean-up
** commit a transaction
!Conclusions about the structure {{red{WIP 12/10}}}
* the ~PImpl needs to be a single (language) pointer. This necessitates having a monolithic Fixture implementation holder
* moreover, this necessitates a tight integration down to implementation level, both with the clean-up and the render processes themselves
The Fixture → [[data structure|FixtureDatastructure]] acts as umbrella to hook up the elements of the render engine's processing nodes network (LowLevelModel).
Each segment within the [[Segmentation]] of any timeline serves as ''extent'' or unit of memory management: it is built up completely during the corresponding build process and becomes immutable thereafter, finally to be discarded as a whole when superseded by a modified version of that segment (new build process) -- but only after all related render processes (→ CalcStream) are known to be terminated.
Each segment owns an AllocationCluster, which in turn manages all the numerous small-sized objects comprising the render network implementing this segment -- thus the central question is when to //release the segment.//
* for one, we easily detect the point when a segment is swapped out of the segmentation; at this point we also have to detect the //tainted calculation streams.//
* but those render processes (calc streams) terminate asynchronously, and that forces us to do some kind of registration and deregistration.
!!question: is ref-counting acceptable here?
//Not sure yet.// Of course it would be the simplest approach. KISS.
Basically the concern is that each new CalcStream had to access the shared counts of all segments it touches.
''Note'': {{{shared_ptr}}} is known to be implemented by a lock-free algorithm (yes it is, at least since boost 1.33. Don't believe what numerous FUD spreaders have written). Thus lock contention isn't a problem, but at least a memory barrier is involved (and if I judge GCC's internal documentation right, currently their barriers extend to //all// globally visible variables)
!!question: alternatives?
There are. As the builder is known to be run again and again, no one forces us to deallocate as soon as we could. That's the classical argument exploited by any garbage collector too. Thus we could just note the fact that a calculation stream is done and re-evaluate all those noted results on later occasion. Obviously, the [[Scheduler]] is in the best position for notifying the rest of the system when this and that [[job|RenderJob]] has terminated, because the Scheduler is the only facility required to touch each job reliably. Thus it seems favourable to add basic support for either termination callbacks or for guaranteed execution of some notification jobs to the [[Scheduler's requirements|SchedulerRequirements]].
!!exploiting the frame-dispatch step
Irrespective of the decision in favour or against ref-counting, it seems reasonable to make use of the //frame dispatch step,// which is necessary anyway. The idea is to give each render process (maybe even each CalcStream) a //copy// of an dispatcher table object -- basically just a list of time breaking points and a pointer to the relevant exit node. If we keep track of those dispatcher tables, add some kind of back-link to identify the process and require the process in turn to deregister, we get a tracking of tainted processes for free.
!!assessment {{red{WIP 12/10}}}
But the primary question here is to judge the impact of such an implementation. What would be the costs?
# creating individual dispatcher tables trades memory for simplified parallelism
# the per-frame lookup is efficient and negligible compared with just building the render context (StateProxy) for that frame
# when a process terminates, we need to take out that dispatcher and do deregistration stuff for each touched segment (?)
!!!Estimations
* number of actually concurrent render processes is at or below 30
* depending on the degree of cleverness on behalf of the scheduler, the throughput of processes might be multiplied (dull means few processes)
* the total number of segments within the Fixture could range into several thousand
* but esp. playback is focussed, which makes a number of rather several hundred tainted segments more likely
⇒ we should try quickly to dispose of the working storage after render process termination and just retain a small notification record
⇒ so the frame dispatcher table should be allocated //within// the process' working storage; moreover it should be tiled
⇒ we should spend a second thought about how actually to find and process the segments to be discarded
!!!identifying tainted and disposable segments
Above estimation hints at the necessity of frequently finding some 30 to 100 segments to be disposed, out of thousands, assumed the original reason for triggering the build process was a typically local change in the high-level model. We can only discard when all related processes are finished, but there is a larger number of segments as candidate for eviction. These candidates are rather easy to pinpoint -- they will be uncovered during a linear comparison pass prior to committing the changed Fixture. Usually, the number of candidates will be low (localised changes), but global manipulations might invalidate thousands of segments.
* if we frequently pick the segments actually to be disposed, there is the danger of performance degeneration when the number of segments is high
* the other question is if we can afford just to keep all of those candidates around, as all of them are bound to get discardable eventually
* and of course there is also the question how to detect //when// they're due.
;Model A
:use a logarithmic datastructure, e.g. a priority queue. Possibly together with LRU ordering
:problem here is that the priorities change, which either means shared access or a lot of "superseded" entries
;Model B
:keep all superseded segments around and track the tainted processes instead
:problem here is how to get the tainted processes precisely and with low overhead
//currently {{red{12/10}}} I tend to prefer Model B...// while the priority queue remains to be investigated in more detail for organising the actual build process.
But actually I'm struck here, because of the yet limited knowledge about those render processes....
* how do we //join// an aborted/changed rendering process to his successor, without creating a jerk in the output?
* is it even possible to continue a process when parts of the covered time-range are affected by a build?
If the latter question is answered with "No!", then the problem gets simple in solution, but maybe memory consuming: In that case, //all// processes linked to a timeline gets affected and thus tainted; we'd just dump them onto a pile and delay releasing all of the superseded segments until all of them are known to be terminated.
The situation focussed by this concept is when an API needs to expose a sequence of results, values or objects, instead of just yielding a function result value. As the naive solution of passing an pointer or array creates coupling to internals, it was superseded by the ~GoF [[Iterator pattern|http://en.wikipedia.org/wiki/Iterator]]. Iteration can be implemented by convention, polymorphically or by generic programming; we use the latter approach.
!Lumiera Forward Iterator concept
''Definition'': An Iterator is a self-contained token value, representing the promise to pull a sequence of data
* rather then deriving from an specific interface, anything behaving appropriately //is a Lumiera Forward Iterator.//
* the client finds a typedef at a suitable, nearby location. Objects of this type can be created, copied and compared.
* any Lumiera forward iterator can be in //exhausted// (invalid) state, which can be checked by {{{bool}}} conversion.
* especially, default constructed iterators are fixed to that state. Non-exhausted iterators may only be obtained by API call.
* the exhausted state is final and can't be reset, meaning that any iterator is a disposable one-way-off object.
* when an iterator is //not// in the exhausted state, it may be //dereferenced// ({{{*i}}}), yielding the "current" value
* moreover, iterators may be incremented ({{{++i}}}) until exhaustion.
!!Discussion
The Lumiera Forward Iterator concept is a blend of the STL iterators and iterator concepts found in Java, C#, Python and Ruby. The chosen syntax should look familiar to C++ programmers and indeed is compatible to STL containers and ranges. To the contrary, while a STL iterator can be thought off as being just a disguised pointer, the semantics of Lumiera Forward Iterators is deliberately reduced to a single, one-way-off forward iteration, they can't be reset, manipulated by any arithmetic, and the result of assigning to an dereferenced iterator is unspecified, as is the meaning of post-increment and stored copies in general. You //should not think of an iterator as denoting a position// — just a one-way off promise to yield data.
Another notable difference to the STL iterators is the default ctor and the {{{bool}}} conversion. The latter allows using iterators painlessly within {{{for}}} and {{{while}}} loops; a default constructed iterator is equivalent to the STL container's {{{end()}}} value — indeed any //container-like// object exposing Lumiera Forward Iteration is encouraged to provide such an {{{end()}}}-function, additionally enabling iteration by {{{std::for_each}}} (or Lumiera's even more convenient {{{util::for_each()}}}).
!!Implementation notes
''iter-adapter.hpp'' provides some helper templates for building Lumiera Forward Iterators.
* __~IterAdapter__ is the most flexible variant, intended for use by custom facilities. An ~IterAdapter maintains an internal back-link to a facilitiy exposing an iteration control API, which is accessed through free functions as extension point. This iteration control API is similar to C#, allowing to advance to the next result and to check the current iteration state.
* __~RangeIter__ wraps two existing iterators — usually obtained from {{{begin()}}} and {{{end()}}} of an STL container embedded within the implementation. This allows for iterator chaining.
* __~PtrDerefIter__ works similar, but can be used on an STL container holding //pointers,// to be dereferenced automatically on access
Similar to the STL habits, Lumiera Forward Iterators should expose typedefs for {{{pointer}}}, {{{reference}}} and {{{value_type}}}.
Additionally, they may be used for resource management purposes by embedding a ref-counting facility, e.g. allowing to keep a snapshot or restult set around until it can't be accessed anymore.
This term has //two meanings, //so care has to be taken for not confusing them.
# in general use, a Frame means one full image of a video clip, i.e an array of rows of pixels. For interlaced footage, one Frame contains two halfimages, commonly called Fields. (Cinelerra2 confuses this terms)
# here in this design, we use Frame as an abstraction for a buffer of raw media data to be processed. If in doubt, we should label this "DataFrame".
#* one video Dataframe contains a single video frame
#* one audio Dataframe contains a block of raw audio samples
#* one OpenGL Dataframe could contain raw texture data (but I am lacking expertise for this topic)
An implementation facility within the RenderEngine, responsible for translating a logical [[calculation stream|CalcStream]] (corresponding to a PlayProcess) into a sequence of individual RenderJob entries, which can then be handed over to the [[Scheduler]]. Performing this operation involves a special application of [[time quantisation|TimeQuant]]: after establishing a suitable starting point, a typically contiguous series of frame numbers need to be generated, together with the time coordinates for each of those frames
/***
|Name|FullScreenPlugin|
|Created by|SaqImtiaz|
|Location|http://tw.lewcid.org/#FullScreenPlugin|
|Version|1.1|
|Requires|~TW2.x|
!Description:
Toggle between viewing tiddlers fullscreen and normally. Very handy for when you need more viewing space.
!Demo:
Click the ↕ button in the toolbar for this tiddler. Click it again to turn off fullscreen.
!Installation:
Copy the contents of this tiddler to your TW, tag with systemConfig, save and reload your TW.
Edit the ViewTemplate to add the fullscreen command to the toolbar.
!History:
*25-07-06: ver 1.1
*20-07-06: ver 1.0
!Code
***/
//{{{
var lewcidFullScreen = false;
config.commands.fullscreen =
{
text:" ↕ ",
tooltip:"Fullscreen mode"
};
config.commands.fullscreen.handler = function (event,src,title)
{
if (lewcidFullScreen == false)
{
lewcidFullScreen = true;
setStylesheet('#sidebar, .header, #mainMenu{display:none;} #displayArea{margin:0em 0 0 0 !important;}',"lewcidFullScreenStyle");
}
else
{
lewcidFullScreen = false;
setStylesheet(' ',"lewcidFullScreenStyle");
}
}
config.macros.fullscreen={};
config.macros.fullscreen.handler = function(place,macroName,params,wikifier,paramString,tiddler)
{
var label = params[0]||" ↕ ";
var tooltip = params[1]||"Fullscreen mode";
createTiddlyButton(place,label,tooltip,config.commands.fullscreen.handler);
}
var lewcid_fullscreen_closeTiddler = Story.prototype.closeTiddler;
Story.prototype.closeTiddler =function(title,animate,slowly)
{
lewcid_fullscreen_closeTiddler.apply(this,arguments);
if (story.isEmpty() && lewcidFullScreen == true)
config.commands.fullscreen.handler();
}
Slider.prototype.lewcidStop = Slider.prototype.stop;
Slider.prototype.stop = function()
{
this.lewcidStop();
if (story.isEmpty() && lewcidFullScreen == true)
config.commands.fullscreen.handler();
}
//}}}
The ''Gmerlin Audio Video Library''. → see [[homepage|http://gmerlin.sourceforge.net/gavl.html]]
Used within Lumiera as a foundation for working with raw video and audio media data
__G__roup __of__ __P__ictures: several compressed video formats don't encode single frames. Normally, such formats are considered mere //delivery formates// but it was one of the key strenghts of Cinelrra from start to be able to do real non linear editing on such formats (like the ~MPEG2-ts unsed in HDV video). The problem of course is that the data backend needs to decode the whole GOP to be serve single raw video frames.
For this Lumiera design, we could consider making GOP just another raw media data frame type and integrate this decoding into the render pipeline, similar to an effect based on several source frames for every calculated output frame.
→see in [[Wikipedia|http://en.wikipedia.org/wiki/Group_of_pictures]]
Each [[Timeline]] has an associated set of global [[pipes|Pipe]] (global busses), similar to the subgroups of a sound mixing desk.
In the typical standard configuration, there is (at least) a video master and a sound master pipe. Like any pipe, ingoing connections attach to the input side, attached effects form a chain, where the last node acts as exit node. The ~Pipe-ID of such a global bus can be used to route media streams, allowing the global pipe to act as a summation bus bar.
→ discussion and design rationale of [[global pipes|GlobalPipeDesign]]
!Properties and decisions
* each timeline carries its own set of global pipes, as each timeline is an top-level element on its own
* like all [[pipes|Pipe]] the global ones are kept separated per stream (proto)type
* any global pipe //not// connected to another OutputDesignation automatically creates a ModelPort
* global pipes //do not appear automagically just by sending output to them// -- they need to be set up explicitly
* the top-level (BusMO) of the global pipes isn't itself a pipe. Thus the top-level of the pipes forms a list (typically a video and sound master)
* below, a tree-like structure //may// be created, building upon the same scope based routing technique as used for the tracks
//This page serves to shape and document the design of the global pipes//
Many aspects regarding the global pipes turned out while clarifying other parts of ~Proc-Layer's design. For some time it wasn't even clear if we'd need global pipes -- common video editing applications get on without. Mostly it was due to the usefulness of the layout found on sound mixing desks, and a vague notion to separate time-dependant from global parts, which finally led me to favouring such a global facility. This decision then helped in separating the concerns of timeline and sequence, making the //former// a collection of non-temporal entities, while the latter concentrates on time varying aspects.
!Design problem with global Pipes
actually building up the implementation of global pipes seems to pose a rather subtle design problem: it is difficult to determine how to do it //right.//
To start with, we need the ability to attach effects to global pipes. There is already an obvious way how to attach effects to clips (=local pipes), and thus it's desirable to handle it the same way for global pipes. At least there should be a really good reason //not// to do it the same way. Thus, we're going to attach these effects by placement into the scope of another placed MObject. And, moreover, this other object should be part of the HighLevelModel's tree, to allow using the PlacementIndex as implementation. So this reasoning brings us to re-using or postulating some kind of object, while lacking a point or reference //outside this design considerations// to justify the existence of the corresponding class or shaping its properties on itself. Which means — from a design view angle — we're entering slippery ground.
!!!~Model-A: dedicated object per pipe
Just for the sake of symmetry, for each global pipe we'd attach some suitable ~MObject as child of the BindingMO representing the timeline. If no further specific properties or functionality is required, we could use Track objects, which are generally used as containers within the model. Individual effects would then be attached as children, while output routing could be specified within the attaching placement, the same way as it's done with clips or tracks in general. As the pipe asset itself already stores a StreamType reference, all we'd need is some kind of link to the pipe asset or pipe-ID, and maybe façade functions within the binding to support the handling.
!!!~Model-B: attaching to the container
Acknowledging the missing justification, we could instead use //just something to attach// — and actually handle the real association elsewhere. The obvious "something" in this case would be the BindingMO, which already acts as implementation of the timeline (which is a façade asset). Thus, for this approach, the bus-level effects would be attached as direct children of the {{{Placement<BindingMO>}}}, just for the sake of beeing attached and stored within the session, with an additional convention for the actual ordering and association to a specific pipe. The Builder then would rather query the ~BindingMO to discover and build up the implementation of the global pipes in terms of the render nodes.
!!!Comparison
While there might still be some compromises or combined solutions — to support the decision, the following table detailes the handling in each case
|>| !~Model-B|!~Model-A |
|Association: | by plug in placement|by scope |
|Output: | entry in routing table|by plug in placement |
|Ordering: |>| stored in placement |
|Building: | sort children by plug+order<br/>query output from routing|build like a clip |
|Extras: | ? |tree-like bus arrangement,<br/> multi-stream bus |
So through this detailed comparison ''~Model-A looks favourable'': while the other model requires us to invent a good deal of the handling specifically for the global pipes, the former can be combined from patterns and solutions already used in other parts of the model, plus it allows some interesting extensions.
On a second thought, the fact that the [[Bus-MObject|BusMO]] is rather void of any specific meaning doesn't weight so much: As the Builder is based on the visitor pattern, the individual objects can be seen as //algebraic data types.// Besides, there is at least one little bit of specific functionality: a Bus object actually needs to //claim// to be the OutputDesignation, by referring to the same ~Pipe-ID used in other parts of the model to request output routing to this Bus. Without this match on both ends, an ~OutputDesignation may be mentioned at will, but no connection whatsoever will happen.
!{{red{WIP}}} Structure of the global pipes
;creating global pipes automatically?
:defining the global bus configuration is considered a crucial part of each project setup. Lumiera isn't meant to support fiddling around thoughtlessly. The user should be able to rely on crucial aspects of the global setup never being changed without notice.
;isn't wiring and routing going to be painful then?
:routing is scope based and we employ a hierarchical structure, so subgroups are routed automatically. Moreover, wiring is done based on best match regarding the stream type. We might consider feeding all non-connected output designations to the GUI after the build process, to allow short-cuts for creating further buses.
;why not making buses just part of the track tree?
:anything on a track has a temporal extension and may vary -- while it's the very nature of the global pipes to be static anchor points.
;why not having one grand unified root, including the outputs?
:you might consider that a matter of taste (or better common-sense). Things different in nature should not be forced into uniformity
;should global pipes be arranged as list or tree?
:sound mixing desks use list style arrangement, and this has proven to be quite viable, when combined with the ability to //send over// output from one mixer stripe to the input of another, allowing to build arbitrary complex filter matrices. On the other hand, organising a mix in //subgroups// can be considered best practice. This leads to arranging the pipes //as wood:// by default and on top level as list, optionally expanding into a subtree with automatic rooting, augmented by the ability to route any output to any input (cycles being detected and flagged as error).
All communication between Proc-Layer and GUI has to be routed through the respective LayerSeparationInterfaces. Following a fundamental design decision within Lumiera, these interface are //intended to be language agnostic// — forcing them to stick to the least common denominator. Which creates the additional problem of how to create a smooth integration without forcing the architecture into functional decomposition style. To solve this problem, we rely on the well known solution of using a __business facade__ and delegation proxies.
Thus, the Proc-Layer exposes (one or several) facade interfaces for the GUI to use it's functionality, and similarily the GUI provides a [[notification facade|GuiNotificationFacade]] for pushing back status information created as result of the edit operations, the build process and the render tasks.
!anatomy of the Proc/GUI interface
* the GuiFacade is used as a general lifecycle facade to start up the GUI and to set up the LayerSeparationInterfaces.
* GuiFacade is implemented by a class //in core// and exposes a notification proxy implementing GuiNotificationFacade, which actually is wired through the InterfaceSystem to forward to the corresponding implementation facade object within the GUI
* similarly, starting the fundamental subsystems within proc installs Interfaces into the InterfaceSystem and exposes them through proxy objects
special LayerSeparationInterface which serves the main purpose to load the GuiStarterPlugin, thus bringing up the Lumiera GTK UI at application start.
Considering how to interface to and integrate with the GUI Layer. Running the GUI is //optional,// but it requires to be [[started up|GuiStart]], installing the necessary LayerSeparationInterfaces. As an example how to integrate the GUI with the lower layers, in 1/2009 we created an PlayerDummy, which "pulls" dummy frames from the (not yet existing) engine and displays them within an XV viewer widget.
Probably the most important aspect regarding the GUI integration is how to get [[access to and operate on the Session|SessionInterface]]. More specifically, this includes [[referring to individual objects|MObjectRef]].
LayerSeparationInterface provided by the GUI.
Access point for the lower layers to push information and state changes (aynchronously) to the GUI. Actually, most operations within Lumiera are initiated by the user through the GUI. In the course of such actions, the GUI uses the services of the lower layer and typically recieves an synchronous response. In some exceptional cases, these operations may cause additional changes to happen asynchronously from the GUI's perspective. For example, an edit operation might trigger a re-build of the low-level model, which then detects an error.
Starting up the GUI is optional and is considered part of the Application start/stop and lifecycle.
* main and AppState activate the lifecyle methods on the ~GuiSubsysDescriptor, accessible via the GuiFacade
* loading a GuiStarterPlugin actually
** loads the GUI (shared lib)
** creates instances of all the GUI services available through LayerSeparationInterfaces
*** GuiNotificationFacade
*** DisplayFacade
** Finally, the ~GTK-main() is invoked.
!public services
The GUI provides a small number of public services, callable through LayerSeparationInterfaces. Besides that, the main purpose of the GUI of course is user interaction. Effectively the behaviour of the whole system is driven by GUI events to a large extent. These events are executed within the event handling thread (~GTK-main-Thread) and may in turn invoke services of the lower layers, again through the respective LayerSeparationInterfaces.
But the question of special interest here is how the //public services// of the GUI are implemented and made accessible for the lower Layers. Layer isolation is an issue here. If implemented in a rigorous manner, no facility within one layer may invoke implementation code of another layer directly. In practice, this tends to put quite some additional burden on the implementer, without and obvious benefit. Thus we decided to lower the barrier somewhat: while we still require that all service invoking calls are written against an public LayerSeparationInterface, actually the GUI (shared lib) is //linked// against the respective shared libs of the lower layers, thus especially enabling the exchange of iterators, closures and functor objects.
Note that we retain strict isolation in the other direction: no part of the lower layers is allowed to call directly into the GUI. Thus it's especially interesting how access to some GUI public service from the lower layers works in detail.
* when the GUI plugin starts, instances of the Services implementing those public service interfaces are created.
* these service objects in turn hold an ~InstanceHandle, which cares to register and open the corresponding C Language Interface
* additionally this InstanceHandle is configured such as to create an "facade proxy" object, which is implemented within liblumieracommon.so
Now, when invoking an operation on some public interface, the code in the lower layers actually executes an implementation of this operation //on the facade proxy,// which in turn forwards the call through the CL interface into the GUI, where it is actually implemented by the corresponding service object instance.
A specially configured LumieraPlugin, which actually contains or loads the complete code of the (GTK)GUI, and additionally is linked dynamically against the application core lib. During the [[UI startup process|GuiStart]], loading of this Plugin is triggered from {{{main()}}}. Actually this causes spawning of the GTK event thread and execution of the GTK main loop.
While the low-level model holds the data used for carrying out the actual media data processing (=rendering), the high-level model is what the user works upon when performing edit operations through the GUI (or script driven in »headless mode«). Its building blocks and combination rules determine largely what structures can be created within the [[Session]].
On the whole, it is a collection of [[media objects|MObjects]] stuck together and arranged by [[placements|Placement]].
Basically, the structure of the high-level model is is a very open and flexible one — every valid connection of the underlying object types is allowed — but the transformation into a low-level node network for rendering follows certain patterns and only takes into account any objects reachable while processing the session data in accordance to these patterns. Taking into account the parameters and the structure of these objects visited when building, the low-level render node network is configured in detail.
The fundamental metaphor or structural pattern is to create processing ''pipes'', which are a linear chain of data processing modules, starting from an source port and providing an exit point. [[Pipes|Pipe]] are a //concept or pattern,// they don't exist as objects. Each pipe has an input side and an output side and is in itself something like a Bus treating a single [[media stream|StreamType]] (but this stream may still have an internal structure, e.g. several channels related to a spatial audio system). Other processing entities like effects and transitions can be placed (attached) at the pipe, resulting them to be appended to form this chain. Optionally, there may be a ''wiring plug'', requesting the exit point to be connected to another pipe. When omitted, the wiring will be figured out automatically.
Thus, when making an connection //to// a pipe, output data will be sent to the //source port// (input side) of the pipe, wheras when making a connection //from// a pipe, data from it's exit point will be routed to the destination. Incidentally, the low-level model and the render engine employ //pull-based processing,// but this is rather of no relevance for the high-level model.
[img[draw/high-level1.png]]
Normally, pipes are limited to a //strictly linear chain// of data processors ("''effects''") working on a single data stream type, and consequently there is a single ''exit point'' which may be wired to an destination. As an exception to this rule, you may insert wire tap nodes (probe points), which explicitly may send data to an arbitrary input port; they are never wired automatically. It is possible to create cyclic connections by such arbitrary wiring, which will be detected by the builder and flagged as an error.
While pipes have a rather rigid and limited structure, it is allowed to make several connections to and from any pipe — even connections requiring an stream type conversion. It is not even necessary to specify //any// output destination, because then the wiring will be figured out automatically by searching the context and finally using some general rule. Connecting multiple outputs to the input of another pipe automatically creates a ''mixing step'' (which optionally can be controlled by a fader). Several pipes may be joined together by a ''transition'', which in the general case simultaneously treats N media streams. Of course, the most common case is to combine two streams into one output, thereby also mixing them. Most available transition plugins belong to this category, but, as said, the model isn't limited to this simple case, and moreover it is possible to attach several overlapping transitions covering the same time interval.
Individual Media Objects are attached, located or joined together by ''Placements''. A [[Placement]] is a handle for a single MObject (implemented as a refcounting smart-ptr) and contains a list of placement specifications, called LocatingPin. Adding an placement to the session acts as if creating an //instance.// (it behaves like a clone in case of multiple placements of the same object). Besides absolute and relative placement, there is also the possibility of a placement to stick directly to another MObject's placement, e.g. for attaching an effect to a clip or to connect an automation data set to an effect. This //stick-to placement// creates sort of a loose clustering of objects: it will derive the position from the placement it is attached to. Note that while the length and the in/out points are a //property of the ~MObject,// it's actual location depends on how it is //placed// and thus can be maintained quite dynamically. Note further that effects can have an length on their own, thus by using these attachement mechaics, the wiring and configuration within the high-level model can be quite time dependant.
[>img[draw/high-level2.png]]
Actually a ''clip'' is handled as if it was comprised of local pipe(s). In the example shown here, a two-channel clip has three effects attached, plus a wiring plug. Each of those attachments is used only if applicable to the media stream type the respective pipe will process. As the clip has two channels (e.g. video and audio), it will have two ''source ports'' pulling from the underlying media. Thus, as showed in the drawing to the right, by chaining up any attached effect applicable to the respective stream type defined by the source port, effectively each channel (sub)clip gets its own specifically adapted processing pipe.
@@clear(right):display(block):@@
!!Example of an complete Session
[img[draw/high-level3.png]]
The Session contains several independent [[sequences|Sequence]] plus an output bus section (''global Pipes'') attached to the [[Timeline]]. Each sequence holds a collection of ~MObjects placed within a ''tree of tracks''.
Within Lumiera, tracks are a rather passive means for organizing media objects, but aren't involved into the data processing themselves. The possibility of nesting tracks allows for easy grouping. Like the other objects, tracks are connected together by placements: A track holds the list of placements of its child tracks. Each sequence holds a single placement pointing to the root track.
As placements have the ability to cooperate and derive any missing placement specifications, this creates a hierarchical structure throughout the session, where parts on any level behave similar if applicable. For example, when a track is anchored to some external entity (label, sync point in sound, etc), all objects placed relatively to this track will adjust and follow automatically. This relation between the track tree and the individual objects is especially important for the wiring, which, if not defined locally within an ~MObject's placement, is derived by searching up this track tree and utilizing the wiring plug locating pins found there, if applicable. In the default configuration, the placement of an sequence's root track contains a wiring plug for video and another wiring plug for audio. This setup is sufficient for getting every object within this sequence wired up automatically to the correct global output pipe. Moreover, when adding another wiring plug to some sub track, we can intercept and reroute the connections of all objects creating output of this specific stream type within this track and on all child tracks.
Besides routing to a global pipe, wiring plugs can also connect to the source port of an ''meta-clip''. In this example session, the outputs of ~Seq-2 as defined by locating pins in it's root track's placement, are directed to the source ports of a [[meta-cllip|VirtualClip]] placed within ~Seq-1. Thus, within ~Seq-1, the contents of ~Seq-2 appear like a pseudo-media, from which the (meta) clip has been taken. They can be adorned with effects and processed further completely similar to a real clip.
Finally, this example shows an ''automation'' data set controlling some parameter of an effect contained in one of the global pipes. From the effect's POV, the automation is simply a ParamProvider, i.e a function yielding a scalar value over time. The automation data set may be implemented as a bézier curve, or by a mathematical function (e.g. sine or fractal pseudo random) or by some captured and interpolated data values. Interestingly, in this example the automation data set has been placed relatively to the meta clip (albeit on another track), thus it will follow and adjust when the latter is moved.
This wiki page is the entry point to detail notes covering some technical decisions, details and problems encountered in the course of the implementation of the Lumiera Renderengine, the Builder and the related parts.
* [[Packages, Interfaces and Namespaces|InterfaceNamespaces]]
* [[Memory Management Issues|MemoryManagement]]
* [[Creating and registering Assets|AssetCreation]]
* [[Creating new Objects|ObjectCreation]]
* [[Multichannel Media|MultichannelMedia]]
* [[Editing Operations|EditingOperations]]
* [[Handling of the current Session|CurrentSession]]
* [[collecting Ideas for Implementation Guidelines|ImplementationGuidelines]]
* [[using the Visitor pattern?|VisitorUse]] -- resulting in [[»Visiting-Tool« library implementation|VisitingToolImpl]]
* [[Handling of Tracks and render Pipes in the session|TrackPipeSequence]]. [[Handling of Tracks|TrackHandling]] and [[Pipes|PipeHandling]]
* [[getting default configured|DefaultsManagement]] Objects relying on [[rule-based Configuration Queries|ConfigRules]]
* [[integrating the Config Query system|ConfigQueryIntegration]]
* [[identifying the basic Builder operations|BasicBuildingOperations]] and [[planning the Implementation|PlanningNodeCreatorTool]]
* [[how to handle »attached placement«|AttachedPlacementProblem]]
* working out the [[basic building situations|BuilderPrimitives]] and [[mechanics of rendering|RenderMechanics]]
* how to classify and [[describe media stream types|StreamType]] and how to [[use them|StreamTypeUse]]
* considerations regarding [[identity and equality|ModelObjectIdentity]] of objects in the HighLevelModel
* the [[identification of frames and nodes|NodeFrameNumbering]]
* the relation of [[Project, Timelines and Sequences|TimelineSequences]]
* how to [[start the GUI|GuiStart]] and how to [[communicate|GuiCommunication]]
* build the first LayerSeparationInterfaces
* create an uniform pattern for [[passing and accessing object collections|ForwardIterator]]
* decide on SessionInterface and create [[Session datastructure layout|SessionDataMem]]
* shaping the GUI/~Proc-Interface, based on MObjectRef and the [[Command frontend|CommandHandling]]
* defining PlacementScope in order to allow for [[discovering session contents|Query]]
* working out a [[Wiring concept|Wiring]] and the foundations of OutputManagement
* shaping the foundations of the [[player subsystem|Player]]
* detail considerations regarding [[time and time quantisation|TimeQuant]]
* [[Timecode]] -- especially the link of [[TC formats and quantisation|TimecodeFormat]]
* designing how to [[build|BuildFixture]] the [[Fixture]] (...{{red{WIP}}}...)
!Observations, Ideas, Proposals
''this page is a scrapbook for collecting ideas'' — please don't take anything noted here too literal. While writing code, I observe that I (ichthyo) follow certain informal guidelines, some of which I'd like to note down because they could evolve into general style guidelines for the Proc-Layer code.
* ''Inversion of Control'' is the leading design principle.
* but deliberately we stay just below the level of using Dependency Injection. Singletons and call-by-name are good enough. We're going to build //one// application, not any conceivable application.
* write error handling code only if the error situation can be actually //handled// at this place. Otherwise, be prepared for exceptions just passing by and thus handle any resources by "resource acquisition is initialisation" (RAII). Remember: error handling defeats decoupling and encapsulation.
* (almost) never {{{delete}}} an object directly, use {{{new}}} only when some smart pointer is at hand.
* clearly distinguish ''value objects'' from objects with ''reference semantics'', i.e. objects having a distict //object identity.//
* when user/client code is intended to create reference-semantics objects, make the ctor protected and provide a factory member called {{{create}}} instead, returning a smart pointer
* similarly, when we need just one instance of a given service, make the ctor protected and provide a factory member called {{{instance}}}, to be implemented by the lumiera::[[Singleton]] factory.
* whenever possible, prefer this (lazy initialised [[Singleton]]) approach and avoid static initialisation magic
* avoid doing anything non-local during the startup phase or shutdown phase of the application, especially avoid doing substantial work in any dtor.
* avoid asuming anything that can't be enforced by types, interfaces or signatures; this means: be prepared for open possibilities
* prefer {{{const}}} and initialisation code over assignment and active changes (inspired by functional programming)
* code is written for ''being read by humans''; code shall convey its meaning //even to the casual reader.//
/***
''InlineJavascriptPlugin for ~TiddlyWiki version 1.2.x and 2.0''
^^author: Eric Shulman - ELS Design Studios
source: http://www.TiddlyTools.com/#InlineJavascriptPlugin
license: [[Creative Commons Attribution-ShareAlike 2.5 License|http://creativecommons.org/licenses/by-sa/2.5/]]^^
Insert Javascript executable code directly into your tiddler content. Lets you ''call directly into TW core utility routines, define new functions, calculate values, add dynamically-generated TiddlyWiki-formatted output'' into tiddler content, or perform any other programmatic actions each time the tiddler is rendered.
!!!!!Usage
<<<
When installed, this plugin adds new wiki syntax for surrounding tiddler content with {{{<script>}}} and {{{</script>}}} markers, so that it can be treated as embedded javascript and executed each time the tiddler is rendered.
''Deferred execution from an 'onClick' link''
By including a label="..." parameter in the initial {{{<script>}}} marker, the plugin will create a link to an 'onclick' script that will only be executed when that specific link is clicked, rather than running the script each time the tiddler is rendered.
''External script source files:''
You can also load javascript from an external source URL, by including a src="..." parameter in the initial {{{<script>}}} marker (e.g., {{{<script src="demo.js"></script>}}}). This is particularly useful when incorporating third-party javascript libraries for use in custom extensions and plugins. The 'foreign' javascript code remains isolated in a separate file that can be easily replaced whenever an updated library file becomes available.
''Defining javascript functions and libraries:''
Although the external javascript file is loaded while the tiddler content is being rendered, any functions it defines will not be available for use until //after// the rendering has been completed. Thus, you cannot load a library and //immediately// use it's functions within the same tiddler. However, once that tiddler has been loaded, the library functions can be freely used in any tiddler (even the one in which it was initially loaded).
To ensure that your javascript functions are always available when needed, you should load the libraries from a tiddler that will be rendered as soon as your TiddlyWiki document is opened. For example, you could put your {{{<script src="..."></script>}}} syntax into a tiddler called LoadScripts, and then add {{{<<tiddler LoadScripts>>}}} in your MainMenu tiddler.
Since the MainMenu is always rendered immediately upon opening your document, the library will always be loaded before any other tiddlers that rely upon the functions it defines. Loading an external javascript library does not produce any direct output in the tiddler, so these definitions should have no impact on the appearance of your MainMenu.
''Creating dynamic tiddler content''
An important difference between this implementation of embedded scripting and conventional embedded javascript techniques for web pages is the method used to produce output that is dynamically inserted into the document:
* In a typical web document, you use the document.write() function to output text sequences (often containing HTML tags) that are then rendered when the entire document is first loaded into the browser window.
* However, in a ~TiddlyWiki document, tiddlers (and other DOM elements) are created, deleted, and rendered "on-the-fly", so writing directly to the global 'document' object does not produce the results you want (i.e., replacing the embedded script within the tiddler content), and completely replaces the entire ~TiddlyWiki document in your browser window.
* To allow these scripts to work unmodified, the plugin automatically converts all occurences of document.write() so that the output is inserted into the tiddler content instead of replacing the entire ~TiddlyWiki document.
If your script does not use document.write() to create dynamically embedded content within a tiddler, your javascript can, as an alternative, explicitly return a text value that the plugin can then pass through the wikify() rendering engine to insert into the tiddler display. For example, using {{{return "thistext"}}} will produce the same output as {{{document.write("thistext")}}}.
//Note: your script code is automatically 'wrapped' inside a function, {{{_out()}}}, so that any return value you provide can be correctly handled by the plugin and inserted into the tiddler. To avoid unpredictable results (and possibly fatal execution errors), this function should never be redefined or called from ''within'' your script code.//
''Accessing the ~TiddlyWiki DOM''
The plugin provides one pre-defined variable, 'place', that is passed in to your javascript code so that it can have direct access to the containing DOM element into which the tiddler output is currently being rendered.
Access to this DOM element allows you to create scripts that can:
* vary their actions based upon the specific location in which they are embedded
* access 'tiddler-relative' information (use findContainingTiddler(place))
* perform direct DOM manipulations (when returning wikified text is not enough)
<<<
!!!!!Examples
<<<
an "alert" message box:
{{{
<script>alert('InlineJavascriptPlugin: this is a demonstration message');</script>
}}}
<script>alert('InlineJavascriptPlugin: this is a demonstration message');</script>
dynamic output:
{{{
<script>return (new Date()).toString();</script>
}}}
<script>return (new Date()).toString();</script>
wikified dynamic output:
{{{
<script>return "link to current user: [["+config.options.txtUserName+"]]";</script>
}}}
<script>return "link to current user: [["+config.options.txtUserName+"]]";</script>
dynamic output using 'place' to get size information for current tiddler
{{{
<script>
if (!window.story) window.story=window;
var title=story.findContainingTiddler(place).id.substr(7);
return title+" is using "+store.getTiddlerText(title).length+" bytes";
</script>
}}}
<script>
if (!window.story) window.story=window;
var title=story.findContainingTiddler(place).id.substr(7);
return title+" is using "+store.getTiddlerText(title).length+" bytes";
</script>
creating an 'onclick' button/link that runs a script
{{{
<script label="click here">
if (!window.story) window.story=window;
alert("Hello World!\nlinktext='"+place.firstChild.data+"'\ntiddler='"+story.findContainingTiddler(place).id.substr(7)+"'");
</script>
}}}
<script label="click here">
if (!window.story) window.story=window;
alert("Hello World!\nlinktext='"+place.firstChild.data+"'\ntiddler='"+story.findContainingTiddler(place).id.substr(7)+"'");
</script>
loading a script from a source url
{{{
<script src="demo.js">return "loading demo.js..."</script>
<script label="click to execute demo() function">demo()</script>
}}}
where http://www.TiddlyTools.com/demo.js contains:
>function demo() { alert('this output is from demo(), defined in demo.js') }
>alert('InlineJavascriptPlugin: demo.js has been loaded');
<script src="demo.js">return "loading demo.js..."</script>
<script label="click to execute demo() function">demo()</script>
<<<
!!!!!Installation
<<<
import (or copy/paste) the following tiddlers into your document:
''InlineJavascriptPlugin'' (tagged with <<tag systemConfig>>)
<<<
!!!!!Revision History
<<<
''2006.01.05 [1.4.0]''
added support 'onclick' scripts. When label="..." param is present, a button/link is created using the indicated label text, and the script is only executed when the button/link is clicked. 'place' value is set to match the clicked button/link element.
''2005.12.13 [1.3.1]''
when catching eval error in IE, e.description contains the error text, instead of e.toString(). Fixed error reporting so IE shows the correct response text. Based on a suggestion by UdoBorkowski
''2005.11.09 [1.3.0]''
for 'inline' scripts (i.e., not scripts loaded with src="..."), automatically replace calls to 'document.write()' with 'place.innerHTML+=' so script output is directed into tiddler content
Based on a suggestion by BradleyMeck
''2005.11.08 [1.2.0]''
handle loading of javascript from an external URL via src="..." syntax
''2005.11.08 [1.1.0]''
pass 'place' param into scripts to provide direct DOM access
''2005.11.08 [1.0.0]''
initial release
<<<
!!!!!Credits
<<<
This feature was developed by EricShulman from [[ELS Design Studios|http:/www.elsdesign.com]]
<<<
!!!!!Code
***/
//{{{
version.extensions.inlineJavascript= {major: 1, minor: 4, revision: 0, date: new Date(2006,1,5)};
config.formatters.push( {
name: "inlineJavascript",
match: "\\<script",
lookahead: "\\<script(?: src=\\\"((?:.|\\n)*?)\\\")?(?: label=\\\"((?:.|\\n)*?)\\\")?\\>((?:.|\\n)*?)\\</script\\>",
handler: function(w) {
var lookaheadRegExp = new RegExp(this.lookahead,"mg");
lookaheadRegExp.lastIndex = w.matchStart;
var lookaheadMatch = lookaheadRegExp.exec(w.source)
if(lookaheadMatch && lookaheadMatch.index == w.matchStart) {
if (lookaheadMatch[1]) { // load a script library
// make script tag, set src, add to body to execute, then remove for cleanup
var script = document.createElement("script"); script.src = lookaheadMatch[1];
document.body.appendChild(script); document.body.removeChild(script);
}
if (lookaheadMatch[2] && lookaheadMatch[3]) { // create a link to an 'onclick' script
// add a link, define click handler, save code in link (pass 'place'), set link attributes
var link=createTiddlyElement(w.output,"a",null,"tiddlyLinkExisting",lookaheadMatch[2]);
link.onclick=function(){try{return(eval(this.code))}catch(e){alert(e.description?e.description:e.toString())}}
link.code="function _out(place){"+lookaheadMatch[3]+"};_out(this);"
link.setAttribute("href","javascript:;"); link.setAttribute("title",""); link.style.cursor="pointer";
}
else if (lookaheadMatch[3]) { // run inline script code
var code="function _out(place){"+lookaheadMatch[3]+"};_out(w.output);"
code=code.replace(/document.write\(/gi,'place.innerHTML+=(');
try { var out = eval(code); } catch(e) { out = e.description?e.description:e.toString(); }
if (out && out.length) wikify(out,w.output);
}
w.nextMatch = lookaheadMatch.index + lookaheadMatch[0].length;
}
}
} )
//}}}
/***
|''Name:''|InlineJavascriptPlugin|
|''Source:''|http://www.TiddlyTools.com/#InlineJavascriptPlugin|
|''Author:''|Eric Shulman - ELS Design Studios|
|''License:''|[[Creative Commons Attribution-ShareAlike 2.5 License|http://creativecommons.org/licenses/by-sa/2.5/]]|
|''~CoreVersion:''|2.0.10|
Insert Javascript executable code directly into your tiddler content. Lets you ''call directly into TW core utility routines, define new functions, calculate values, add dynamically-generated TiddlyWiki-formatted output'' into tiddler content, or perform any other programmatic actions each time the tiddler is rendered.
!!!!!Usage
<<<
When installed, this plugin adds new wiki syntax for surrounding tiddler content with {{{<script>}}} and {{{</script>}}} markers, so that it can be treated as embedded javascript and executed each time the tiddler is rendered.
''Deferred execution from an 'onClick' link''
By including a label="..." parameter in the initial {{{<script>}}} marker, the plugin will create a link to an 'onclick' script that will only be executed when that specific link is clicked, rather than running the script each time the tiddler is rendered.
''External script source files:''
You can also load javascript from an external source URL, by including a src="..." parameter in the initial {{{<script>}}} marker (e.g., {{{<script src="demo.js"></script>}}}). This is particularly useful when incorporating third-party javascript libraries for use in custom extensions and plugins. The 'foreign' javascript code remains isolated in a separate file that can be easily replaced whenever an updated library file becomes available.
''Display script source in tiddler output''
By including the keyword parameter "show", in the initial {{{<script>}}} marker, the plugin will include the script source code in the output that it displays in the tiddler.
''Defining javascript functions and libraries:''
Although the external javascript file is loaded while the tiddler content is being rendered, any functions it defines will not be available for use until //after// the rendering has been completed. Thus, you cannot load a library and //immediately// use it's functions within the same tiddler. However, once that tiddler has been loaded, the library functions can be freely used in any tiddler (even the one in which it was initially loaded).
To ensure that your javascript functions are always available when needed, you should load the libraries from a tiddler that will be rendered as soon as your TiddlyWiki document is opened. For example, you could put your {{{<script src="..."></script>}}} syntax into a tiddler called LoadScripts, and then add {{{<<tiddler LoadScripts>>}}} in your MainMenu tiddler.
Since the MainMenu is always rendered immediately upon opening your document, the library will always be loaded before any other tiddlers that rely upon the functions it defines. Loading an external javascript library does not produce any direct output in the tiddler, so these definitions should have no impact on the appearance of your MainMenu.
''Creating dynamic tiddler content''
An important difference between this implementation of embedded scripting and conventional embedded javascript techniques for web pages is the method used to produce output that is dynamically inserted into the document:
* In a typical web document, you use the document.write() function to output text sequences (often containing HTML tags) that are then rendered when the entire document is first loaded into the browser window.
* However, in a ~TiddlyWiki document, tiddlers (and other DOM elements) are created, deleted, and rendered "on-the-fly", so writing directly to the global 'document' object does not produce the results you want (i.e., replacing the embedded script within the tiddler content), and completely replaces the entire ~TiddlyWiki document in your browser window.
* To allow these scripts to work unmodified, the plugin automatically converts all occurences of document.write() so that the output is inserted into the tiddler content instead of replacing the entire ~TiddlyWiki document.
If your script does not use document.write() to create dynamically embedded content within a tiddler, your javascript can, as an alternative, explicitly return a text value that the plugin can then pass through the wikify() rendering engine to insert into the tiddler display. For example, using {{{return "thistext"}}} will produce the same output as {{{document.write("thistext")}}}.
//Note: your script code is automatically 'wrapped' inside a function, {{{_out()}}}, so that any return value you provide can be correctly handled by the plugin and inserted into the tiddler. To avoid unpredictable results (and possibly fatal execution errors), this function should never be redefined or called from ''within'' your script code.//
''Accessing the ~TiddlyWiki DOM''
The plugin provides one pre-defined variable, 'place', that is passed in to your javascript code so that it can have direct access to the containing DOM element into which the tiddler output is currently being rendered.
Access to this DOM element allows you to create scripts that can:
* vary their actions based upon the specific location in which they are embedded
* access 'tiddler-relative' information (use findContainingTiddler(place))
* perform direct DOM manipulations (when returning wikified text is not enough)
<<<
!!!!!Examples
<<<
an "alert" message box:
><script show>
alert('InlineJavascriptPlugin: this is a demonstration message');
</script>
dynamic output:
><script show>
return (new Date()).toString();
</script>
wikified dynamic output:
><script show>
return "link to current user: [["+config.options.txtUserName+"]]";
</script>
dynamic output using 'place' to get size information for current tiddler:
><script show>
if (!window.story) window.story=window;
var title=story.findContainingTiddler(place).id.substr(7);
return title+" is using "+store.getTiddlerText(title).length+" bytes";
</script>
creating an 'onclick' button/link that runs a script:
><script label="click here" show>
if (!window.story) window.story=window;
alert("Hello World!\nlinktext='"+place.firstChild.data+"'\ntiddler='"+story.findContainingTiddler(place).id.substr(7)+"'");
</script>
loading a script from a source url:
>http://www.TiddlyTools.com/demo.js contains:
>>{{{function demo() { alert('this output is from demo(), defined in demo.js') } }}}
>>{{{alert('InlineJavascriptPlugin: demo.js has been loaded'); }}}
><script src="demo.js" show>
return "loading demo.js..."
</script>
><script label="click to execute demo() function" show>
demo()
</script>
<<<
!!!!!Installation
<<<
import (or copy/paste) the following tiddlers into your document:
''InlineJavascriptPlugin'' (tagged with <<tag systemConfig>>)
<<<
!!!!!Revision History
<<<
''2006.06.01 [1.5.1]'' when calling wikify() on script return value, pass hightlightRegExp and tiddler params so macros that rely on these values can render properly
''2006.04.19 [1.5.0]'' added 'show' parameter to force display of javascript source code in tiddler output
''2006.01.05 [1.4.0]'' added support 'onclick' scripts. When label="..." param is present, a button/link is created using the indicated label text, and the script is only executed when the button/link is clicked. 'place' value is set to match the clicked button/link element.
''2005.12.13 [1.3.1]'' when catching eval error in IE, e.description contains the error text, instead of e.toString(). Fixed error reporting so IE shows the correct response text. Based on a suggestion by UdoBorkowski
''2005.11.09 [1.3.0]'' for 'inline' scripts (i.e., not scripts loaded with src="..."), automatically replace calls to 'document.write()' with 'place.innerHTML+=' so script output is directed into tiddler content. Based on a suggestion by BradleyMeck
''2005.11.08 [1.2.0]'' handle loading of javascript from an external URL via src="..." syntax
''2005.11.08 [1.1.0]'' pass 'place' param into scripts to provide direct DOM access
''2005.11.08 [1.0.0]'' initial release
<<<
!!!!!Credits
<<<
This feature was developed by EricShulman from [[ELS Design Studios|http:/www.elsdesign.com]]
<<<
!!!!!Code
***/
//{{{
version.extensions.inlineJavascript= {major: 1, minor: 5, revision: 1, date: new Date(2006,6,1)};
config.formatters.push( {
name: "inlineJavascript",
match: "\\<script",
lookahead: "\\<script(?: src=\\\"((?:.|\\n)*?)\\\")?(?: label=\\\"((?:.|\\n)*?)\\\")?( show)?\\>((?:.|\\n)*?)\\</script\\>",
handler: function(w) {
var lookaheadRegExp = new RegExp(this.lookahead,"mg");
lookaheadRegExp.lastIndex = w.matchStart;
var lookaheadMatch = lookaheadRegExp.exec(w.source)
if(lookaheadMatch && lookaheadMatch.index == w.matchStart) {
if (lookaheadMatch[1]) { // load a script library
// make script tag, set src, add to body to execute, then remove for cleanup
var script = document.createElement("script"); script.src = lookaheadMatch[1];
document.body.appendChild(script); document.body.removeChild(script);
}
if (lookaheadMatch[4]) { // there is script code
if (lookaheadMatch[3]) // show inline script code in tiddler output
wikify("{{{\n"+lookaheadMatch[0]+"\n}}}\n",w.output);
if (lookaheadMatch[2]) { // create a link to an 'onclick' script
// add a link, define click handler, save code in link (pass 'place'), set link attributes
var link=createTiddlyElement(w.output,"a",null,"tiddlyLinkExisting",lookaheadMatch[2]);
link.onclick=function(){try{return(eval(this.code))}catch(e){alert(e.description?e.description:e.toString())}}
link.code="function _out(place){"+lookaheadMatch[4]+"};_out(this);"
link.setAttribute("href","javascript:;"); link.setAttribute("title",""); link.style.cursor="pointer";
}
else { // run inline script code
var code="function _out(place){"+lookaheadMatch[4]+"};_out(w.output);"
code=code.replace(/document.write\(/gi,'place.innerHTML+=(');
try { var out = eval(code); } catch(e) { out = e.description?e.description:e.toString(); }
if (out && out.length) wikify(out,w.output,w.highlightRegExp,w.tiddler);
}
}
w.nextMatch = lookaheadMatch.index + lookaheadMatch[0].length;
}
}
} )
//}}}
An RAII class, used to manage a [[facade interface between layers|LayerSeparationInterface]].
The InstanceHandle is created by the service implementation and will automatically
* either register and open a ''C Language Interface'', using Lumiera's InterfaceSystem
* or load and open a LumieraPlugin, also by using the InterfaceSystem
* and optionally also create and manage a facade proxy to allow transparent access for client code
→ see [[detailed description here|LayerSeparationInterfaces]]
Because we rely on strong decoupling and separation into self contained components, there is not much need for a common quasi-global namespace. Operations needing the cooperation of another subsystem will be delegated or even dispatched, consequently implementation code needs only the service acces points from "direct cooperation partner" subsystems. Hierarchical scopes besides classes are needed only when multiple subsystems share a set of common abstractions. Interface and Implementation use separate namespaces.
!common definitions
# surely there will be the need to use some macros (albeit code written in C++ can successfully avoid macros to some extent)
# there will be one global ''Application State'' representation (and some application services)
# we have some lib facilities, especially a common [[Time]] abstraction
For these we have one special (bilingual) header file __lumiera.h__, which places in its C++ part some declarations into __namespace lumiera__. These declarations should be pulled into the specific namespaces or (still better) into the implementation //on demand//. There is no nesting (we deliberately don't want an symbol appearing //automatically// in every part of the system).
!subsystem interface and facade
These large scale interfaces reside in special namespaces "~XXX_interface" (where XXX is the subsystem). The accompanning //definitions// depend on the implementation namespace and are placed in the top-level source folder of the corresponding subsystem.
→ Example: [[Interfaces/Namespaces of the Session Subsystem(s)|InterfacesSession]]
!contract checks and test code
From experiences with other middle scale projects, I prefer having the test code in a separate tree (because test code easily doubles the number of source files). But of course it should be placed into the same namespace as the code being checked, or (better?) into a nested namespace "test". It is esp. desirable to have good coverage on the contracts of the subsystem interfaces and mayor components (while it is not always feasible or advisable to cover every implementation detail).
→ see also [[testsuite documentation in the main wiki|index.html#TestSuite]]
* Subdir src/proc contains Interface within namespace proc_interface
* Subdir src/proc/mobject contains commonly used entities (namespace mobject)
** nested namespace controller
** nested namespace builder
** nested namespace session
* Subdir src/proc/engine (namespace engine) uses directly the (local) interface components StateProxy and ParamProvider; triggering of the render process is initiated by the controller and can be requested via the controller facade. Normally, the playback/render controller located in the backend will just use this interface and won't be aware of the build process at all.
[img[Example: Interfaces/Namespaces of the ~Session-Subsystems|uml/fig130053.png]]
A major //business interface// — used by the layers for interfacing to each other; also to be invoked externally by scripts.
→ [[overfiew and technical details|LayerSeparationInterfaces]]
Lumiera uses a 3-layered architecture. Separation between layers is crucial. Any communication between the layers regarding the normal operation of the application should //at least be initiated// through ''Layer abstraction interfaces'' (Facade Interfaces). This is a low-impact version of layering, because, aside from this triggering, direct cooperation of parts within the single Lumiera process is allowed, under the condition that is is implemented using additional abstractions (interfaces with implementation level granularity). We stick to the policy of //disallowing direct coupling of implementations located in different layers.//
[>img[Anatomy of a Layer Separation Interface|uml/fig132869.png]]
The goal is for the interface to remain fairly transparent for the client and to get an automatic lifecycle management.
To implement such a structure, each layer separation interface actually is comprised of several parts:
* an C Language Interface ("''CL Interface''") to be installed into the InterfaceSystem
* a ''facade interface'' defining the respective abstractions in terms of the client side impl. language (C or C++)
* a ''service implementation'' directly addressed by the implementation instantiated within the InterfaceSystem
* a ''facade proxy'' object on the client side, which usually is given inline alongside with the CL interface definition.
!opening and closing
Handling the lifecycle can be tricky, because client- and service-side need to carry out the opening and closing operations in sync and observing a specific call order to ensure calls get blocked already on the client side unless the whole interface compound is really up and running. To add to this complexity, plugins and built-in interfaces are handled differently regarding the question who is in charge of the lifecycle: interfaces are installed on the service side, whereas the loading of plugins is triggered from client side by requesting the plugin from the loader.
Anyway, interfaces are resources which best should be managed automatically. At least within the C++ part of the application we can ensure this by using the InstanceHandle template. This way the handling of plugins and interfaces can be unified and the opening and closing of the facade proxy happens automatically.
The general idea is, that each facade interface actually provides access to a specific service; there will always be a single implementation object somewhere, which can be thought of as acting as "the" service. This service-providing object will then contain the mentioned InstanceHandle; thus, its lifecycle becomes identical with the service lifecycle.
* when the service relies on a [[plugin|LumieraPlugin]], this service providing object (containing the InstanceHandle) needs to sit at the service accessing side (as the plugin may not yet be loaded and someone has to pull it up).
* otherwise, when the service is just an interface to an already loaded facility, the service providing object (containing the InstanceHandle) will live on the service providing side, not the client side. Then the ctor of the service providing object needs to be able to access an interface instance definition (→ InterfaceSystem); the only difference to the plugin case is to create the InstanceHandle with a different ctor, passing the interface and the implementing instance.
!outline of the major interfaces
|!Layer|>|!Interface |
|GUI|GuiFacade|UI lifecycle → GuiStart|
|~|GuiNotificationFacade|status/error messages, asynchronous object status change notifications, trigger shutdown|
|~|DisplayFacade|pushing frames to a display/viewer|
|Proc|SessionFacade|session lifecycle|
|~|EditFacade|edit operations, object mutations|
|~|PlayerDummy|player mockup, maybe move to backend?|
|//Lumiera's major interfaces//|c
An implementation facility used by the session manager implementation to ensure a consistent lifecycle. A template-method-like skeleton of operations can be invoked by the session manager to go through the various lifecycle stages, while the actual implementation functionality is delegated to facilities within the session. The assumption is for the lifecycle advisor to be executed within a controlled environment, a single instance and single threaded.
---------------------
!Implementation notes
The goal on the implementation level is to get a consolidated view on the whole lifecycle.
Reading the source code should convey a complete picture about what is going on with respect to the session lifecycle.
!!entrance points
;close
: the shutdown sequence cleanly unwinds any contents and returns into //uninitialised state.//
;reset
: is comprised of two phases: shutdown and startup of an empty new session
: shutdown will be skipped automatically when uninitialised
: startup has to inject default session content
;load
: again, first an existing session is brought down cleanly
: the startup sequence behaves similarly to __reset__, but injects serialised content
Note that, while causing a short //freeze period,// __saving__ and __(re-)building__ aren't considered lifecycle operations
!!operation sequences
The ''pull up'' sequence performs basic initialisation of the session facilities and then, after the pImpl-switch, executes the various loading and startup phases. Any previous session switched away is assumed to unwind automatically, nothing is done especially to disconnect such an existing session, we simply require the session subsystem to be in a pristine state initially.
The ''shut down'' sequence does exactly that: halt processing and rendering, disconnect an existing session, if any, get back into initial state. It doesn't care for unwinding session contents, which is assumed to happen automatically when references to previous session contents go out of scope.
Opening and accessing media files on disk poses several problems, most of which belong to the domain of Lumiera's data backend. Here, we focus on the questions related to making media data available to the session and the render engine. Each media will be represented by an MediaAsset object, which indeed could be a compound object (in case of MultichannelMedia). Building this asset object thus includes getting information from the real file on disk. For delegating this to the backend, we use the following query interface:
* {{{queryFile(char* name)}}} requests accessing the file and yields some (opaque) handle when successful.
* {{{queryChannel(fHandle, int)}}} will then be issued in sequence with ascending index numbers, until it returns {{{NULL}}}.
* the returned struct (pointer) will provide the following information:
** some identifier which can be used to create a name for the corresponding media (channel) asset
** some identifier characterizing the access method (codec) needed to get at the media data. This should be rather a high level description of the media stream type, e.g. "H264"
** some (opaque) handle usable for accessing this specific stream. When the render engine later on pulls data for this channel, it will pass this handle down to the backend.
{{red{to be defined in more detail later...}}}
→ see "~MediaAccessFacade" for a (preliminary) interface definitioin
→ see "~MediaAccessMock" for a mock/test implementaion
Used to actually implement the various kinds of [[Placement]] of ~MObjects. ~LocatingPin is the root of a hierarchy of different kinds of placing, constraining and locating a Media Object. Basically, this is an instance of the ''state pattern'': The user sees one Placement object with value semantics, but when the properties of the Placement are changed, actually a ~LocatingPin object (or rather a chain of ~LocatingPins) is changed within the Placement. Subclasses of ~LocatingPin implement different placing/constraining behaviour:
* {{{FixedLocation}}} places a MObject to a fixed temporal position and track
* {{{RelativeLocation}}} is used to atach the MObject to some other anchor MObject
* //additional constraints, placement objectives, range restrictions, pattern rules will follow...//
All sorts of "things" to be placed and manipulated by the user in the session. This interface abstracts the details and just supposes
* the media object has a duration
* it is allways //placed// in some manner, i.e. it is allways accessed via a [[Placement]]
* {{red{and what else?}}}
→ [[overview of the MObject hierarchy|MObjects]]
''The Problem of referring to an [[MObject]]'' stems from the object //as a concept// encompassing a wider scope then just the current implementation instance. If the object was just a runtime entity in memory, we could use a simple (language) reference or pointer. Actually, this isn't sufficient, as the object reference will pass LayerSeparationInterfaces, will be handed over to code not written in the same implementation language, will be included in an ''UNDO'' record for the UndoManager, and thus will need to be serialized and stored permanently within the SessionStorage.
Moreover [[MObject instances|MObject]] have a 2-level structure: the core object holds just the properties in a strict sense, i.e. the properties which the object //owns.// Any properties due to putting the object into a specific context, i.e. all relation properties are represented as [[Placement]] of the object. Thus, when viewed from the client side, a reference to a specific ~MObject //instance,// actually denotes a //specific// Placement of this object into the Session.
!Requirements
* just the reference allone is sufficient to access the placement //and// the core object.
* the reference needs to be valid even after internal restructuring in the object store.
* there must be a way to pass these references through serialisation/deserialisation
* we need a plain-C representation of the reference, which ideally should be incorporated into the complete implementation
* references should either be integrated into memory management, or at least it should be possible to detect dangling references
* it should be possible to make handling of the references completely transparent, if the implementation language supports doing so.
!Two models
For the implementation of the object references, as linked to the memory management in general, there seem to be the following approaches
* the handle is a simple table index. Consequently we'll need a garbage collection to deal with deleted objects. This model has several flavours
** using an generation ID to tag the index handle, keep an translation table on each GC, mapping old indices to new ones and raise an error when encountering an outdated handle
** use a specific datastructure allowing the handles to remain valid in case of cleanup/reorganisation. (Hashtable or similar)
** keep track of all index handles and rewrite them on GC
* handles are refcounting smart pointers. This solution is C++ specific, albeit more elegant and minimalistic. As all the main interfaces should be accessible via C bindings, we'd need to use a replacement mechanism on the LayerSeparationInterfaces, which could be mapped to or handled by an C++ smart-ptr. (We used a similar approach for the PlayerDummy design study)
Obviously, the second approach has quite some appeal — but, in order to use it, we'd have to mitigate its drawbacks: it bears the danger of creating a second separate code path for C language based clients, presumably receiving lesser care, maintenance and stress testing. The mentioned solution worked out earlier this year for the player mockup (1/2009) tries at least partially to integrate the C functionality, as the actual implementation is derived from the C struct used as handle, thus allowing to use the same pointers for both kinds of interface, and in turn by doing the de-allocation by a call through the C dtor function, which is installed as deleter function with the boost::smart_ptr used as base class of the handle. Conceptually, the interface on this handle is related to the actual implementation refered to by the handle (handle == smart_ptr) as if the latter was a subclass of the former. But on the level of the implementation, there is no inheritance relation between the handle and the referent, and especially this allows to define the handle's interface in terms of abstract interface types usable on the client side, while the referent's interface operates on the types of the service implementation. Thus, the drawback of using a C language interface is turned into the advantage of completely separating implementation and client.
!Implementation concept
Presumably, none of the both models is usable as-is; rather we try to reconstruct the viable properties of both, starting out with the more elegant second model. Thus, basically the ''reference is a smart-ptr'' referring to the core object. Additionally, it incorporates a ''systematic ID denoting the location of the placement''. This ID without the smart-ptr part is used for the C-implementation, making the full handle implementation a shortcut for an access sequence, which first querries the placement from the Session, followed by dereferencing the placement to get at the core object. Thus, the implementation builds upon another abstraction, the → PlacementRef, which in turn assumes for an index within the implementation of the [[session datastructure|SessionDataMem]] to track and retrieve the actual Placement.
[img[Structure of MObjectRef and PlacementRef|uml/fig136581.png]]
!using ~MObject references
~MObject references have a distinct lifecycle: usually, they are created //empty// (invalid ref, inactive state), followed by activating them by attachment to an existing placement within the session. Later on, the reference can be closed (detached, deactivated). Activation can be done either directly by a {{{Placement<MO>&}}}, or indirectly by any {{{Placement::ID}}} tag or even plain LUID denoting a placement known to the PlacementIndex embedded within the [[Session]]. Activation can fail, because the validity of the reference is checked. In this respect, they behave exactly like PlacementRef, and indeed are implemented on top of the latter. From this point onward, the referred ~MObject is kept alive by the reference — but note, this doesn't extend to the placement, which still may be modified or removed from the session without further notice. {{red{TODO investigate synchronisation guarantees or a locking mechanism}}} Thus, client code should never store direct references to the placement.
~MObject references have value semantics, i.e. they don't have an identity and can be copied freely. Dereferencing yields a direct (language) ref to the MObject, while the placement can be accessed by a separate function {{{getPlacement()}}}. Moreover, the MObjectRef instance provides a direct API to some of the most common query functions you could imagine to call on both the object and the placement (i.e. to find out about the start time, length, ....)
The ~MObjects Subsystem contains everything related to the [[Session]] and the various Media Objects placed within. It is complemented by the Asset Management (see → [[Asset]]). Examples for [[MObjects |MObject]](→ def) being:
* audio/video clips
* [[effects and plugins|EffectHandling]]
* special facilities like mask and projector
* [[Automation]] sets
* labels and other (maybe functional) markup
This Design strives to achieve a StrongSeparation between the low-level Structures used to carry out the actual rendering and the high level Entities living in the session and being manipulated by the user. In this high level view, the Objects are grouped and located by [[Placements|Placement]], providing a flexible and open way to express different groupings, locations and ordering constraints between the Media Objects.
→ EditingOperations
→ PlacementHandling
→ SessionOverview
[img[Classess related to the session|uml/fig128133.png]]
The HighLevelModel consists of MObjects, which are attached to one another through their [[Placement]]. While this is a generic scheme to arrange objects in a tree of [[scopes|PlacementScope]], some attachments are handled specifically and may trigger side-effects
{{red{drafted feature as of 6/2010}}}
* a [[binding|BindingMO]] attached to root is linked to a [[Timeline]]
* a [[Track]] attached to root corresponds to a [[Sequence]]
→ see ModelDependencies
''[[Lumiera|index.html]]''
[[Proc-Layer|ProcLayer and Engine]]
[[MObjects]]
[[Implementation|ImplementationDetails]]
[[Admin]]
<<fullscreen>>
Problem is: when removing an Asset, all corresponding MObjects need to disappear. This means, besides the obvious ~Ref-Link (MObject referring to an asset) we need backlinks or a sort of registry. And still worse: we need to remove the affected MObject from the object network in the session and rebuild the Fixture...
→ for a general design discussion see [[Relation of Clip and Asset|RelationClipAsset]]
//Currently// Ichthyo considers the following approach:
* all references between MObjects and Assets are implemented as __refcounting__ boost::shared_ptr
* the opposite direction is also a __strong reference__, effectively keeping the clip-MO alive even if it is no longer in use in the session (note this is a cyclic dependency that needs to be actively broken on deletion). This design decision is based on logical considerations (→ see "deletions, Model-2" [[here|RelationClipAsset]]). This back-link is implemented by a Placement which is stored internally within the asset::Clip, it is needed for clean deletion of Assets, for GUI search functions and for adding the Clip to the session (again after been removed, or multiple times as if cloned).
* MObjects and Assets implement an {{{unlink()}}} function releasing any internal links causing circular dependencies. It is always implemented such as to drop //optional// associations while retaining those associations mandatory for fulfilling the objects contract.
* Instead of a delete, we call this unlink() function and let the shared_ptr handle the actual deletion. Thus, even if the object is already unlinked, it is still valid and usable as long as some other entity holds a smart-ptr. An ongoing render process for example can still use a clip asset and the corresponding media asset linked as parent, but this media asset's link to the dependant clip has already been cleared (and the media is no longer registered with the AssetManager of course).
* so the back-link from dependant asset up to the parent asset is mandatory for the child asset to be functional, but preferably it should be {{{const}}} (only used for information retrieval)
* the whole hierarchy has to be crafted accordingly, but this isn't much of a limitation
* we don't use a registry, rather we model the real dependencies by individual dependency links. So a MediaAsset gets links to all Clips created from this Asset and by traversing this tree, we can handle the deletion
* after the deletion, the Fixture needs to be rebuilt.
* but any render processes still can have pointers to the Asset to be removed, and the shared_ptr will ensure, that the referred objects stay alive as long as needed.
{{red{let's see if this approach works...}}}
Contrary to the →[[Assets and MObjects|ManagementAssetRelation]], the usage pattern for [[render nodes|ProcNode]] is quite simple: All nodes are created together every time a new segment of the network is being build and will be used together until this segment is re-built, at which point they can be thrown away altogether. While it would be easy to handle the nodes automatically by smart-ptr (the creation is accessible only by use of the {{{NodeFactory}}} anyways), it //seems advisable to care for a bulk allocation/deallocation here.// The reason being not so much the amount of memory (which is expected to be moderate), but the fact the build process can be triggered repeatedly several times a second when tweaking the session, which could lead to fragmentation and memory pressure.
__10/2008__: the allocation mechanism can surely be improved later, but for now I am going for a simple implementation based on heap allocated objects owned by a vector or smart-ptrs. For each segment of the render nodes network, we have several families of objects, each of with will be maintained by a separate low-level memory manager (as said, for now implemented as vector of smart-ptrs). Together, they form an AllocationCluster; all objects contained in such a cluster will be destroyed together.
<!--{{{-->
<link rel='alternate' type='application/rss+xml' title='RSS' href='index.xml'/>
<!--}}}-->
<style type="text/css">#contentWrapper {display:none;}</style><div id="SplashScreen" style="border: 3px solid #ccc; display: block; text-align: center; width: 320px; margin: 100px auto; padding: 50px; color:#000; font-size: 28px; font-family:Tahoma; background-color:#eee;">loading <b>Proc-Layer</b> devel doku<blink> ...</blink><br><br><span style="font-size: 14px; color:red;">Requires Javascript.</span></div>
The Interface asset::Media is a //key abstraction// It ties together several concepts and enables to deal with them on the interfaces in a uniform manner. Besides, like every Asset kind, it belongs rather to the bookkeeping view: an asset::Media holds the specific properties and parametrisation of the media source it stands for. Regarding the __inward interface__ — as used from within the [[model|HighLevelModel]] or the [[render nodes|ProcNode]], it is irrelevant if any given asset::Media object stands for a complete media source, just a clip taken from this source or if a placeholder version of the real media source is used instead.
[img[Asset Classess|uml/fig130437.png]]
{{red{NOTE 3/2010:}}} Considering to change that significantly. Especially considering to collapse clip-asset and clip-MO into a single entity with multiple inheritance
→ regarding MultichannelMedia (see the notes at bottom)
→ see also LoadingMedia
The Proc-Layer is designed such as to avoid unnecessary assumptions regarding the properties of the media data and streams. Thus, for anything which is not completely generic, we rely on an abstract [[type description|StreamTypeDescriptor]], which provides a ''Facade'' to an actual library implementation. This way, the fundamental operations can be invoked, like allocating a buffer to hold media data.
In the context of Lumiera and especially in the Proc-Layer, __media implementation library__ means
* a subsystem which allows to work with media data of a specific kind
* such as to provide the minimal set of operations
** allocating a frame buffer
** describing the type of the data within such a buffer
* and this subsystem or external library has been integrated to be used through Lumiera by writing adaptation code for accessing these basic operations through the [[implementation facade interface|StreamTypeImplFacade]]
* such a link to an type implementation is registered and maintained by the [[stream type manager|STypeManager]]
!Problem of the implementation data types
Because we deliberately won't make any asumptions about the implementation library (besides the ones imposed indirectly by the facade interface), we can't integrate the data types of the library first class into the type system. All we can do is to use marker types and rely on the builder to have checked the compatibility of the actual data beforehand.
It would be possible to circumvent this problem by requiring all supported implementation libraries to be known at compile time, because then the actual media implementation type could be linked to a facade type by generic programming. Indeed, Lumiera follows this route with regards to the possible kinds of MObject or [[Asset]] — but to the contraty, for the problem in question here, being able to include support for a new media data type just by adding a plugin by far outweights the benefits of compile-time checked implementation type selection. So, as a consequence of this design decision we //note the possibility of the media file type discovery code to be misconfigured// and select the //wrong implementation library at runtime.// And thus the render engine needs to be prepared for the source reading node of any pipe to flounder completely, and protect the rest of the system accordingly
Of course: Cinelerra currently leaks memory and crashes regularilly. For the newly written code, besides retaining the same level of performance, a main goal is to use methods and techniques known to support the writing of quality code. So, besides the MultithreadConsiderations, a solid strategy for managing the ownership of allocated memory blocks is necessary right from start.
!Problems
# Memory management needs to work correct in a //fault tolerant environment//. That means that we need to be prepared to //handle on a non-local scale// some sorts of error conditions (without aborting the application). To be more precise: some error condition arises locally, which leads to a local abort and just the disabling/failing of some subsystem without affecting the application as a whole. This can happen on a regular base (e.g. rendering fails) and thus is __no excuse for leaking memory__
# Some (not all) parts of the core application are non-deterministic. That means, we can't tie the memory management to any assumptions on behalf of the execution path
!C++ solution
First of all -- this doesn't concern //every// allocation. It rather means there are certain //dangerous areas// which need to be identified. Anyhow, instead of carrying inherent complexities of the problem into the solution, we should rather look for common solution pattern(s) which help factoring out complexity.
For the case here in question this seems to be the ''resource allocation is construction'' pattern. Which boils down to basically never using bare pointers when concerned with ownership. Instead, ownership should be handled by smart-pointers.
!!usage scenarios
# __existence is being used__: Objects just live for being referred to in a object network. In this case, use refcounting smart-pointers for every ref. (note: problem with cyclic refs)
# __entity bound ownership__: Objects can be tied to some long living entity in the program, which holds the smart-pointer
#* if the existence of these ref-holding entity can be //guaranteed// (as if by contract), then the other users can build a object network with conventional pointers
#* otherwise, when the ref-holding entity //can disappear// in a regular program state, we need weak-refs and checking (because by our postulate the controlled resource needs to be destructed immediately, otherwise we would have the first case, existence == being used)
!!!dangerous uses
* the render nodes → [[detail analysis|ManagementRenderNodes]] {{red{TODO}}}
* the MObjects in the session → [[detail analysis|ManagementMObjects]] {{red{TODO}}}
* Asset - MObject relationship. → [[detail analysis|ManagementAssetRelation]] {{red{TODO}}}
!!!rather harmless
* Frames (buffers), because they belong to a given [[RenderProcess (=StateProxy)|StateProxy]] and are just passed in into the individual [[ProcNode]]s. This can be handled consistently with conventional methods.
* each StateProxy belongs to one top-level call to the [[Controller-Facade|Controller]]
* similar for the builder tools, which belong to a build process. Moreover, they are pooled and reused.
* the [[sequences|Sequence]] and the defined [[assets|Asset]] belong together to one [[Session]]. If the Session is closed, this means a internal shutdown of the whole ProcLayer, i.e. closing of all GUI representations and terminating all render processes. If these calles are implemented as blocking operations, we can assert that as long as any GUI representation or any render process is running, there is a valid session and model.
!using Factories
And, last but not least, doing large scale allocations is the job of the backend. Exceptions being long-lived objects, like the session or the sequences, which are created once and don't bear the danger of causing memory pressure. Besides, the ProcLayer code shouldn't issue "new" and "delete" when it comes in hand, rather it should use some centralized [[Factories]] for all allocation and freeing, so we can redirect these calls down to the backend, which may use pooling or special placement allocators or the like. The rationale is, for modern hardware/architectures, care has to be taken with heap allocations, esp. with many small objects and irregular usage patterns.
This category is comprised of the various aspects of the way the application controls and manages its own behaviour. They are more related to the way the application behaves, as opposed to the way the edited data is structured and organised (which is the realm of [[structural assets|StructAsset]]
* StreamType → a type system for describing and relating media data streams
* ScaleGrid → to manage time scales and frame alignment
!accessing meta assets
It turns out that all meta assets follow a distinct usage pattern: //they aren't built as individual entities.// Either, they are introduced into the system as part of a larger scale configuration activity, or they are //derived from category.// The latter fits in with a prototype-like approach; initially, the individual entry just serves to keep track of a categorisation, while at some point, such a link into a describing category may evolve into a local differentiation of some settings.
Another distinct property of meta assets is to be just a generic front-end to some very specific data entry, which needs to be allocated and maintained and provided on demand. Consider for example configuration rules, which have both a textual and an AST representation and will be assembled and composed into an effective rule set, depending on usage scope. Another example would be the enormous amount of parameter data created by parameter automation in the session. While certainly the raw data needs to be stored and retrieved somehow, the purpose of the corresponding meta asset is to access and manipulate this data in a structured and specific fashion.
!!!self referential structure
These observation leads to a design relying on a self referential structure: each meta asset is a {{{meta::Descriptor}}}. In the most basic version -- as provided by the generic implementation by {{{asset::Meta}}}, this descriptor is just the link to another descriptor, which represents a category. Thus, meta assets are created or accessed by
* just an EntryID, which implicitly also establishes a type, the intent being "get me yet another of this kind"
* a descriptor and an EntryID, to get a new element with a more distinct characterisation.
!!!mutating meta assets
Meta assets are ''immutable'' -- but they can be //superseded.//
For each meta asset instance, initially a //builder// is created for setting up the properties; when done, the builder will "drop off" the new meta asset instance. The same procedure is used for augmenting or superseding an existing element.
Lumiera's Proc-Layer is built around //two interconnected models,// mediated by the [[Builder]]. Basically, the →[[Session]] is an external interface to the HighLevelModel, while the →RenderEngine operates the structures of the LowLevelModel.
Our design of the models (both [[high-level|HighLevelModel]] and [[low-level|LowLevelModel]]) relies partially on dependent objects being kept consitently in sync. Currently (2/2010), __ichthyo__'s assessment is to consider this topic not important and pervasive enough to justify building a dedicated solution, like e.g. a central tracking and registration service. An important point to consider with this assesment is the fact that the session implementation is beeing kept mostly single-threaded. Thus, lacking one central place to handle this issue, care has to be taken to capture and treat all the relevant individual dependencies properly at the implementation level.
!known interdependencies
[>img[Fundamental object relations used in the session|uml/fig136453.png]]
* the session API relies on two kinds of facade like assets: [[Timeline]] and [[Sequence]], linked to the BindingMO and Track objects within the model respectively.
* conceptually, the DefaultsManagement and the AssetManager count as being part of the [[global model scope|ModelRootMO]], but, due to their importance, these facilities are accessible through an singleton interface.
* currently as of 2/2010 the exact dependency of the automation calculation during the render process onto the automation definitions within the HighLevelModel remains to be specified.
!!Timelines and Sequences
While implemented as StructAsset, additionally we need to assure every instance gets linked to the relevant parts of the model and registered with the session. Contrast this with other kinds of assets, which may just remain enlisted, but never actually used.
;the Session
:...is linked 1:1 with timelines and sequences. Registration and deregistration is directly tied to creation and destruction.
: __created__ ⇒ default timeline
: __destroy__ ⇒ discard all timelines, discard all sequences
;Timeline
:acts as facade and is implemented by an root-attached BindingMO. Can't exist in isolation.
: __created__ ⇒ create a dedicated new binding, either useing an existing sequence, or a newly created empty sequence
: __destroy__ ⇒ remove binding, while the previously bound sequence remains in model.
;root-placed Binding
:while generally a Binding can exist in the model, when attached to root, a Timeline will be created
: __created__ ⇒ an existing sequence might be given on creation, otherwhise a default configured sequence is created
: __destroy__ ⇒ implemented by detaching from root (see below) prior to purging from the model.
: __attached__ to root ⇒ invoke Timeline creation
: __detached__ from root ⇒ will care to destroy the corresponding timeline
;Sequence
:is completely dependent on a root-scoped track, can optionally be bound, into one/multiple timelines/VirtualClip, or unbound
: __created__ ⇒ mandates specification of an track-MO, (necessarily) placed into root scope — {{red{TODO: what to do with non-root tracks?}}}
: __destroy__ ⇒ destroy any binding using this sequence, purge the corresponding track from model, if applicable, including all contents
: __querying__ ⇒ forwards to creating a root-placed track, unless the queried sequence exists already
;root-placed Track
:attachment of a track to root scope is detected magically and causes creation of a Sequence
: __attached__ to root ⇒ invoke sequence creation
: __detached__ from root ⇒ find and detach from corresponding sequence, which is then destroyed. //Note:// contents remain unaffected
: irrespective if the track exists, is empty, or gets purged entirely, only the connection to root scope counts; thus relocating a track by [[Placement]] might cause its scope with all nested contents to become a sequence of its own or become part of another sequence. As sequences aren't required to be bound into a timeline, they may be present in the model as invisible, passive container
!!Magic attachments
While generally the HighLevelModel allows all kinds of arrangements and attachments, certain connections are [[detected automatically|ScopeTrigger]] and may trigger special actions, like the creation of Timeline or Sequence façade objects as described above. The implementation of such [[magic attachments|MagicAttachment]] relies on the PlacementIndex.
When it comes to addressing and distinguishing object instances, there are two different models of treatment, and usually any class can be related to one of these: An object with ''value semantics'' is completely defined through this "value", and not distinguishable beyond that. Usually, value objects can be copied, handled and passed freely, without any ownership. To the contrary, an object with ''reference semantics'' has an unique identity, even if otherwise completely opaque. It is rather like a facility, "living" somewhere, often owned and managed by another object (or behaving special in some other way). Usually, client code deals with such objects through a reference token (which has value semantics). Care has to be taken with //mutable objects,// as any change might influence the object's identity. While this usually is acceptable for value objects, it is prohibited for objects with reference semantics. These are typically created by //factories// — and this fabrication is the only process to define the identity.
!Assets
Each [[Asset]] holds an identification tuple; the hash derived from this constant tuple is used as ~Asset-ID.
* the {{{Asset::Ident}}} tuple contains the following information
*# a __name-ID__, which is a human understandable but sanitised word
*# a tree-like classification of the asset's __category__, comprised of
*#* asset kind {{{{AUDIO, VIDEO, EFFECT, CODEC, STRUCT, META}}}}
*#* a path in a virtual classification tree. Some »folders« have magic meanings
*# an __origin-ID__ to denote the origin, authorship or organisation, acting like a namespace
*# a __version number__.
Of these, the tuple {{{(org,category,name)}}} defines the unique asset identity, and is hashed into the asset-ID. At most one asset with a given ident-tuple (including version) can exist in the whole system. Any higher version is supposed to be fully backwards compatible to all previous versions. Zero is reserved for internal purposes. {{red{1/10: shouldn't we extend the version to (major,minor), to match the plug-in versioning?}}} Thus, the version can be incremented without changing the identity, but the system won't allow co-existence of multiple versions.
* Assets are ''equality'' comparable (even ''ordered''), based on their //identity// — sans version.
* Categories are ''equality'' comparable value objects
!~MObjects
As of 1/10, MObjects are mostly placeholders or dummies, because the actual SessionLogic has still to be defined in detail.
The following properties can be considered as settled:
* reference semantics
* non-copyable, created by MObjectFactory, managed automatically
* each ~MObject is associated n:1 to an asset.
* besides that, it has an opaque instance identity, which is never made explicit.
* this identity springs from the way the object is created and manipulated. It is //persistent.//
* because of the ~MObject-identity's nature, there is //no point in comparing ~MObjects.//
* ~MObjects are always attached to the session by a placement, which creates kind-of an //instance//
* thus, because placements act as a subdivision of ~MObject identification, in practice always placements will be compared.
!Placements
[[Placements|Placement]] are somewhat special, as they mix value and reference semantics. First off, they are configuration values, copyable and smart-pointers, referring to a primary subject (clip, effect, track, label, binding,....). But, //by adding a placement to the session,// we create an unique instance-identity. This is implemented by copying the placement into the internal session store and thereby creating a new hash-ID, which is then registered within the PlacementIndex. Thus, a ''placement into the model'' has a distict identity.
* Placements are ''equality'' comparable, based on this instance identity (hash-ID)
* besides, there is an equivalence relation regarding the "placement specification" contained in the [[locating pins|LocatingPin]] of the Placement.
** they can be compared for ''equivalent definition'': the contained definitions are the same and in the same order
** alternatively, they can be checked for ''effective equivalence'': both placements to be compared resolve to the same position
note: the placement equality relation carries over to ~PlacementRef and ~MObjectRef
!Commands
{{red{WIP}}} For now, commands are denoted by an unique, human-readable ID, which is hard-coded in the source. We might add an LUID and a version numbering scheme later on.
Commands are used as ''prototype object'' — thus we face special challenges regarding the identity, which haven't yet been addressed.
!References and Handles
These are used as token for dealing with other objects and have no identity of their own. PlacementRef tokens embody a copy of the referred placement's hash-ID. MObjectRef handles are built on top of the former, additionally holding a smart-ptr to the primary subject.
* these reference handles simply reflect the equality relation on placements, by virtue of comparing the hash-ID.
* besides, we could build upon the placement locating chain equivalence relations to define a semantic equivalence on ~MObjectRefs
Any point where output possibly might be produced. Model port entities are located within the [[Fixture]] — model port as a concept spans the high-level and low-level view. A model port can be associated both to a pipe in the HighLevelModel but at the same time denotes a set of corresponding [[exit nodes|ExitNode]] within the [[segments|Segmentation]] of the render nodes network.
A model port is rather derived than configured; it emerges when a pipe [[claims|WiringClaim]] an output destination, while some other entity at the same time actually //uses this designation as a target,// either directly or indirectly. This match of provision and usage is detected during the build process and produces an entry in the fixture's model port table. These model ports in the fixture are keyed by ~Pipe-ID, thus each model port has an associated StreamType.
Model ports are the effective, resulting outputs of each timeline; additional ports result from [[connecting a viewer|ViewConnection]]. Any render or display process happens at a model port.
!formal specification
Model port is a //conceptual entity,// denoting the possibility to pull generated data of a distinct (stream)type from a specific bus within the model -- any possible output produced or provided by Lumiera is bound to appear at a model port. The namespace of model ports is global, each being associated with a ~Pipe-ID.
Model ports are represented by small non-copyable descriptor objects with distinct identity, which are owned and managed by the [[Fixture]]. Clients are bound to resolve a model port on each usage, as configuration changes within the model might cause ports to appear and decease. To stress this usage pattern, actually {{{ModelPort}}} instances are small copyable value objects (smart handles), which can be used to access data within an opaque registry. Each model port belongs to a specific Timeline and is aware of this association, as is the timeline, allowing to get the collection of all ports of a given timeline. Besides, within the Fixture each model port refers to a specific [[segmentation of the time axis|Segmentation]] (relevant for this special timeline actually). Thus, with the help of this segmentation, a model port can yield an ExitNode to pull frames for a given time.
Model ports are conceptual entities, denoting the points where output might possibly be produced → see [[definition|ModelPort]].
But there is an actual representation, a collection of small descriptor objects managed by the Fixture and organised within the model port table datastructure. Because model ports are discovered during the build process, we need the ability to (re)build this table dynamically, finally swapping in the modified configuration with a transactional switch. Only the builder is allowed to perform such mutations, while for client code model ports are immutable.
!supported operations
* get the model port by ~Pipe-ID
* a collection of model ports per timeline
* sub-grouping by media kind, possibly even by StreamType
* possibility to enumerate model ports in distinct //order,// defined within the timeline
* with the additional possibility to filter by media kind or suitable stream type.
* a way to express the fact of //not having a model port.//
!!!mutating and rebuilding
Model ports are added once and never changed. The corresponding timeline and pipe is known at setup time, but the overall number of model ports is determined only as a result of completing the build process. At that point, the newly built configuration is swapped in transactionally to become the current configuration.
!Implementation considerations
The transactional switch creates a clear partitioning in the lifespan of the model port table. //Before// that point, entries are just added, but not accessed in any way. //After// that point, no further mutation occurs, but lookup is frequent and happens in a variety of different configurations and transient orderings.
This observation leads to the idea of using //model port references// as frontend to provide all kinds of access, searching and reordering. These encapsulate the actual access by silently assuming reference to "the" global current model port configuration. This way the actual model port descriptors could be bulk allocated in a similar manner as the processing nodes and wiring descriptors. Access to stale model ports could be detected by the port references, allowing also for a {{{bool}}} checkable "has no port" information.
A model port registry, maintained by the builder, is responsible for storing the discovered model ports within a model port table, which is then swapped in after completing the build process. The {{{builder::ModelPortRegistry}}} acts as management interface, while client code accesses just the {{{ModelPort}}} frontend. A link to the actual registry instance is hooked into that frontend when bringing up the builder subsystem.
A special kind of MObject, serving as a marker or entry point at the root of the HighLevelModel. As any ~MObject, it is attached to the model by a [[Placement]]. And in this special case, this placement froms the ''root scope'' of the model, thus containing any other PlacementScope (e.g. tracks, clips with effects,...)
This special ''session root object'' provides a link between the model part and the »bookkeeping« part of the session, i.e. the [[assets|Asset]]. It is created and maintained by the session (implementation level) — allowing to store and load the asset definitions as contents of the model root element.
__Note__: nothing within the PlacementIndex requires the root object to be of a specific type; the index just assumes a {{{Placement<MObject>}}} (or subclass) to exist as root element. And indeed, for several unit tests we create an Index mock with a tree of dummy ~MObjects and temporarily shaddow the "real" PlacementIndex by this mock (→ see SessionServices for the API allowing to access this //mock index//- functionality)
Based on practical experiences, Ichthyo tends to consider Multichannel Media as the base case, while counting media files providing just one single media stream as exotic corner cases. This may seem counter intuitive at first sight; you should think of it as an attempt to avoid right from start some of the common shortcomings found in many video editors, especially
* having to deal with keeping a "link" between audio and video clips
* silly limitations on the supported audio setups (e.g. "sound is mono, stereo or Dolby-5.1")
* unnecessary complexity when dealing with more realistic setups, esp. when working on dialogue scenes
* inability to edit stereoscopic (3D) video in a natural fashion
!Compound Media
[>img[Outline of the Build Process|uml/fig131333.png]]
Basically, each [[media asset|MediaAsset]] is considered to be a compound of several elementary media (tracks), possibly of various different media kinds. Adding support for placeholders (''proxy clips'') at some point in future will add still more complexity (because then there will be even dependencies between some of these elementary media). To handle, edit and render compound media, we need to impose some structural limitations. But anyhow, we try to configure as much as possible already at the "asset level" and make the rest of the proc layer behave just according to the configuration given with each asset.
So, when creating a clip out of such a compound media asset, the clip has to be a compound of elementary clips mirroring the given media asset's structure. Besides, it should be possible to //detach// and //attach// elementary clips from a compound clip. On the other hand, the [[Fixture]] created from the current state of the session is explicit to a great extent. So, in the Fixture we deal only with elementary clips placed to absolute positions, and thus the builder will see only simple non-compound clips and translate them into the corresponding source reading nodes.
!Handling
* from a Media asset, we can get a [[Processing Pattern (ProcPatt)|ProcPatt]] describing how to build a render pipeline for this media
* we can create a Clip (MObject) from each Media, which will be linked back to the media asset internally.
* moreover, creating a Clip will create and register a Clip asset as well, and this Clip asset will be tied to the original Clip and will show up in some special Category
* media can be compound and the created Clips will mirror this compound structure
* we distinguish elementay (non-compound) Clips from compound clips by concrete subtype. {{red{really?? doesn't seem so much like a good idea to me anymore 1/10}}} The builder can only handle elementary clips, because he needs to build a separate pipeline for every output channel. So the work of splitting common effect stacks for clips with several channels needs to be done when calculating the [[Fixture]] for the current session. The Builder expects to be able to build the render nodes corresponding to each entity found in the Fixture one by one.
* the Builder gets at the ProcPatt (descriptor) of the underlying media for each clip and uses this description as a template to build the render pipeline. That is, the ProcPatt specifies the codec asset and maybe some additional effect assets (deinterlace, scale) necessary for feeding media data corresponding to this clip/media into the render nodes network.
!{{red{Reviewed 3/2010}}}
While the general approach and reasoning remains valid, a lot of the details looks dated meanwhile.
* it is //not// true that the builder can be limited to the handling single processing chains. Some effects ineed need to be fed with mutliple channels — most notably panners and compressors.
* consequently there is the need for an internal representation of the media StreamType.
* thus we can assume the presence of some kind of //type system//
* which transforms the individual "stream" into a entirely conceptual entity within the HighLevelModel
* moreover it is clear that the channel configuration needs to be flexible, and not automatically bound to the existence of a single media with that configuration.
* and last but not least, we handle nested sequences as virtual clips with virtual media.
⇒ conclusions
* while the asset related parts remain as specified, we get a distinct ChannelConfig asset instead of the clip asset (which seems to be redundant)
* either the ~ClipMO referres this ChannelConfig asset — or in case of the VirtualClip a BindingMO takes this role. Clip Asset and MO could be joined into a single entity
* as the BindingMO is also used to implement the top-level timelines, the treatment of global and local pipes is united
* every pipe (bus) should be able to carry multiple channels, //but with the limitation to only a single media StreamType//
* this "multichannel-of-same-kind" capability carries over to the ModelPort entries and even the OutputSlot elements
* only when allocating / "opening" an OutputSlot, we get multiple handles for plain single channels.
* this can be considered the breaking point, where we enter the realm of the render engine. Here, indeed, only single channels are processed
Various aspects of the individual [[render node|ProcNode]] are subject to configuration and may influence the output quality or the behaviour of the render process.
* the selection //what// actual implementation (plugin) to used for a formally defined »[[Effect|EffectHandling]]«
* the intermediary/common StreamType to use within a [[Pipe]]
* the render technology (CPU, hardware accelerated {{red{→ Future}}})
* the ScheduleStrategy (possibly subdividing the calculation of a single frame)
* if this node becomes a possible CachePoint or DataMigrationPoint in RenderFarm mode
* details of picking a suitable [[operation mode|RenderImplDetails]] of the node (e.g. utilitsing "in-place" calculation)
~NodeCreatorTool is a [[visiting tool|VisitorUse]] used as second step in the [[Builder]]. Starting out from a [[Fixture]], the builder first [[divides the Timeline into segments|SegmentationTool]] and then processes each segment with the ~NodeCreatorTool to build a render nodes network (Render Engine) for this part of the timeline. While visiting individual Objects and Placements, the ~NodeCreatorTool creates and wires the necessary [[nodes|ProcNode]]
!Problem of Frame identification
!Problem of Node numbering
In the most general case the render network may be just a DAG (not just a tree). Especially, multiple exit points may lead down to the same node, and following each of this possible paths the node may be at a different depth on each. This rules out a simple counter starting from the exit level, leaving us with the possibility of either employing a rather convoluted addressing scheme or using arbitrary ID numbers.{{red{...which is what we do for now}}}
The [[nodes|ProcNode]] are wired to form a "Directed Acyclic Graph"; each node knows its predecessor(s), but not its successor(s). The RenderProcess is organized according to the ''pull principle'', thus we find an operation {{{pull()}}} at the core of this process. Meaning that there isn't an central entity invoking nodes consecutively. Rather, the nodes themselves contain the detailed knowledg regarding prerequisites, so the calculation plan is worked out recursively. Yet there are some prerequisite resources to be made available for any calculation to happen. Thus the actual calculation is broken down into atomic chunks of work, resulting in a 2-phase invocation whenever "pulling" a node. For this to work, we need the nodes to adhere to a specific protocol:
;planning phase
:when a node invocation is foreseeable to be required for getting a specific frame for a specific nominal and actual time, the engine has to find out the actual operations to happen
:# the planning is initiated by issuing an "get me output" request, finally resulting in a JobTicket
:# recursively, the node propagates "get me output" requests for its prerequisites
:# after retrieving the planning information for these prerequisites, the node encodes specifics of the actual invocation situation into a closure called StateAdapter <br/>{{red{TODO: why not just labeling this »~StateClosure«?}}}
:# finally, all this information is packaged into a JobTicket representing the planning results.
;pull phase
:now the actual node invocation is embedded within a job, activated through the scheduler to deliver //just in time.//
:# Node is pulled, with a StateProxy object as parameter (encapsulating BufferProvider for access to the required frames or buffers)
:# Node may now retrieve current parameter values, using the state accessible via the StateProxy
:# to prepare for the actual {{{process()}}} call, the node now has to retrieve the input prerequisites
:#* when the planning phase determined availability from the cache, then just these cached buffer(s) are now retrieved, dereferencing a BuffHandle
:#* alternatively the planning might have arranged for some other kind of input to be provided through a prerequisite Job. Again, the corresponding BuffHandle can now be dereferenced
:#* Nodes may be planned to have a nested structure, thus directly invoking {{{pull()}}} call(s) to prerequisite nodes without further scheduling
:# when input is ready prior to the {{{process()}}} call, output buffers will be allocated by locking the output [[buffer handles|BuffHandle]] prepared during the planning phase
:# since all buffers and prerequistes are available, the Node may now prepare a frame pointer array and finally invoke the external {{{process()}}} to kick off the actual calculations
:# finally, when the {{{pull()}}} call returns, "parent" state originating the pull holds onto the buffers containing the calculated output result.
{{red{WIP as of 9/11 -- many details here are still to be worked out and might change as we go}}}
some points to note:
* the WiringDescriptor is {{{const}}} and precalculated while building (remember another thread may call in parallel)
* when a node is "inplace-capable", input and output buffer may actually point to the same location
* but there is no guarantee for this to happen, because the cache may be involved (and we can't overwrite the contents of a cache frame)
* generally, a node may have N inputs and M output frames, which are expected to be processed in a single call
* some of the technical details of buffer management are encapsulated within the BufferTable of each invocation
→ the [["mechanics" of the render process|RenderMechanics]]
→ more fine grained [[implementation details|RenderImplDetails]]
We have to consider carefully how to handle the Creation of new class instances. Because, when done naively, it can defeat all efforts of separating subsystems, or — the other extreme — lead to a //switch-on-typeID// programming style. We strive at a solution somewhere in the middle by utilizing __Abstract Factories__ on Interface or key abstraction classes, but providing specialized overloads for the different use cases. So in each use case we have to decide if we want to create a instance of some general concept (Interface), or if we have a direct collaboration and thus need the Factory to provide a more specific sub-Interface or even a concrete type.
!Object creation use cases
!![[Assets|Asset]]
|!Action|>|!creates |
|loading a media file|asset::Media, asset::Codec| |
|viewing media|asset::Sequence, session::Clip and Placement (on hold)| for the whole Media, if not already existent|
|mark selection as clip|session::Clip, Placement with unspec. LocatingPin| doesn't add to session|
|loading Plugin|asset::Effect| usually at program startup|
|create Session|asset::Sequence, asset::Timeline, asset::Pipe| |
→ [[creating and registering Assets|AssetCreation]]
→ [[loading media|LoadingMedia]]
!![[MObjects|MObject]]
|add media to sequence|session::Clip, Placement with unspecified LocatingPin| creating whole-media clip on-the-fly |
|add Clip to sequence|copy of Placement| creates intependent Placement of existing ~Clip-MO|
|attach Effect|session::Effect, Placement with RelativeLocation| |
|start using Automation|session::Auto, asset::Dataset, RelativeLocation Placement| |
!Invariants
when creating Objects, certain invariants have to be maintained. Because creating an Object can be considered an atomic operation and must not leave any related objects in an inconsistent state. Each of our interfaces implies some invariants:
* every Placement has a Subject it places
* MObjects are always created to be placed in some way or the other
* [[Assets|Asset]] manage a dependency graph. Creating a derived Object (e.g. a Clip from a Media) implies a new dependency. (→ [[memory management|ManagementAssetRelation]] relies on this)
Cinelerra2 introduced OpenGL support for rendering previews. I must admit, I am very unhappy with this, because
* it just supports some hardware
* it makes building difficult
* it can't handle all color models Cinelerra is capable of
* it introduces a separate codepath including some complicated copying of video data into the textures (and back?)
* it can't be used for rendering
So my judgement would be: in contrary to a realtime/gaming application, for quality video editing it is not worth the effort implementing OpenGL support in all details and with all its complexity. I would accept ~OpenGL as an option, if it could be pushed down into a Library, so it can be handled and maintained transparently and doesnt bind our limited developer manpower.
But because I know the opinions on this topc are varying (users tend to be delighted if they hear "~OpenGL", because it seems to be likted to the notion of "speed" and "power" todays) — I try to integrate ~OpenGL as a possibility into this design of the Render Engine. But I insist on putting up the requirement that it //must not jeopardize the code structure.//
My proposed aproach is to treat OpenGL as a separate video raw data type, requiring separete and specialized [[Processing Nodes|ProcNode]] for all calculations. Thus the Builder could connect OpenGL nodes if it is possible to cover the render path in whole or partially or maybe even just for preview.
A low-level abstraction within the [[Builder]] — it serves to encapsulate the details of making multi-channel connections between the render nodes: In some cases, a node can handle N channels internally, while in other cases we need to replicate the node N times and wire each channel individually. As it stands, the OperationPoint marks the ''borderline between high-level and low-level model'': it is invoked in terms of ~MObjects and other entities of the high-level view, but internally it manages to create ProcNode and similar entities of the low-level model.
The operation point is provided by the current BuilderMould and used by the [[processing pattern|ProcPatt]] executing within this mould and conducting the current build step. The operation point's interface allows //to abstract// these details, as well as to //gain additional control// if necessary (e.g. addressing only one of the channels). The most prominent build instruction used within the processing patterns (which is the instruction {{{"attach"}}}) relies on the aforementioned //approach of abstracted handling,// letting the operation point determine automatically how to make the connection.
This is possible because the operation point has been provided (by the mould) with information about the media stream type to be wired, which, together with information accessible at the [[render node interface|ProcNode]] and from the [[referred processing assets|ProcAsset]], with the help of the [[connection manager|ConManager]] allows to figure out what's possible and how to do the desired connections. Additionally, in the course of deciding about possible connections, the PathManager is consulted to guide strategic decisions regarding the [[render node configuration|NodeConfiguration]], possible type conversions and the rendering technology to employ.
An ever recurring problem in the design of Luimiera's ~Proc-Layer is how to refer to output destinations, and how to organise them.
Wiring the flexible interconnections between the [[pipes|Pipe]] should take into account both the StreamType and the specific usage context ([[scope|PlacementScope]]) -- and the challenge is to avoid hard-linking of connections and tangling with the specifics of the target to be addressed and connected. This page, started __6/2010__ by collecting observations to work out the relations, arrives at defining a //key abstraction// of output management.
!Observations
* effectively each [[Timeline]] is known to expose a set of global Pipes
* when connecting a Sequence to a Timeline or a VirtualClip, we also establish a mapping
* this mapping translates possible media stream channels produced by the sequence into (real) output slots located in the enclosing entity
* uncertainty on who has to provide the global pipes, implementation wise —
** as Timeline is just a façade, BindingMO has to expose something which can be referred for attaching effects (to global pipes)
** when used as VirtualClip, there is somehow a channel configuration, either as asset, or exposed by the BindingMO
* Placements always resolve at least two dimensions: time and output. The latter means that a [[Placement]] can figure out to where to connect
* the resolution ability of Placements could help to overcome the problems in conjunction with a VirtualClip: missing output destination information could be inherited down....
* expanding on the basic concept of a Placement in N-dimensional configuration space, this //figuring out// would denote the ability to resolve the final output destination
* this resolution to a final destination is explicitly context dependent. We engage into quite some complexities to make this happen (→ BindingScopeProblem)
* [[processing patterns|ProcPatt]] are used for creating nodes on the source network of a clip, and similarly for fader, overlay and mixing into a summation pipe
* in case the track tree of a sequence doesn't contain specific routing advice, connections will be done directly to the global pipes in order and by matching StreamType (i.e. typically video to video master, audio to stereo audio master). When a monitor (viewer window) is attached to a timeline, similar output connections are made from the timeline's global pipes, i.e. the video display will take the contents of the first video (master) bus, and the first stereo audio pipe will be pulled and sent to system audio out.
* a mismatch between the system output possibilities and the stream type of a bus to be monitored should result in the same adaptation mechanism to kick in, as is used internally, when connecting an ~MObject to the next bus. Possibly we'll use separate rules in this case (allow 3D to flat, stereo to mono, render 5.1 into Ambisonics...)
!Conclusions
* there are //direct, indirect// and //relative// referrals to an output designation
* //indirect// means to derive the destination transitively (looking at the destination's output designation and so on)
* //relative// in this context means that we refer to "the N^^th^^ of this kind" (e.g. the second video out)
* we need to attach some metadata with an output; especially we need an associated StreamType
* the referral to an output designation can be observed on and is structured into several //levels://
** within the body of the model, mostly we address output destinations relatively
** at some point, we'll address a //subgroup// within the global pipes, which acts like a summation sink
** there might be //master pipe(s),// collecting the output of various subgroups
** finally, there is the hardware output or the distinct channel within the rendered result — never to be referenced explicitly
!!!relation to Pipes
in almost every case mentioned above, the output designation is identical with the starting point of a [[Pipe]]. This might be a global pipe or a ClipSourcePort. Thus it sounds reasonable to use pipe-~IDs directly as output designation. Pipes, as an accountable entity (=asset) just //leap into existence by being referred.// On the other hand, the //actual// pipe is a semantic concept, a specific structural arrangement of objects. Basically it means that some object acts as attachment point and thereby //claims// to be the entrance side of a pipe, while other processor objects chain up in sequence.
!!!system outputs
System level output connections seem to be an exception to the above rule. There is no pipe at an external port, and these externally visible connection points can appear and disappear, driven by external conditions. Yet the question is if system outputs shall be directly addressable at all as output designation? Generally speaking, Lumiera is not intended to be a system wide connection manager or a real time performance software. Thus it's advisable to refrain from direct referrals to system level connections from within the model. Rather, there should be a separate OutputManagement to handle external outputs and displays, both in windows or full screen. So these external outputs become rather a matter of application configuration — and for all the other purposes we're free to ''use pipe-~IDs as output designation''.
!!!consequences of mentioning an output designation
The immediate consequence is that a [[Pipe]] with the given ID exists as an accountable entity. Only if — additionally — a suitable object within the model //claims to root this pipe,// a connection to this designation is wired, using an appropriate [[processing pattern|ProcPatt]]. A further consequence then is for the mentioned output designation to become a possible candidate for a ModelPort, an exit node of the built render nodes network. By default, only those designations without further outgoing connections actually become active model ports (but an existing and wired pipe exit node can be promoted to a model port explicitly).
→ OutputManagement
!!Challenge: mapping of output designations
An entire sequence can be embedded within another sequence as a VirtualClip. While for a simple clip there is a Clip-~MObject placed into the model, holding an reference to a media asset, in case of a virtual clip an BindingMO takes on the role of the clip object. Note that, within another context, BindingMO also acts as [[Timeline]] implementation — indeed even the same sequence might be bound simultaneously as a timeline and into a virtual clip. This flexibility creates a special twist, as the contents of this sequence have no way of finding out if they're used as timeline or embedded virtual clip. So parts of this sequence might mention a OutputDesignation which, when using the sequence as virtual clip, needs to be translated into a virtual media channel, which can be treated in the same manner like any channel (video, audio,...) found within real media. Especially, a new output designation has to be derived in a sensible way, which largely depends on how the original output designation has been specified.
→ see OutputMapping regarding details and implementation of this mapping mechanism
!Output designation definition
OutputDesignation is a handle, denoting a target [[Pipe]] to connect.
It exposes a function to resolve to a Pipe, and to retrieve the StreamType of that resolved output. It can be ''defined'' either explicitly by ~Pipe-ID, or by an indirect or relative specification. The later cases are resolved on demand only (which may be later and might even change the meaning, depending on the context). It's done this way intentionally to gain flexibility and avoid hard wiring (=explicit ~Pipe-ID)
!!!Implementation notes
Thus the output designation needs to be a copyable value object, but opaque beyond that. Mandated by the various ways to specify an output designation, a hidden state arises regarding the partial resolution. The implementation solves that dilemma by relying on the [[State|http://en.wikipedia.org/wiki/State_pattern]] pattern in combination with an opaque in-place buffer.
!Use of output designations
While actually data frames are //pulled,// on a conceptual level data is assumed to "flow" through the pipes from source to output. This conceptual ("high level") model of data flow is comprised of the pipes (which are rather rigid), and flexible interconnections. The purpose of an output designation is to specify where the data should go, especially through these flexible interconnections. Thus, when reaching the exit point of a pipe, an output designation will be //queried.// Finding a suitable output designation involves two parts:
* first of all, we need to know //what to route// -- kind of the traits of the data. This is given by the //current pipe.//
* then we'll need to determine an output designation //suitable for this data.// This is given by a "Plug" (WiringRequest) in the placement, and may be derived.
* finally, this output designation will be //resolved// -- at least partially, resulting in a target pipe to be used for the wiring
As both of these specifications are given by [[Pipe]]-~IDs, the actual designation information may be reduced. Much can be infered from the circumstances, because any pipe includes a StreamType, and an output designation for an incompatible stream type is irrelevant. (e.g. and audio output when the pipe currently in question deals with video)
//writing down some thoughts//
* ruled out the system outputs as OutputDesignation.
* thus, any designation is a [[Pipe]]-ID.
* consequently, it is not obviously clear if such an designation is the final exit point
* please note the [[Engine interface proposal|http://lumiera.org/documentation/devel/rfc_pending/EngineInterfaceOverview.html]]
* this introduces the notion of a ModelPort: //a point in the (high level) model where output can be produced//
* thus obviously we need an OutputManager element to track the association of OutputDesignation to OutputSlot
Do we get a single [[Fixture]] — guess yes
From the implementation side, the only interesting exit nodes are the ones to be //actually pulled,// i.e. those immeditately corresponding to the final output.
* __rendering__ happens immediately at the output of a GlobalPipe (i.e. at a [[Timeline]], which is top-level)
* __playback__ always happens at a viewer element
!Attaching and mapping of exit nodes
Initially, [[Output designations|OutputDesignation]] are typically just local or relative references to another OutputDesignation; yet after some resolution steps, we'll arrive at an OutputDesignation //defined absolute.// Basically, these are equivalent to a [[Pipe]]-ID choosen as target for the connection and -- they become //real// by some object //claiming to root this pipe.// The applicability of this pattern is figured out dynamically while building the render network, resulting in a collection of [[model ports|ModelPort]] as part of the current [[Fixture]]. A RenderProcess can be started to pull from these -- and only from these -- active exit points of the model. Besides, when the timeline enclosing these model ports is [[connected to a viewer|ViewerPlayConnection]], an [[output network|OutputNetwork]] //is built to allow hooking exit points to the viewer component.// Both cases encompass a mapping of exit nodes to actual output channels. Usually, this mapping just relies on relative addressing of the output sinks, starting to allocate connections with the //first of each kind// (e.g. "connect to the first usable audio output destination").
We should note that in both cases this [[mapping operation|OutputMapping]] is controlled and driven and constrained by the output side of the connection: A viewer has fixed output capabilities, and rendering targets a specific container format -- again with fixed and pre-settled channel configuration ({{red{TODO 9/11}}} when configurting a render process, it might be necessary to pre-compute the //possible kinds of output streams,// so to provide a sensible pre-selection of possible output container formats for the user to select from). Thus, as a starting point, we'll create a default configured mapping, assigning channels in order. This mapping then should be exposed to modification and tweaking by the user. For rendering, this is part of the render options dialog, while in case of a viwer connection, a switch board is created to allow modifying the default mapping.
!Connection to external outputs
External output destinations are never addressed directly from within the model. This is an design decision. Rather, model parts connect to an OutputDesignation, and these in turn may be [[connected to a viewer element|ViewerPlayConnection]]. At this point, related to the viewer element, there is a mapping to external destination(s): for images, a viewer typically has an implicit, natural destination (read: actually there is a corresponding viewer window or widget), while for sound we use an mapping rule, which could be overridden locally in the viewer.
Any external output sink is managed as a [[slot|DisplayerSlot]] in the ~OutputManager. Such a slot can be opened and allocated for a playback process, which allows the latter to push calculated data frames to the output. Depending on the kind of output, there might be various, often tight requirements on the timed delivery of output data, but any details are abstracted away — any slot implementation provides a way to handle time-outs gracefully, e.g. by just showing the last video frame delivered, or by looping and fading sound
→ the OutputManager interface describes handling this mapping association
→ see also the PlayService
!the global output manager
Within the model routing is done mostly just by referring to an OutputDesignation -- but at some point finally we need to map these abstract designations to real output capabilities. This happens at the //output managing elements.// This interface, OutputManager, exposes these mappings of logical to real outputs and allows to manage and control them. Several elements within the application, most notably the [[viewers|ViewerAsset]], provide an implementation of this interface -- yet there is one primary implementation, the ''global output manager'', known as OutputDirector. It can be accessed through the {{{Output}}} façade interface and is the final authority when it comes to allocating and mapping of real output possibilities. The OutputDirector tracks all the OutputSlot elements currently installed and available for output.
The relation between the central OutputDirector and the peripheral OutputManager implementations is hierarchical. Because output slots are usually registered rather at some peripheral output manager implementation, a direct mapping from OutputDesignation (i.e. global pipe) to these slots is created foremost at that peripheral level. Resolving a global pipe into an output slot is the core concern of any OutputManager implementation. Thus, when there is a locally preconfigured mapping, like e.g. for a viewer's video master pipe to the output slot installed by the corresponding GUI viewer element, then this mapping will be picked up foremost to resolve the video master output.
For a viewer widget in the GUI this yields exactly the expeted behaviour, but in other cases, e.g. for sound output, we need more general, more globally scoped output slots. In these cases, when a local mapping is absent, the query for output resolution is passed on up to the OutputDirector, drawing on the collection of globally available output slots for that specific kind of media.
{{red{open question 11/11: is it possible to retrieve a slot from another peripheral node?}}}
!!!output modes
Most output connections and drivers embody some kind of //operation mode:// Display is characterised by resolution and colour depth, sound by number of channels and sampling rate, amongst others. There might be a mismatch with the output expectations represented by [[output designations|OutputDesignation]] within the model. Nontheless we limit those actual operation modes strictly to the OutputManager realm. They should not leak out into the model within the session.
In practice, this decision might turn out to be rather rigid, but some additional mechanisms allow for more flexibility
* when [[connecting|ViewerPlayConnection]] timeline to viewer and output, stream type conversions may be added automatically or manually
* since resolution of an output designation into an OutputSlot is initiated by querying an output manager, this query might include additional constraints, which //some// (not all) concrete output implementations might evaluate to provide an more suitably configured output slot variant.
The term »''Output Manager''« might denote two things: first, there is an {{{proc::play::OutputManager}}} interface, which can be exposed by several components within the application, most notably the [[viewer elements|ViewerAsset]]. And then, there is "the" global output manager, the OutputDirector, which finally tracks all registered OutputSlot elements and thus is the gatekeeper for any output leaving the application.
→ see [[output management overview|OutputManagement]]
→ see OutputSlot
→ see ViewerPlayConnection
!Role of an output manager
The output manager interface describes an entity handling two distinct concerns, tied together within a local //scope.//
* a ''mapping'' to resolve a ModelPort (given as ~Pipe-ID) into an OutputSlot
* the ''registration'' and management of possible output slots, thereby creating a preferred local mapping for future connections
Note that an OutputSlot acts as a unit for registration and also for allocating / "opening" an output, while generally there might still be multiple physical outputs grouped into a single slot. This is relevant especially for sound output. A single slot is just the ability to allocate output ports up to a given limit (e.g. 2 for a stereo device, or 6 for a 5.1 device). These multiple channels are allways connected following a natural channel ordering. Thus the mapping is a simple 1:1 association from pipe to slot, assuming that the media types are compatible (and this has been checked already on a higher level).
The //registration// of an output slot installs a functor or association rule, which later on allows to claim and connect up to a preconfigured number of channels. This allocation or usage of a slot is exclusive (i.e. only a single client at a time can allocate a slot, even if not using all the possible channels). Each output manager instance may or may not be configured with a //fall-back rule:// when no association or mapping can be established locally, the connection request might be passed down to the global OutputDirector. Again, we can expect this to be the standard behaviour for sound, while video likely will rather be handled locally, e.g. within a GUI widget (but we're not bound to configure it exactly this way)
An output mapping serves to //resolve// [[output designations|OutputDesignation]].
!Mapping situations
;external outputs
:external outputs aren't addressed directly. Rather we set up a default (channel) mapping, which then can be overridden by local rules.
:Thus, in this case we query with an internal OutputDesignation as parameter and expect an OutputSlot
;viewer connections
:any Timeline produces a number of [[model ports|ModelPort]]. On the other hand, any viewer exposes a small number of output designations, representing the image and sound output(s).
:Thus, in this case we resolve similar to a bus connection, possibly overridden by already pre-existing or predefined connections.
;switch board
:a viewer might receive multiple outputs and overlays, necessitating a user operated control to select what's actually to be displayed
:Thus, in this case we need a backwards resolution at the lower end of the output network, to connect to the model port as selected through the viewer's SwitchBoard
;global pipes or virtual media
:when binding a Sequence as Timeline or VirtualClip, a mapping from output designations used within the Sequence to virtual channels or global pipes is required
:Thus, in this case we need to associate output designations with ~Pipe-IDs encountered in the context according to some rules — again maybe overridden by pre-existing connections
!Conclusions
All these mapping steps are listed here, because they exhibit a common pattern.
* the immediately triggering input key is a [[Pipe]]-ID (and thus provides a stream type); additional connection hints may be given.
* the mapped result draws from a limited selection of elements, which again can be keyed by ~Pipe-IDs
* the mapping is initialised based on default mapping rules acting as fallback
* the mapping may be extended by explicitly setting some associations
* the mapping itself is a stateful value object
* there is an //unconnected// state.
!Implementation notes
Thus the mapping is a copyable value object, using an associative array. It may be attached to a model object and persisted alongside. The mapping is assumed to run a defaults query when necessary. To allow for that, it should be configured with a query template (string). Frequently, special //default pipe// markers will be used at places where no distinct pipe-ID is specified explicitly. Besides that, invocations might supply additional predicates (e.g. {{{ord(2)}}} to point at "the second stream of this kind") thereby hinting the defaults resolution. Moreover, the mapping needs a way to retrieve the set of possible results, allowing to filter the results of the rules based default. Mappings might be defined explicitly. Instead of storing a //bottom value,// an {{{isDefined()}}} predicate might be preferable.
First and foremost, mapping can be seen as a //functional abstraction.// As it's used at implementation level, encapsulation of detail types in't the primary concern, so it's a candidate for generic programming: For each of those use cases outlined above, a distinct mapping type is created by instantiating the {{{OutputMapping<DEF>}}} template with a specifically tailored definition context ({{{DEF}}}), which takes on the role of a strategy. Individual instances of this concrete mapping type may be default created and copied freely. This instantiation process includes picking up the concrete result type and building a functor object for resolving on the fly. Thus, in the way typical for generic programming, the more involved special details are moved out of sight, while being still in scope for the purpose of inlining. But there //is// a concern better to be encapsulated and concealed at the usage site, namely accessing the rules system. Thus mapping leads itself to the frequently used implementation pattern where there is a generic frontend as header, calling into opaque functions embedded within a separate compilation unit.
Within the Lumiera player and output subsystem, actually sending data to an external output requires to allocate an ''output slot''
This is the central metaphor for the organisation of actual (system level) outputs; using this concept allows to separate and abstract the data calculation and the organisation of playback and rendering from the specifics of the actual output sink. Actual output possibilities (video in GUI window, video fullscreen, sound, Jack, rendering to file) can be added and removed dynamically from various components (backend, GUI), all using the same resolution and mapping mechanisms (→ OutputManagement)
!Properties of an output slot
Each OutputSlot is an unique and distinguishable entity. It corresponds explicitly to an external output, or a group of such outputs (e.g. left and right soundcard output channels), or an output file or similar capability accepting media content. First off, an output slot needs to be provided, configured and registered, using an implementation for the kind of media data to be output (sound, video) and the special circumstances of the output capability (render a file, display video in a GUI widget, send video to a full screen display, establish a Jack port, just use some kind of "sound out"). An output slot is always limited to a single kind of media, and to a single connection unit, but this connection may still be comprised of multiple channels (stereoscopic video, multichannel sound).
In order to be usable as //output sink,// an output slot needs to be //allocated,// i.e. tied to and locked for a specific client. At any time, there may be only a single client using a given output slot this way. To stress this point: output slots don't provide any kind of inherent mixing capability; any adaptation, mixing, overlaying and sharing needs to be done within the nodes network producing the output data fed to the slot. (in special cases, some external output capabilities -- e.g. the Jack audio connection system -- may still provide additional mixing capabilities, but that's beyond the scope of the Lumiera application)
[>img[Outputslot implementation structures|uml/fig151685.png]]
Once allocated, the output slot returns a set of concrete ''sink handles'' (one for each physical channel expecting data). The calculating process feeds its results into those handles. Size and other characteristics of the data frames are assumed to be suitable, which typically won't be verified at that level anymore (but the sink handle provides a hook for assertions). Besides that, the allocation of an output slot reveals detailed ''timing expectations''. The client is required to comply to these timings when ''emitting'' data -- he's even required to provide a //current time specification,// alongside with the data. Based on this information, the output slot has the ability to handle timing failures gracefully; the concrete output slot implementation is expected to provide some kind of de-click or de-flicker facility, which kicks in automatically when a timing failure is detected.
!!!usage and implementation
Clients retrieve just a reference to an output slot by asking a suitable OutputManager for an output possibility supporting a specific format. Usually, they just "claim" this slot by invoking {{{allocate()}}}, which behind the scenes causes building of the actual output connections and mechanisms. For each such connection -- corresponding to a single channel within the media format handled by this ~OutputSlot -- the client gets a smart-handle {{{DataSink}}}. The concrete ~OutputSlot implementation performs operations quite specific to the kind of output and external interface in question. All tese specific handling is embodied within the concrete connection implementation used by the concrete ~OutputSlot
!!!timing expectations
Besides the sink handles, allocation of an output slot defines some timing constraints, which are binding for the client. These [[Timings]] are detailed and explicit, including a grid of deadlines for each frame to deliver, plus a fixed //latency.// Within this context, »latency« means the requirement to be ahead of the nominal time by a certain amount, to compensate for the processing time necessary to propagate the media to the physical output pin. The output slot implementation itself is bound by external constraints to deliver data at a fixed framerate and aligned to an externally defined timing grid, plus the data needs to be handed over ahead of these time points by an time amount given by the latency. Depending on the data exchange model, there is an additional time window limiting the buffer management.
The assumption is for the client to have elaborate timing capabilities at his disposal. More specifically, the client is assumed to be a job running within the engine scheduler and thus can be configured to run //after// another job has finished, and to run within certain time limits. Thus the client is able to provide a //current nominal time// -- which is suitably close to the actual wall clock time. The output slot implementation can be written such as to work out from this time specification if the call is timely or overdue -- and react accordingly.
!!!output modes
some concrete output connections and drivers embody a specific operation mode (e.g. sample rate or number of channels). The decision and setup of these operational configuration is initiated together with the [[resolution|OutputMapping]] of an OutputDesignation within the OutputManager, finally leading to an output slot (reference), which can be assumed to be suitably configured, before the client allocates this slot for active use. Moreover, an individual output sink (corresponding to a single channel) may just remain unused -- until there is an {{{emit()}}} call and successful data handover, this channel will just feature silence or remain black. (More flexible system, e.g. Jack, allow to generate an arbitrary number of output pins -- Lumiera will support this by allowing to set up additional output slots and attach this information to the current session → SessionConfigurationAttachment)
!!!Lifecycle and storage
The concrete OutputSlot implementation is owned and managed by the facility actually providing the output possibility in question. For example, the GUI provides viewer widgets, while some sound output backend provides sound ports. The associated OutputSlot implementation object is required to stay alive as long as it's registered with some OutputManager. It needs to be de-registered explicitly prior to destruction -- and this deregistration may block until all clients using this slot did terminate. Beyond that, an output slot implementation is expected to handle all kinds of failures gracefully -- preferably just emitting a signal (callback functor).
{{red{TODO 7/11: Deregistration is an unsolved problem....}}}
!!!unified data exchange cycle
The planned delivery time of a frame is used as an ID throughout that cycle
# within a defined time window prior to delivery, the client can ''allocate and retrieve the buffer'' from the BufferProvider.
# the client has to ''emit'' within a (short) time window prior to deadline
# now the slot gets exclusive access to the buffer for output, signalling the buffer release to the buffer provider when done.
!!!lapses
This data exchange protocol operates on a rather low level; there is only limited protection against timing glitches
| !step|!problem ||!consequences | !protection |
| (1)|out of time window ||interference with previous/later use of the buffer | prevent in scheduler! |
|~|does not happen ||harmless as such | emit ignored |
|~|buffer unavailable ||inhibits further operation | ↯ |
| (2)|out of time window ||harmless as such | emit ignored |
|~|out of order ||allowed, unless out of time | -- |
|~|does not happen ||frame treated as glitch | -- |
|~|buffer unallocated ||frame treated as glitch | emit ignored |
| (3)|emit missing ||frame treated as glitch | -- |
|~|fail to release buffer ||unable to use buffer further | mark unavailable |
|~|buffer unavailable ||serious malfunction of playback | request playback stop |
Thus there are two serious problem situations
* allocating the buffer out of time window bears the danger of output data corruption; but the general assumption is for the scheduler to ensure each job start time remains within the defined window and all prerequisite jobs have terminated successfully. To handle clean-up, we additionally need special jobs running always, in order, and be notified of prerequisite failures.
* failure to release a buffer in a timely fashion blocks any further use of that buffer, any further jobs in need of that buffer will die immediately. This situation can only be caused by a serious problem //within the slot, related to the output mechanism.// Thus there should be some kind of trigger (e.g. when this happens 2 times consecutively) to request aborting the playback or render as a whole.
→ SchedulerRequirements
→ OutputSlotDesign
→ OutputSlotImpl
The OutputSlot interface describes a point where generated media data can actually be sent to the external world. It is expected to be implemented by adapters and bridges to existing drivers or external interface libraries, like a viewer widget in the GUI, ALSA or Jack sound output, rendering to file, using an external media format library. The design of this core facility was rather difficult and stretched out over quite some time span -- this page documents the considerations and the decisions taken.
!intention
OutpotSlot is a metaphor to unify the organisation of actual (system level) outputs; using this concept allows to separate and abstract the data calculation and the organisation of playback and rendering from the specifics of the actual output sink. Actual output possibilities (video in GUI window, video fullscreen, sound, Jack, rendering to file) can be added and removed dynamically from various components (backend, GUI), all using the same resolution and mapping mechanisms (→ OutputManagement)
!design possibilities
!!properties as a starting point
* each OutputSlot is an unique and distinguishable entity. It corresponds explicitly to an external output
* an output slot needs to be provided, configured and registered, using an implementation specifically tailored for the kind of media data
* an output slot is always limited to a single kind of media, and to a single connection unit, but this connection may still be comprised of multiple channels.
* in order to be usable as //output sink,// an output slot needs to be //allocated,// i.e. tied to and locked for a specific client.
* this allocation is exclusive: at any time, there may be only a single client using a given output slot.
* the calculating process feeds its results into //sink handles// provided by the allocated output slot.
* allocation of an output slot leads to very specific [[timing expectations|Timings]]
* the client is required to comply to these timings and operate according to a strictly defined protocol.
* timing glitches will be detected due to this protocol; the output slot provides mechanisms for failing gracefully in these cases
!!data exchange models
Data is handed over by the client invoking an {{{emit(time,...)}}} function on the sink handle. Theoretically there are two different models how this data hand-over might be performed. This corresponds to the fact, that in some cases our own code manages the output and the buffers, while in other situations we intend to use existing library solutions or even external server applications to handle output
;buffer handover model
:the client owns the data buffer and cares for allocation and de-allocation. The {{{emit()}}}-call just propagates a pointer to the buffer holding the data ready for output. The output slot implementation in turn has the liability to copy or otherwise use this data within a given time limit.
;shared buffer model
:here the output mechanism owns the buffer. Within a certain time window prior to the expected time of the {{{emit()}}}-call, the client may obtain this buffer (pointer) to fill in the data. The slot implementation won't touch this buffer until the {{{emit()}}} handover, which in this case just provides the time and signalles the client is done with that buffer. If the data emitting handshake doesn't happen at all, it counts as late and superseded by the next handshake.
!!relation to timing
Besides the sink handles, allocation of an output slot defines some timing constraints, which are binding for the client. These include a grid of deadlines for each frame to deliver, plus a fixed //latency.// The output slot implementation itself is bound by external constraints to deliver data at a fixed framerate and aligned to an externally defined timing grid, plus the data needs to be handed over ahead of these time points by an time amount given by the latency. Depending on the data exchange model, there is an additional time window limiting the buffer management.
The assumption is for the client to have elaborate timing capabilities at his disposal. More specifically, the client is assumed to be a job running within the engine scheduler and thus can be configured to run //after// another job has finished, and to run within certain time limits. Thus the client is able to provide a //current nominal time// -- which is suitably close to the actual wall clock time. The output slot implementation can be written such as to work out from this time specification if the call is timely or overdue -- and react accordingly.
{{red{TODO 6/11}}}in this spec, both data exchange models exhibit a weakness regarding the releasing of buffers. At which time is it safe to release a buffer, when the handover didn't happen? Do we need an explicit callback, and how could this callback be triggered? This is similar to the problem of closing a network connection, i.e. the problem is generally unsolvable, but can be handled pragmatically within certain limits.
{{red{WIP 11/11}}}meanwhile I've worked out the BufferProvider interface in deail. There's now a deatiled buffer handover protocol defined, which is supported by a little state engine tracking BufferMetadata. This mechanism provides a transition to {{{BLOCKED}}} state when order or timing constraints are being violated, which practically solves this problem. How to detect and resolve such a floundered state from the engine point of view still remains to be addressed.
!!!Lifecycle and storage
The concrete OutputSlot implementation is owned and managed by the facility actually providing the output possibility in question. For example, the GUI provides viewer widgets, while some sound output backend provides sound ports. The associated OutputSlot implementation object is required to stay alive as long as it's registered with some OutputManager. It needs to be de-registered explicitly prior to destruction -- and this deregistration may block until all clients using this slot did terminate. Beyond that, an output slot implementation is expected to handle all kinds of failures gracefully -- preferably just emitting a signal (callback functor).
{{red{TODO 7/11: Deregistration is an unsolved problem....}}}
-----
!Implementation / design problems
How to handle the selection of those two data exchange models!
-- Problem is, typically this choice isn't up to the client; rather, the concrete OutputSlot implementation dictates what model to use. But, as it stands, the client needs to cooperate and behave differently to a certain degree. Unless we manage to factor out an common denominator of both models.
Thus: Client gets an {{{OutputSlot&}}}, without knowing the exact implementation type ⇒ how can the client pick up the right strategy?
Solving this problem through //generic programming// -- i.e coding both cases effectively different, but similar -- would require to provide both implementation options already at //compile time!//
{{red{currently}}} I see two possible, yet quite different approaches...
;generic
:when creating individual jobs, we utilise a //factory obtained from the output slot.//
;unified
:extend and adapt the protocol such to make both models similar; concentrate all differences //within a separate buffer provider.//
!!!discussion
the generic approach looks as it's becoming rather convoluted in practice. We'd need to hand over additional parameters to the factory, which passes them through to the actual job implementation created. And there would be a coupling between slot and job (the slot is aware it's going to be used by a job, and even provides the implementation). Obviously, a benefit is that the actual code path executed within the job is without indirections, and all written down in a single location. Another benefit is the possibility to extend this approach to cover further buffer handling models -- it doesn't pose any requirements on the structure of the buffer handling --
On the other hand, if we accept to retrieve the buffer(s) via an indirection, which we kind of do anyway //within the render node implementation// -- the unified model looks more like a clean solution. It's more like doing away with some local optimisations possible if we handle the models explicitly, so it's not much of a loss, given that the majority of the processing time will be spent within the inner pixel calculation loops for frame processing anyway. When following this approach, the BufferProvider becomes a third, independent partner, and the slot cooperates tightly with this buffer provider, while the client (processing node) still just talks to the slot. Basically, this unified solution works like extending the shared buffer model to both cases.
⇒ __conclusion__: go for the unified approach!
!!!unified data exchange cycle
The planned delivery time of a frame is used as an ID throughout that cycle
# within a defined time window prior to delivery, the client can ''allocate and retrieve the buffer'' from the BufferProvider.
# the client has to ''emit'' within a (short) time window prior to deadline
# now the slot gets exclusive access to the buffer for output, signalling the buffer release to the buffer provider when done.
→ OutputSlotImpl
OutputSlot is an abstraction, allowing unified treatment of various physical output connections from within the render jobs. The actual output slot is a subclass object, created and managed from the "driver code" for a specific output connection. Moreover, each output slot will be outfitted with a concrete BufferProvider to reflect the actual buffer handling policy applicable for this specific output connection. Some output connections might e.g. require delivery of the media data into a buffer residing on external hardware, while others work just fine when pointed to some arbitrary memory block holding generated data.
!operation steps
[>img[Sequenz of output data exchange steps|uml/fig145157.png]]The OutputSlot class defines some standard operations as protected virtual functions. These represent the actual steps in the data exchange protocol corresponding to this output slot.
;lock()
:claims the specified buffer for exclusive use.
:Client may now dispose output data
;transfer()
:transfers control from the client to the actual output connection for feeding data.
:Triggered by the client invoking {{{emit()}}}
;pushout()
:request the actual implementation to push the data to output.
:Usually invoked from the {{{transfer()}}} implementation, alternatively from a separate thread.
!buffer states
While the BufferProvider abstracts away the actual access to the output buffers and just hands out a ''buffer handle'', the server side (here the concrete output slot) is allowed to associate and maintain a ''state flag'' with each buffer. The general assumption is that writing this state flag is atomic, and that other facilities will care for the necessary memory barriers (that is: the output slot and the buffer provider will just access this state flag without much ado). The generic part of the output slot implementation utilises this buffer state flag to implement a state machine, which -- together with the timing constraints established with the [[help of the scheduler|SchedulerRequirements]] -- ensures sane access to the buffer without collisions.
| !state||>| !lock() |>| !transfer() |>| !pushout() |
| {{{NIL}}}||↯| · |↯| | ↯ | |
| {{{FREE}}}||✔|↷ locked |✔|· (ignore) | · | · |
| {{{LOCKED}}}||↯|↷ emitted |✔|↷ emitted | · |↷ free |
| {{{EMITTED}}}||↯|↷ blocked |↯|↷ blocked | · |↷ free |
| {{{BLOCKED}}}||↯| ✗ |↯| ✗ | ∅ |↷ free |
where · means no operation, ✔ marks the standard cases (OK response to caller), ↯ will throw and thus kill the calling job, ∅ will treat this frame as //glitch,// ✗ will request playback stop.
The rationale is for all states out-of-order to transition into the {{{BLOCKED}}}-state eventually, which, when hit by the next operation, will request playback stop.
The Lumiera Processing Layer is comprised of various subsystems and can be separated into a low-level and a high-level part. At the low-level end is the [[Render Engine|OverviewRenderEngine]] which basically is a network of render nodes cooperating closely with the Backend Layer in order to carry out the actual playback and media transforming calculations. Whereas on the high-level side we find several different [[Media Objects|MObjects]] that can be placed into the session, edited and manipulated. This is complemented by the [[Asset Management|Asset]], which is the "bookkeeping view" of all the different "things" within each [[Session|SessionOverview]].
There is rather strong separation between these two levels, and — <br/>correspondingly you'll encounter the data held within the Processing Layer organized in two different views, the ''high-level-model'' and the ''low-level-model''
* from users (and GUI) perspective, you'll see a [[Session|SessionOverview]] with a timeline-like structure, where various [[Media Objects|MObjects]] are arranged and [[placed|Placement]]. By looking closer, you'll find that there are data connections and all processing is organized around processing chains or [[pipes|Pipe]], which can be either global (in the Session) or local (in real or [[virtual|VirtualClip]] clips)
* when dealing with the actual calculations in the Engine (→ see OverviewRenderEngine), you won't find any Tracks, Media Objects or Pipes — rather you'll find a network of interconnected [[render nodes|ProcNode]] forming the low level model. Each structurally constant segment of the timeline corresponds to a separate node network providing an ExitNode corresponding to each of the global pipes; pulling frames from them means running the engine.
* it is the job of the [[Builder]] create and wire up this render nodes network when provided with a given hig-level-model. So, actually the builder (together with the so called [[Fixture]]) form an isolation layer in the middle, separating the //editing part // from the //processing part.//
[img[Block Diagram|uml/fig128005.png]]
Render Engine, [[Builder]] and [[Controller]] are closely related Subsystems. Actually, the [[Builder]] //creates// a newly configured Render Engine //for every// RenderProcess. Before doing so, it queries from the Session (or, to be more precise, from the [[Fixture]] within the current Session) all necessary Media Object Placement information. The [[Builder]] then derives from this information the actual assembly of [[Processing Nodes|ProcNode]] comprising the Render Engine. Thus:
* the source of the build process is a sequence of absolute (explicit) [[Placements|Placement]] called the [[Playlist]]
* the [[build process|BuildProcess]] is driven, configured and controlled by the [[Controller]] subsystem component. It encompasses the actual playback configuration and State of the System.
* the resulting Render Engine is a list of [[Processors]], each configured to calculate a segment of the timeline with uniform properties. Each of these Processors in turn is a graph of interconnected ProcNode.s.
see also: RenderEntities, [[two Examples (Object diagrams)|Examples]]
{{red{TODO: adjust terminology in this drawing: "Playlist" → "Fixture" and "Graph" → "Segment"}}}
[img[Overview: Components of the Renderengine|uml/fig128261.png]]
<!--{{{-->
<div class='header' macro='gradient vert [[ColorPalette::PrimaryLight]] [[ColorPalette::PrimaryMid]]'>
<div class='headerShadow'>
<span class='siteTitle' refresh='content' tiddler='SiteTitle'></span>
<span class='siteSubtitle' refresh='content' tiddler='SiteSubtitle'></span>
</div>
<div class='headerForeground'>
<span class='siteTitle' refresh='content' tiddler='SiteTitle'></span>
<span class='siteSubtitle' refresh='content' tiddler='SiteSubtitle'></span>
</div>
</div>
<!-- horizontal MainMenu -->
<div id='topMenu' refresh='content' tiddler='MainMenu'></div>
<!-- original MainMenu menu -->
<!-- <div id='mainMenu' refresh='content' tiddler='MainMenu'></div> -->
<div id='sidebar'>
<div id='sidebarOptions' refresh='content' tiddler='SideBarOptions'></div>
<div id='sidebarTabs' refresh='content' force='true' tiddler='SideBarTabs'></div>
</div>
<div id='displayArea'>
<div id='messageArea'></div>
<div id='tiddlerDisplay'></div>
</div>
<!--}}}-->
A ParamProvider is the counterpart for (one or many) [[Parameter]] instances. It implements the value access function made available by the Parameter object to its clients.
To give a concrete example:
* a Fade Plugin needs the actual fade value for Frame t=xxx
* the Plugin has a Parameter Object (from which we could query the information of this parameter being a continuous float function)
* this Parameter Object provides a getValue() function, which is internally linked (i.e. by configuration) to a //Parameter Provider//
* the actual object implementing the ParamProvider Interface could be a Automation MObject located somewhere in the session and would do bezier interpolation on a given keyframe set.
* Param providers are created on demand; while building the Render Engine configuration actually at work, the Builder would have to setup a link between the Plugin Parameter Object and the ParamProvider; he can do so, because he sees the link between the Automation MObject and the corresponding Effect MObject
!!ParamProvider ownership and lifecycle
Actually, ParamProvider is just an interface which is implemented either by a constant or an [[Automation]] function. In both cases, access is via direct reference, while the link to the ParamProvider is maintained by a smart-ptr, which — in the case of automation may share ownership with the [[Placement]] of the automation data set.
→ see the class diagram for [[Automation]]
→ see EffectHandling
Parameters are all possibly variable control values used within the Render Engine. Contrast this with configuration values, which are considered to be fixed and need an internal reset of the application (or session) state to take effect.
A ''Parameter Object'' provides a descriptor of the kind of parameter, together with a function used to pull the //actual value// of this parameter. Here, //actual// has a two-fold meaning:
* if called without a time specification, it is either a global (but variable) system or session parameter or a default value for automated Parameters. (the intention is to treat this cases uniformly)
* if called with a time specification, it is the query for an — probably interpolated — [[Automation]] value at this absolute time. The corresponding ParamProvider should fall back transparently to a default or session value if no time varying data is available
{{red{TODO: define how Automation works}}}
/***
|<html><a name="Top"/></html>''Name:''|PartTiddlerPlugin|
|''Version:''|1.0.6 (2006-11-07)|
|''Source:''|http://tiddlywiki.abego-software.de/#PartTiddlerPlugin|
|''Author:''|UdoBorkowski (ub [at] abego-software [dot] de)|
|''Licence:''|[[BSD open source license]]|
|''TiddlyWiki:''|2.0|
|''Browser:''|Firefox 1.0.4+; InternetExplorer 6.0|
!Table of Content<html><a name="TOC"/></html>
* <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Description',null, event)">Description, Syntax</a></html>
* <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Applications',null, event)">Applications</a></html>
** <html><a href="javascript:;" onclick="window.scrollAnchorVisible('LongTiddler',null, event)">Refering to Paragraphs of a Longer Tiddler</a></html>
** <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Citation',null, event)">Citation Index</a></html>
** <html><a href="javascript:;" onclick="window.scrollAnchorVisible('TableCells',null, event)">Creating "multi-line" Table Cells</a></html>
** <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Tabs',null, event)">Creating Tabs</a></html>
** <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Sliders',null, event)">Using Sliders</a></html>
* <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Revisions',null, event)">Revision History</a></html>
* <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Code',null, event)">Code</a></html>
!Description<html><a name="Description"/></html>
With the {{{<part aPartName> ... </part>}}} feature you can structure your tiddler text into separate (named) parts.
Each part can be referenced as a "normal" tiddler, using the "//tiddlerName//''/''//partName//" syntax (e.g. "About/Features"). E.g. you may create links to the parts, use it in {{{<<tiddler...>>}}} or {{{<<tabs...>>}}} macros etc.
''Syntax:''
|>|''<part'' //partName// [''hidden''] ''>'' //any tiddler content// ''</part>''|
|//partName//|The name of the part. You may reference a part tiddler with the combined tiddler name "//nameOfContainerTidder//''/''//partName//.|
|''hidden''|When defined the content of the part is not displayed in the container tiddler. But when the part is explicitly referenced (e.g. in a {{{<<tiddler...>>}}} macro or in a link) the part's content is displayed.|
|<html><i>any tiddler content</i></html>|<html>The content of the part.<br>A part can have any content that a "normal" tiddler may have, e.g. you may use all the formattings and macros defined.</html>|
|>|~~Syntax formatting: Keywords in ''bold'', optional parts in [...]. 'or' means that exactly one of the two alternatives must exist.~~|
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!Applications<html><a name="Applications"/></html>
!!Refering to Paragraphs of a Longer Tiddler<html><a name="LongTiddler"/></html>
Assume you have written a long description in a tiddler and now you want to refer to the content of a certain paragraph in that tiddler (e.g. some definition.) Just wrap the text with a ''part'' block, give it a nice name, create a "pretty link" (like {{{[[Discussion Groups|Introduction/DiscussionGroups]]}}}) and you are done.
Notice this complements the approach to first writing a lot of small tiddlers and combine these tiddlers to one larger tiddler in a second step (e.g. using the {{{<<tiddler...>>}}} macro). Using the ''part'' feature you can first write a "classic" (longer) text that can be read "from top to bottom" and later "reuse" parts of this text for some more "non-linear" reading.
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!!Citation Index<html><a name="Citation"/></html>
Create a tiddler "Citations" that contains your "citations".
Wrap every citation with a part and a proper name.
''Example''
{{{
<part BAX98>Baxter, Ira D. et al: //Clone Detection Using Abstract Syntax Trees.//
in //Proc. ICSM//, 1998.</part>
<part BEL02>Bellon, Stefan: //Vergleich von Techniken zur Erkennung duplizierten Quellcodes.//
Thesis, Uni Stuttgart, 2002.</part>
<part DUC99>Ducasse, Stéfane et al: //A Language Independent Approach for Detecting Duplicated Code.//
in //Proc. ICSM//, 1999.</part>
}}}
You may now "cite" them just by using a pretty link like {{{[[Citations/BAX98]]}}} or even more pretty, like this {{{[[BAX98|Citations/BAX98]]}}}.
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!!Creating "multi-line" Table Cells<html><a name="TableCells"/></html>
You may have noticed that it is hard to create table cells with "multi-line" content. E.g. if you want to create a bullet list inside a table cell you cannot just write the bullet list
{{{
* Item 1
* Item 2
* Item 3
}}}
into a table cell (i.e. between the | ... | bars) because every bullet item must start in a new line but all cells of a table row must be in one line.
Using the ''part'' feature this problem can be solved. Just create a hidden part that contains the cells content and use a {{{<<tiddler >>}}} macro to include its content in the table's cell.
''Example''
{{{
|!Subject|!Items|
|subject1|<<tiddler ./Cell1>>|
|subject2|<<tiddler ./Cell2>>|
<part Cell1 hidden>
* Item 1
* Item 2
* Item 3
</part>
...
}}}
Notice that inside the {{{<<tiddler ...>>}}} macro you may refer to the "current tiddler" using the ".".
BTW: The same approach can be used to create bullet lists with items that contain more than one line.
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!!Creating Tabs<html><a name="Tabs"/></html>
The build-in {{{<<tabs ...>>}}} macro requires that you defined an additional tiddler for every tab it displays. When you want to have "nested" tabs you need to define a tiddler for the "main tab" and one for every tab it contains. I.e. the definition of a set of tabs that is visually displayed at one place is distributed across multiple tiddlers.
With the ''part'' feature you can put the complete definition in one tiddler, making it easier to keep an overview and maintain the tab sets.
''Example''
The standard tabs at the sidebar are defined by the following eight tiddlers:
* SideBarTabs
* TabAll
* TabMore
* TabMoreMissing
* TabMoreOrphans
* TabMoreShadowed
* TabTags
* TabTimeline
Instead of these eight tiddlers one could define the following SideBarTabs tiddler that uses the ''part'' feature:
{{{
<<tabs txtMainTab
Timeline Timeline SideBarTabs/Timeline
All 'All tiddlers' SideBarTabs/All
Tags 'All tags' SideBarTabs/Tags
More 'More lists' SideBarTabs/More>>
<part Timeline hidden><<timeline>></part>
<part All hidden><<list all>></part>
<part Tags hidden><<allTags>></part>
<part More hidden><<tabs txtMoreTab
Missing 'Missing tiddlers' SideBarTabs/Missing
Orphans 'Orphaned tiddlers' SideBarTabs/Orphans
Shadowed 'Shadowed tiddlers' SideBarTabs/Shadowed>></part>
<part Missing hidden><<list missing>></part>
<part Orphans hidden><<list orphans>></part>
<part Shadowed hidden><<list shadowed>></part>
}}}
Notice that you can easily "overwrite" individual parts in separate tiddlers that have the full name of the part.
E.g. if you don't like the classic timeline tab but only want to see the 100 most recent tiddlers you could create a tiddler "~SideBarTabs/Timeline" with the following content:
{{{
<<forEachTiddler
sortBy 'tiddler.modified' descending
write '(index < 100) ? "* [["+tiddler.title+"]]\n":""'>>
}}}
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!!Using Sliders<html><a name="Sliders"/></html>
Very similar to the build-in {{{<<tabs ...>>}}} macro (see above) the {{{<<slider ...>>}}} macro requires that you defined an additional tiddler that holds the content "to be slid". You can avoid creating this extra tiddler by using the ''part'' feature
''Example''
In a tiddler "About" we may use the slider to show some details that are documented in the tiddler's "Details" part.
{{{
...
<<slider chkAboutDetails About/Details details "Click here to see more details">>
<part Details hidden>
To give you a better overview ...
</part>
...
}}}
Notice that putting the content of the slider into the slider's tiddler also has an extra benefit: When you decide you need to edit the content of the slider you can just doubleclick the content, the tiddler opens for editing and you can directly start editing the content (in the part section). In the "old" approach you would doubleclick the tiddler, see that the slider is using tiddler X, have to look for the tiddler X and can finally open it for editing. So using the ''part'' approach results in a much short workflow.
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!Revision history<html><a name="Revisions"/></html>
* v1.0.6 (2006-11-07)
** Bugfix: cannot edit tiddler when UploadPlugin by Bidix is installed. Thanks to José Luis González Castro for reporting the bug.
* v1.0.5 (2006-03-02)
** Bugfix: Example with multi-line table cells does not work in IE6. Thanks to Paulo Soares for reporting the bug.
* v1.0.4 (2006-02-28)
** Bugfix: Shadow tiddlers cannot be edited (in TW 2.0.6). Thanks to Torsten Vanek for reporting the bug.
* v1.0.3 (2006-02-26)
** Adapt code to newly introduced Tiddler.prototype.isReadOnly() function (in TW 2.0.6). Thanks to Paulo Soares for reporting the problem.
* v1.0.2 (2006-02-05)
** Also allow other macros than the "tiddler" macro use the "." in the part reference (to refer to "this" tiddler)
* v1.0.1 (2006-01-27)
** Added Table of Content for plugin documentation. Thanks to RichCarrillo for suggesting.
** Bugfix: newReminder plugin does not work when PartTiddler is installed. Thanks to PauloSoares for reporting.
* v1.0.0 (2006-01-25)
** initial version
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!Code<html><a name="Code"/></html>
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
***/
//{{{
//============================================================================
// PartTiddlerPlugin
// Ensure that the PartTiddler Plugin is only installed once.
//
if (!version.extensions.PartTiddlerPlugin) {
version.extensions.PartTiddlerPlugin = {
major: 1, minor: 0, revision: 6,
date: new Date(2006, 10, 7),
type: 'plugin',
source: "http://tiddlywiki.abego-software.de/#PartTiddlerPlugin"
};
if (!window.abego) window.abego = {};
if (version.major < 2) alertAndThrow("PartTiddlerPlugin requires TiddlyWiki 2.0 or newer.");
//============================================================================
// Common Helpers
// Looks for the next newline, starting at the index-th char of text.
//
// If there are only whitespaces between index and the newline
// the index behind the newline is returned,
// otherwise (or when no newline is found) index is returned.
//
var skipEmptyEndOfLine = function(text, index) {
var re = /(\n|[^\s])/g;
re.lastIndex = index;
var result = re.exec(text);
return (result && text.charAt(result.index) == '\n')
? result.index+1
: index;
}
//============================================================================
// Constants
var partEndOrStartTagRE = /(<\/part>)|(<part(?:\s+)((?:[^>])+)>)/mg;
var partEndTagREString = "<\\/part>";
var partEndTagString = "</part>";
//============================================================================
// Plugin Specific Helpers
// Parse the parameters inside a <part ...> tag and return the result.
//
// @return [may be null] {partName: ..., isHidden: ...}
//
var parseStartTagParams = function(paramText) {
var params = paramText.readMacroParams();
if (params.length == 0 || params[0].length == 0) return null;
var name = params[0];
var paramsIndex = 1;
var hidden = false;
if (paramsIndex < params.length) {
hidden = params[paramsIndex] == "hidden";
paramsIndex++;
}
return {
partName: name,
isHidden: hidden
};
}
// Returns the match to the next (end or start) part tag in the text,
// starting the search at startIndex.
//
// When no such tag is found null is returned, otherwise a "Match" is returned:
// [0]: full match
// [1]: matched "end" tag (or null when no end tag match)
// [2]: matched "start" tag (or null when no start tag match)
// [3]: content of start tag (or null if no start tag match)
//
var findNextPartEndOrStartTagMatch = function(text, startIndex) {
var re = new RegExp(partEndOrStartTagRE);
re.lastIndex = startIndex;
var match = re.exec(text);
return match;
}
//============================================================================
// Formatter
// Process the <part ...> ... </part> starting at (w.source, w.matchStart) for formatting.
//
// @return true if a complete part section (including the end tag) could be processed, false otherwise.
//
var handlePartSection = function(w) {
var tagMatch = findNextPartEndOrStartTagMatch(w.source, w.matchStart);
if (!tagMatch) return false;
if (tagMatch.index != w.matchStart || !tagMatch[2]) return false;
// Parse the start tag parameters
var arguments = parseStartTagParams(tagMatch[3]);
if (!arguments) return false;
// Continue processing
var startTagEndIndex = skipEmptyEndOfLine(w.source, tagMatch.index + tagMatch[0].length);
var endMatch = findNextPartEndOrStartTagMatch(w.source, startTagEndIndex);
if (endMatch && endMatch[1]) {
if (!arguments.isHidden) {
w.nextMatch = startTagEndIndex;
w.subWikify(w.output,partEndTagREString);
}
w.nextMatch = skipEmptyEndOfLine(w.source, endMatch.index + endMatch[0].length);
return true;
}
return false;
}
config.formatters.push( {
name: "part",
match: "<part\\s+[^>]+>",
handler: function(w) {
if (!handlePartSection(w)) {
w.outputText(w.output,w.matchStart,w.matchStart+w.matchLength);
}
}
} )
//============================================================================
// Extend "fetchTiddler" functionality to also recognize "part"s of tiddlers
// as tiddlers.
var currentParent = null; // used for the "." parent (e.g. in the "tiddler" macro)
// Return the match to the first <part ...> tag of the text that has the
// requrest partName.
//
// @return [may be null]
//
var findPartStartTagByName = function(text, partName) {
var i = 0;
while (true) {
var tagMatch = findNextPartEndOrStartTagMatch(text, i);
if (!tagMatch) return null;
if (tagMatch[2]) {
// Is start tag
// Check the name
var arguments = parseStartTagParams(tagMatch[3]);
if (arguments && arguments.partName == partName) {
return tagMatch;
}
}
i += tagMatch[0].length;
}
}
// Return the part "partName" of the given parentTiddler as a "readOnly" Tiddler
// object, using fullName as the Tiddler's title.
//
// All remaining properties of the new Tiddler (tags etc.) are inherited from
// the parentTiddler.
//
// @return [may be null]
//
var getPart = function(parentTiddler, partName, fullName) {
var text = parentTiddler.text;
var startTag = findPartStartTagByName(text, partName);
if (!startTag) return null;
var endIndexOfStartTag = skipEmptyEndOfLine(text, startTag.index+startTag[0].length);
var indexOfEndTag = text.indexOf(partEndTagString, endIndexOfStartTag);
if (indexOfEndTag >= 0) {
var partTiddlerText = text.substring(endIndexOfStartTag,indexOfEndTag);
var partTiddler = new Tiddler();
partTiddler.set(
fullName,
partTiddlerText,
parentTiddler.modifier,
parentTiddler.modified,
parentTiddler.tags,
parentTiddler.created);
partTiddler.abegoIsPartTiddler = true;
return partTiddler;
}
return null;
}
// Hijack the store.fetchTiddler to recognize the "part" addresses.
//
var oldFetchTiddler = store.fetchTiddler ;
store.fetchTiddler = function(title) {
var result = oldFetchTiddler.apply(this, arguments);
if (!result && title) {
var i = title.lastIndexOf('/');
if (i > 0) {
var parentName = title.substring(0, i);
var partName = title.substring(i+1);
var parent = (parentName == ".")
? currentParent
: oldFetchTiddler.apply(this, [parentName]);
if (parent) {
return getPart(parent, partName, parent.title+"/"+partName);
}
}
}
return result;
};
// The user must not edit a readOnly/partTiddler
//
config.commands.editTiddler.oldIsReadOnlyFunction = Tiddler.prototype.isReadOnly;
Tiddler.prototype.isReadOnly = function() {
// Tiddler.isReadOnly was introduced with TW 2.0.6.
// For older version we explicitly check the global readOnly flag
if (config.commands.editTiddler.oldIsReadOnlyFunction) {
if (config.commands.editTiddler.oldIsReadOnlyFunction.apply(this, arguments)) return true;
} else {
if (readOnly) return true;
}
return this.abegoIsPartTiddler;
}
config.commands.editTiddler.handler = function(event,src,title)
{
var t = store.getTiddler(title);
// Edit the tiddler if it either is not a tiddler (but a shadowTiddler)
// or the tiddler is not readOnly
if(!t || !t.abegoIsPartTiddler)
{
clearMessage();
story.displayTiddler(null,title,DEFAULT_EDIT_TEMPLATE);
story.focusTiddler(title,"text");
return false;
}
}
// To allow the "./partName" syntax in macros we need to hijack
// the invokeMacro to define the "currentParent" while it is running.
//
var oldInvokeMacro = window.invokeMacro;
function myInvokeMacro(place,macro,params,wikifier,tiddler) {
var oldCurrentParent = currentParent;
if (tiddler) currentParent = tiddler;
try {
oldInvokeMacro.apply(this, arguments);
} finally {
currentParent = oldCurrentParent;
}
}
window.invokeMacro = myInvokeMacro;
// Scroll the anchor anchorName in the viewer of the given tiddler visible.
// When no tiddler is defined use the tiddler of the target given event is used.
window.scrollAnchorVisible = function(anchorName, tiddler, evt) {
var tiddlerElem = null;
if (tiddler) {
tiddlerElem = document.getElementById(story.idPrefix + tiddler);
}
if (!tiddlerElem && evt) {
var target = resolveTarget(evt);
tiddlerElem = story.findContainingTiddler(target);
}
if (!tiddlerElem) return;
var children = tiddlerElem.getElementsByTagName("a");
for (var i = 0; i < children.length; i++) {
var child = children[i];
var name = child.getAttribute("name");
if (name == anchorName) {
var y = findPosY(child);
window.scrollTo(0,y);
return;
}
}
}
} // of "install only once"
//}}}
/***
<html><sub><a href="javascript:;" onclick="scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!Licence and Copyright
Copyright (c) abego Software ~GmbH, 2006 ([[www.abego-software.de|http://www.abego-software.de]])
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this
list of conditions and the following disclaimer in the documentation and/or other
materials provided with the distribution.
Neither the name of abego Software nor the names of its contributors may be
used to endorse or promote products derived from this software without specific
prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
<html><sub><a href="javascript:;" onclick="scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
***/
Facility guiding decisions regarding the strategy to employ for rendering or wiring up connections. The PathManager is querried through the OperationPoint, when executing the connection steps within the Build process.
Pipes play an central role within the Proc Layer, because for everything placed and handled within the session, the final goal is to get it transformed into data which can be retrieved at some pipe's exit port. Pipes are special facilities, rather like inventory, separate and not treated like all the other objects.
We don't distinguish between "input" and "output" ports — rather, pipes are thought to be ''hooks for making connections to''. By following this line of thought, each pipe has an input side and an output side and is in itself something like a ''Bus'' or ''processing chain''. Other processing entities like effects and transitions can be placed (attached) at the pipe, resulting them to be appended to form this chain. Likewise, we can place [[wiring requests|WiringRequest]] to the pipe, meaning we want it connected so to send it's output to another destination pipe. The [[Builder]] may generate further wiring requests to fulfil the placement of other entities.
Thus //Pipes are the basic building blocks// of the whole render network. We distinguish ''global available'' Pipes, which are like the sum groups of a mixing console, and the ''lokal pipe'' or [[source ports|ClipSourcePort]] of the individual clips, which exist only within the duration of the corresponding clip. The design //limits the possible kinds of pipes // to these two types — thus we can build local processing chains at clips and global processing chains at the global pipes of the session and that's all we can do. (because of the flexibility which comes with the concept of [[placements|Placement]], this is no real limitation)
Pipes are denoted and addressed by ~Pipe-IDs, implemented as ID of a pipe asset. Besides explicitly named pipes, there are some generic placeholder ~IDs for a standard configured pipe of a given type. This is done to avoid creating a separate ~Pipe-ID for each and every clip to build.
While pipes are rather rigid building blocks -- and especially are limited to a single StreamType without conversions, the interconnections or ''routing'' links to the contrary are more flexible. They are specified and controled through the use of an OutputDesignation, which, when fully resolved, should again yield a target ~Pipe-ID to connect. The mentioned resolution of an output designation typically involves an OutputMapping.
Similar structures and mechanisms are extended beyond the core model: The GUI can connect the viewer(s) to some pipe (and moreover can use [[probe points|ProbePoint]] placed like effects and connected to some pipe), and likewise, when starting a ''render'', we get the opportunity to specify the ModelPort (exit point of a GlobalPipe) to pull the data from. Pulling data from some pipe is the (only) way to activate the render nodes network reachable from this pipe.
→ [[Handling of Tracks|TrackHandling]]
→ [[Handling of Pipes|PipeHandling]]
!Identification
Pipes are distinct objects and can be identified by their asset ~IDs. Besides, as for all [[structural assets|StructAsset]] there are extended query capabilities, including a symbolic pipe-id and a media (stream) type id. Any pipe can accept and deliver exactly one media stream kind (which may be inherently structured though, e.g. spatial sound systems or stereoscopic video)
!creating pipes
Pipe assets are created automatically by being used and referred. Each [[Timeline]] holds a collection of global pipes, attached to the BindingMO, which is the representation or attachement point of the Timeline within the HighLevelModel ([[Session]]) ({{red{todo: implementation missing as of 11/09}}}), and further pipes can be created by using a new pipe reference in some placement. Moreover, every clip has an (implicit) [[source port|ClipSourcePort]], which will appear as pipe asset when first used (referred) while [[building|BuildProcess]]. Note that creating a new pipe implies using a [[processing pattern|ProcPatt]], which will be queried from the [[Defaults Manager|DefaultsManagement]] (resulting in the use of some preconfigured pattern or maybe the creation of a new ProcPatt object if necessary)
!removal
Deleting a Pipe is an advanced operation, because it includes finding and "detaching" all references, otherwise the pipe will leap back into existence immediately. Thus, global pipe entries in the Session and pipe references in [[locating pins|LocatingPin]] within any placement have to be removed, while clips using a given source port will be disabled. {{red{todo: implementation deferred}}}
!using Pipes
there is not much you can do directly with a pipe asset. It is an point of reference, after all. Any connection or routing to a target pipe is done by a placement with a suitable WiringPlug in some part of the timeline (→ OutputDesignation), so it isn't stored with the pipe. You can edit the (user visible) description an you can globally disable a pipe asset. The pipe's ID and media stream type of course are fixed, because any connection and referral (via the asset ID) is based on them. Later on, we should provide a {{{rewire(oldPipe, newPipe)}}} to search any ref to the {{{oldPipe}}} and try to rewrite it to use the {{{newPipe}}}, possibly with a new media stream type.
Pipes are integrated with the [[management of defaults|DefaultsManagement]]. For example, any pipe implicitly uses some [[processing pattern|ProcPatt]] — it may default to the empty pattern. This way, any kind of standard wiring might be applied to the pipes (e.g a fader for audio, similar to the classic mixing consoles). This //is // a global property of the pipe, but — contrary to the stream type — this pattern may be switched {{red{really? -- lots of open questions here as of 11/10}}}
A Placement represents a //relation:// it is always linked to a //Subject// (this being a [[Media Object|MObject]]) and has the meaning to //place// this Subject in some manner, either relatively to other Media Objects, by some Constraint or simply absolute at (time, output). The latter case is especially important for the build process and thus represented by a special [[Sub-Interface ExplicitPlacement|ExplicitPlacement]]. Besides this simple cases, Placements can also express more specific kinds of "locating" an object, like placing a sound source at a pan position or placing a video clip at a given layer (above or below another video clip)
So basically placements represent a query interface: you can allways ask the placement to find out about the position of the related object in terms of (time, output), and — depending on the specific object and situation — also about these additional [[placement derived dimensions|PlacementDerivedDimension]] like sound pan or layer order or similar things which also fit into the general concept of "placing" an object.
The fact of being placed in the [[Session|SessionOverview]] is constitutive for all sorts of [[MObject]]s, without Placement they make no sense. Thus — technically — Placements act as ''smart pointers''. Of course, there are several kinds of Placements and they are templated on the type of MObject they are refering to. Placements can be //aggregated// to increasingly constrain the resulting "location" of the refered ~MObject. See → [[handling of Placements|PlacementHandling]] for more details
!Placements as instance
Effectively, the placement of a given MObject into the Session acts as setting up an concrete instance of this object. This way, placements exhibit a dual nature. When viewed on themselves, like any reference or smart-pointer they behave like values. But, by adding a placement to the session, we again create a unique distinguishable entity with reference semantics: there could be multiple placements of the same object but with varying placement properties. Such a placement-bound-into-the-session is denoted by an generic placement-ID or (as we call it) → PlacementRef; behind the scenes there is a PlacementIndex keeping track of those "instances" — allowing us to hand out the PlacementRef (which is just an opaque id) to client code outside the Proc-Layer and generally use it as an shorthand, behaving as if it was an MObject instance
For any [[media object|MObject]] within the session, we can allways at least query the time (reference/start) point and the output destination from the [[Placement]], by which the object is being handled. But the simple act of placing an object in some way, can — depending on the context — create additional degrees of freedom. To list some important examples:
* placing a video clip overlapping with other clips on other tracks creates the possibility for the clip to be above another clip or to be combined in various other ways with the other clips at the same time position
* placing a mono sound object plugged to a stereophoic output destination creates the freedom to define the pan position
The Placement interface allows to query for these additional //parameter values derived from the fact of being placed.//
!defining additional dimensions
probably a LocatingPin but... {{red{TODO any details are yet unknown as of 5/08}}}
!querying additional dimensions
basically you resolve the Placement, yielding an ExplicitPlacement... {{red{TODO but any details of how additional dimensions are resolved is still undefined as of 5/08}}}
[[Placement]]s are at the very core of all [[editing operations|EditingOperations]], because they act as handles (smart pointers) to access the [[media objects|MObject]] to be manipulated. Placements themselves are lightweight and can be handled with //value semantics//. But, when adding a Placement to the [[Session]], it gains an distinguishable identity and should be treated by reference from then on: changing the location properties of this placement has a tangible effect on the way the placed object appears in the context of the session. Besides the direct (language) references, there is a special PlacementRef type which builds on this registration of the placement within the session, can be represented as POD and thus passed over external interfaces. Many editing tasks include finding some Placement in the session and reference as parameter. By acting //on the Placement object,// we can change parameters of the way the media object is placed (e.g. adjust an offset), while by //dereferencing// the Placement object, we access the "real" media object (e.g. for trimming its length). Placements are ''templated'' on the type of the actual ~MObject they refer to, thus defining the interface/methods usable on this object.
Actually, the way each Placement ties and locates its subject is implemented by one or several small LocatingPin objects, where subclasses of LocatingPin implement the various different methods of placing and resolving the final location. Notably, we can give a ~FixedLocation or we can atach to another ~MObject to get a ~RelativeLocation, etc. In the typical use case, these ~LocatingPins are added to the Placement, but never retrieved directly. Rather the Placement acts as a ''query interface'' for determining the location of the related object. Here, "location" can be thought of as encompassing multiple dimenstions at the same time. An object can be
* located at a specific point in time
* related to and plugged into a specific output or global bus
* defined to have a position within some [[context-dependant additional dimensions|PlacementDerivedDimension]] like
** the pan position, either on the stereophoic base, or within a fully periphoic (spatial) sound system
** the layer order and overlay mode for video (normal, additive, subtractive, masking)
** the stereoscopic window position (depth parameter) for 3D video
** channel and parameter selection for MIDI data
Placements have //value semantics,// i.e. we don't stress the identity of a placement object (~MObjects on the other hand //do have// a distinguishable identity): initially, you create a Placement parametrized to some specific kind by adding [[LocatingPin]]s (fixed, relative,...) and possibliy you use a subclass of {{{Placement<MObject>}}} to encode additional type information, say {{{Placement<Clip>}}}, but later on, you treat the placement polymorphically and don't care about its kind. The sole purpose of the placement's kind is to select some virtual function implementing the desired behaviour. There is no limitation to one single Placement per ~MObject, indeed we can have several different Placements of the same MObject (from a users point of view, these behave like being clones). Besides, we can ''aggregate'' additional [[LocatingPin]]s to one Placements, resulting in their properties and constraints being combined to yield the actual position of the referred ~MObject.
!design decisions
* the actual way of placing is implemented similar to the ''State Pattern'' by small embedded LocatingPin objects.
* these LocatingPin objects form a ''decorator'' like chain
* resolving into an ExplicitPlacement traverses this chain
* //overconstraining// a placement is not an error, we just stop traversing the chain (ignoring the remaining additional Placements) at the moment the position is completely defined.
* placements can be treated like values, but incorporate an identity tag for the purpose of registering with the session.
* we provide subclasses to be able to form collections of e.g. {{{Placement<Effect>}}}, but we don't stress polymorphism here. → PlacementType
* Why was the question how to access a ~MObject subinterface a Problem?
*# we want the Session/Fixture to be a collection of Placements. This means, either we store pointers, or Placement needs to be //one// unique type!
*# but if Placement is //a single type//, then we can get only MObjects from a Placement.
*# then either we had to do everything by a visitor (which gets the concrete subtype dynamically), or we'd end up switching on type.
An implementation facility used to keep track of individual Placements and their relations.
Especially, the [[Session]] maintains such an index, allowing to use the (opaque) PlacementRef tags for referring to a specific "instance" of an MObject, //placed// in a unique way into the current session. And, moreover, this index allows for one placement referring to another placement, so to implement a //relative// placement mode. Because there is an index behind the scenes, it is possible to actually access such a referral in the reverse direction, which is necessary for implementing the desired placement behaviour (if an object instance used as anchor is moved, all objects placed relatively to it have to move accordingly, which necessitates finding those other objects).
Besides searching, [[placement instances|Placement]] can be added to the index, thereby creating a copy managed by the backing data structure. Thus, the session's PlacementIndex is the backbone of the session data structure, and the session's contents are actually contained within it.
!rationale of the choosen implementation
What implementation approach to take for the index largely depends on the usage pattern. Generally speaking, the Lumiera [[Session]] is a collection of MObjects attached by [[Placement]]; any relations between these objects are established on a logical level, implemented as markers within the respective placements. For [[building the render nodes graph|BuildProcess]], a consolidated view of the session's effective contents is created ("[[Fixture]]"), then to be traversed while emitting the LowLevelModel. The GUI is expected to query the index to discover parts of the structure; the [[object reference tags|MObjectRef]] returned by these queries will be used as arguments to any mutating operation on the objects within the session.
Using a ''flat hashtable'' allows to access a Placement denoted by ID in O(1). This way we get the Placement, but nothing more. So, additionally we'd have to set up an data record holding additional information:
* the [[scope|PlacementScope]] containing this placement
* allowing to create a path "up" from this scope, which is used for resolving any queries
* (maybe/planned) relations to other placements
Alternatively, we could try to use a ''structure based index'', thereby avoiding the mentioned description record by folding any of the contained information into the surrounding data structure:
* the scope would be obvious from the index, resp. from the path used to resolve this index
* any other information, especially the relations, would be folded into the placement
* this way, the "index" could be reduced to being the session data structure itself.
//does a placement need to know it's own ID?// Obviously, there needs to be a way to find out the ID for a given placement, especially in the following situations:
* for most of the operations above, when querying additional information from index for a given placement
* to create a PlacementRef (this is a variant of the "shared ptr from this"-problem)
On second sight, this problem turns out to be more involved, because either we have to keep a second index table for the reverse lookup (memory address -> ID), or have to tie the placement by a back-link when adding it to the index/session data structure, or (alternatively) it forces us to store a copy of the ID //within// the placement itself. The last possibility seems to create the least impact; but implementing it this way effectively gears the implementation towards a hashtable based approach.
!supported operations
The placement index is utilized by editing operations and by executing the build process. Besides these core operations it allows for resolving PlacementRef objects. This latter functionality is used by all kinds of relative placements and for dealing with them while building, but it is also used to resolve [[object reference tags|MObjectRef]], which possibly may have been handed out via an external API or may have crossed layer boundaries. From these use cases we derive the following main operations to support:
* find the actual [[Placement]] object for a given ID
* find the //scope// a given placement resides in. More specifically, find the [[placement defining this scope|PlacementScope]]
* find (any/all) other placements referring to a given placement ("within this scope")
* add a new placement to a scope given as parameter
* remove a placement from index
* (planned) move a placement to a different scope within the session
!!!Handling of Subtypes
While usually type relations don't carry over to smart-poitner like types, in case of Placement I used a special definition pattern to artificially create such type relations ({{{Placement<session::Clip>}}} is subclass of {{{Placement<MObject>}}}). Now, as we're going to copy and maintain Placements within the storage backing the index, the question is: do we actually store subtypes (all having the same size btw) or do we use a vtable based mechanism to recover the type information on access?
Actually, the handling of placement types quite flexible; the actual hierarchy of Placement types can be determined in the //usage context// — it is not really stored within the placement, and there is no point in storing it within the index. Only the type of the //pointee// can be checked with the help of Placement's vtable.
Thus, things just fall into place here, without the need of any additional implementation logic. The index stores {{{Placement<MObject>}}} instances. The usage context will provide a suitable meaning for more specifically typed placements, and as long as this is in line with the type relations on the pointee(s), as checked by the {{{Placement::isCompatible<TY>()}}} call, the placement relations will just work out right by the the cast happening automatically on results retrieval.
→ see PlacementType
!implementation
Consequently, we incorporate a random hash (implemented as {{{LUID}}}) into the individual placement, this way creating an distinguishable //placement identity,// which is //not retained on copying.// The actual ID tag is complemented by a compile time type (template parameter), thus allowing to pass on additional context information through API calls. Placements themselves use a vtable (and thus RTTI), allowing to re-discover the exact type at runtime. Any further relation information is contained within the placement's [[locating pins|LocatingPin]], thus, any further description records can be avoided by storing the placements immediately //within the index.// To summarise, the implementation is comprised of
* a main table resolving hash-ID to storage location
* information about the enclosing scope for each placement, stored within the main entry
* a reverse lookup table to find all placements contained within a given scope
* an instance holding and managing facility based on pooled allocation
A generic reference mechanism for Placements, as added to the current session.
While this reference itself is not tied to the actual memory layout (meaning it's //not// a disguised pointer), the implementation relies on a [[placement index facility|PlacementIndex]] for tracking and retrieving the actual Placement implementation object. As a plus, this approach allows to create active interconnections between placements. We utilise this possibility to create a system of [[nested scopes|PlacementScope]]. The index facility allows to //reverse// the relation denoted by such a reference, inasmuch it is possible to retrieve all other placements referring to a given target placement. But for an (external) user, this link to an index implementation is kept transparent and implicit.
!implementation considerations
From the usage context it is clear that the PlacementRef needs to incorporate a simple ID as the only actual data in memory, so it can be downcasted to a POD and passed as such via LayerSeparationInterfaces. And, of course, this ID tag should be the one used by PlacementIndex for organising the Placement index entries, thus enabling the PlacementRef to be passed immediately to the underlying index for resolution. Thus, this design decision is interconnected with the implementation technique used for the index (→ PlacementIndex). From the requirement of the ID tag to be contained in a fixed sized storage, and also from the expected kinds of queries Ichthyo (5/09) choose to incorporate a {{{LUID}}} as a random hash immediately into the placement and build the ID tag on top of it.
!using placement references
Placement references can be created directly from a given placement, or just from an {{{Placement::ID}}} tag. Creation and dereferencing can fail, because the validity of the reference is checked with the index. This implies accessing the //current session// behind the scenes. Placement references have value semantics. Dereferencing searches the denoted Placement via index, yielding a direct (language) ref.
Placement references mimic the behaviour of a real placement, i.e. they proxy the placement API (actually the passive, query-related part of placement's API functions), while directly forwarding calls to the pointee (~MObejct or subclass) when using {{{operator->()}}}. They can be copied and especially allow a lot of assignments (from placement, placement-ID or even plain LUID), even including a dynamic downcast on the pointee. Bottom line is that a placement ref can pretty much be used in place of a language ref to a real placement, which is crucial for implementing MObjectRef.
MObjects are attached into the [[Session]] by adding a [[Placement]]. Because this especially includes the possibility of //grouping or container objects,// e.g. [[sequences|Sequence]] or [[tracks|Track]] or [[meta-clips|VirtualClip]], any placement may optionally define and root a scope, and every placement is at least contained in one encompassing scope — of course with the exception of the absolute top level, which can be thought off as being contained in a scope of handling rules.
Thus, while the [[sequences|Sequence]] act as generic container holding a pile of placments, actually there is a more fine grained structure based on the nesting of the tracks, which especially in Lumiera's HighLevelModel belong to the sequence (they aren't a property of the top level timeline as one might expect). Building upon these observations, we actually require each addition of a placement to specify a scope. Consequently, for each Placement at hand it is possible to determine an //containing scope,// which in turn is associated with some Placement of a top-level ~MObject for this scope. The latter is called the ''scope top''. An example would be the {{{Placement<Track>}}} acting as scope of all the clips placed onto this track. The //implementation// of this tie-to-scope is provided by the same mechanism as utilised for relative placements, i.e. an directional placement relation. Actually, this relation is implemented by the PlacementIndex within the current [[Session]].
[>img[Structure of Placment Scopes|draw/ScopeStructure1.png]]
!Kinds of scopes
There is only a limited number of situations constituting a scope
* conceptually, the very top level is a scope of general rules.
* the next level is the link of [[binding|BindingMO]] of a [[Sequence]] into either a (top-level) [[Timeline]] or as virtual media into a VirtualClip. It is implemented through a {{{Placement<Binding>}}}.
* each sequence has at least one (manadtory) top-level placement holding its root track
* tracks may contain nested sub tracks.
* clips and (track-level) effects likewise are associated with an enclosing track.
* an important special case of relative placement is when an object is [[attached|AttachedPlacementProblem]] to another leading object, like e.g. an effect modifying a clip
__note__: attaching a Sequence in multiple ways → [[causes scoping problems|BindingScopeProblem]]
!Purpose of Placement scoping
Similar to the common mechanisms of object visibility in programming languages, placement scopes guide the search for and resolution of properties of placement. Any such property //not defined locally// within the placement is queried ascending through the sequence of nested scopes. Thus, global definitions can be shadowed by local ones.
Placement is a smart-ptr. As such, usually smart-pointers are templated on the pointee type, but a type relation between different target types doesn't carry over into a type relation on the corresponding smart-pointers. Now, as a [[Placement]] or a PlacementRef often is used to designate a specific "instance" of an MObject placed into the current session, the type parametrisation plays a crucial role when it comes to processing the objects contained within the session. Because the session deliberately has not much additional structure, besides the structure created by [[scopes and aggregations|PlacementScope]] within the session's contents.
To this end, we're using a special definition pattern for Placements, so
* a placement can refer to a specific sub-Interface like Track, Clip, Effect
* a specialised placement can stand-in for the more generic type.
!generic handling
Thus, ~MObject and Placement<~MObject> relate to the "everything is an object" view of affairs. More specific placements are registered, searched and retrieved within the session through this generic interface. In a similar vein, ~PlacementRef<~MObject> and MObjectRef is used on LayerSeparationInterfaces. This works, because it is possible to re-discover the more fine-grained target type.
* ''active type rediscovery'' happens when a [[using visitors|VisitorUse]], which requires support by the pointee types (~MObject subclasses), so the visitor implementation is able to build a trampoline table to dispatch into a specifically typed context.
* ''passive type rediscovery'' is possible whenever the usage context //is already specifically typed.// Because in this case we can check the type (by RTTI) and filter out any placement not convertible to the type requested within the given context.
!downcasting and slicing
Deliberately, all Placements have the same runtime size. Handling them value-like under certain circumstances is intended and acceptable. Of course then slicing on the level of the Placement will happen. But because the Placement actually is a smart-pointer, the pointee remains unaffected, and can be used later to re-gain the fully typed context.
On the other hand, care has to be taken when ''downcasting'' a placement. When possible, this should be preceded by a {{{Placement::isCompatible<TY>()}}}-call, which checks based on the pointee's RTTI. Client code is encouraged to avoid explicit casting and rather rely on the provided facilities:
* invoking one of the templated access functions of the PlacementIndex
* using the QueryFocus to issue a specifically typed {{{ScopeQuery<TY>}}} (which yields an suitable iterator)
* create an specifically typed MObjectRef and bind it to some reference source (~Placement-ID, LUID, Placement instance within the session)
* implementing a visitor (~BuilderTool)
//This page is a scrapbook for working out the implementation of how to (re)build the [[Fixture]]//
Structurally, (re)building the Fixture rather belongs to [[Session]], but it is implemented very similar to the render engine build process: by treating all ~MObjects found in the various [[sequences|Sequence]] with a common [[visiting tool|VisitorUse]], this tool collects a simplified view with everyting made explicit, which can be pulled of as Fixture, i.e. (special kind of sequence list) afterwards.
* there is a //gathering phase// and a //solving phase//, the gathering is done by visiting.
* during the gathering phase, there ''need to be a lock'' preventing any other edit operation.
* the solving is delegated to the individual ~Placements. It is effectively a {{{const}}} operation creating a ExplicitPlacement (copy)
* thus the Fixture contains these newly created ~ExplicitPlacements, refering to ~MObjects shared with the original Placements within the sequences
!!!prerequisites
* Session and sequences exist.
* Pipes exist and are configured
!!!postconditions
* the Fixture contains one sorted timeline of ExplicitPlacement instances
* Anything in this list is actually to be rendered
* {{red{TODO: how to store and group the effects?}}}
* any meta-clips or other funny things have been resolved to normal clips with placement
* any multichannel clips has been broken down to elementary clips {{red{TODO: what is "elementary". e.g. stereo sound streams?}}}
* any globally or otherwise strangely placed effects have been attached either to a clip or to some pipe
* we have one unified list of tracks
<<tasksum start>>
<<taskadder below>>
<<task >> work out how to get the processing of effects chained to some clip right
<<task >> work out how to handle multichannel audio (and stereo video)
!gathering phase
!!preparing
<<task>>what data collections to build?
!!treating a Track
<<task>>work out how to refer to pipes and do other config
<<task>>get some uniqe identifier and get relevant properties
!!treating a {{{Placement<Clip>}}}
<<task>>check the direct enablement status
<<task>>asses the compound status, maybe process recursively
!!treating an {{{Placement<Effect>}}}
<<task>>find out the application point {{red{really?}}}
!solving phase
<<task>>trigger solving on all placements
<<task>>sort the resulting ~ExplicitPlacements
<<tasksum end>>
//This page is a scrapbook for working out the implementation of the builder//
* NodeCreatorTool is a [[visiting tool|VisitorUse]]
* the render engine to be built is contained as state within this tool object while it is passed around
!!!prerequisites
* Session and sequences exist.
* Pipes exist and are configured
* Fixture contains ExplicitPlacement for every MObject to be rendered, and nothing else
<<tasksum start>>
<<taskadder>>
!!preparing
We need a way of addressing existing [[pipes|Pipe]]. Besides, as the Pipes and Tracks are referred by the Placements we are processing, they are guaranteed to exist.
!!treating a Pipe
<<task>>get the [[processing pattern|ProcPatt]] of the pipe by accessing the underlying pipe asset.
<<task>>process this ProcPatt recursively
!!treating a processing pattern
<<task>>{{red{finally go ahead and define what a ProcPatt need to be...}}}
!!treating a {{{Placement<Clip>}}}
<<task>>get the ProcPatt of the underlying media (asset)
<<task>>process the ProcPatt recursively
<<task>>access the ClipSourcePort (which may be created on-the-fly)
<<task>>enqueue an WiringRequest for connecting the source pipeline to the source port
<<task>>process the clip's render pipe recursively (thus adding the camera etc.)
<<task>>enqueue an WiringRequest for any placement to some pipe for this clip.
* __note__: we suppose
** all wiring requests will be done after the processing of entities
** all effects placed to this clip will be processed after this clip (but before the wiring requests)
!!treating an {{{Placement<Effect>}}}
<<task>>{{red{how to assure that effecs are processed after clips/pipes??}}}
<<task>>find out the application point
<<task>>build a transforming node for the effect and insert it there
!!postprocessing
<<task>>sort and group the assembled list of [[wiring requests|WiringRequest]] by pipes
<<tasksum end>>
With //play process// we denote an ongoing effort to calculate a stream of frames for playback or rendering.
The play process is an conceptual entity linking together several activities in the [[Backend]] and the RenderEngine. Creating a play process is the central service provided by the [[player subsystem|Player]]: it maintains a registration entry for the process to keep track of associated entities, resources allocated and calls [[dispatched|FrameDispatcher]] as a consequence, and it wires and exposes a PlayController to serve as an interface and information hub.
''Note'': the player is in no way engaged in any of the actual calculation and management tasks necessary to make this [[stream of calculations|CalcStream]] happen. The play process code contained within the player subsystem is largely comprised of organisational concerns and not especially performance critical.
* the [[Backend]] is responsible for [[dispatching|Dispatcher]] the [[calculation stream|CalcStream]] and scheduling individual calculation jobs
* the RenderEngine has the ability to carry out individual frame calculations
* the OutputSlot exposed by the [[output manager|OutputManagement]] is responsible for accepting timed frame delivery
[>img[Anatomy of a Play Process|uml/fig144005.png]]
!Anatomy of a Play Process
The Controller is exposed to the client and acts as frontend handle, while the play process body groups and manages all the various parts cooperating to generate output. For each of the participating global pipes we get a feed to drive that pipeline to deliver media of a specific kind.
Right within the play process, there is a separation into two realms, relying on different programming paradigms. Obviously the play controller is a state machine, and similarily the body object (play process) has a distinct operation state. Moreover, the current collection of individual objects hooked up at any given instance is a stateful variable. To the contrary, when we enter the realm of actual processing, operations are carried out in parallel, relying on stateless descriptor objects, wired into individual calculation jobs, to be scheduled as non-blocking units of operation. For each series of consecutive frames to be calculated, there is a descriptor object, the CalcStream, which also links to a specificaly tailored dispatcher table, allowing to schedule the individual frame jobs. Whenever the controller determines a change in the playback plan (speed change, skip, scrubbing, looping, ...), a new CalcStream is created, while the existing one is just used to mark any not-yet processed job as superseded.
The [[Player]] is an independent [[Subsystem]] within Lumiera, located at Proc-Layer level. A more precise term would be "rendering and playback coordination subsystem". It provides the capability to generate media data, based on a high-level model object, and send this generated data to an OutputDesignation, creating an continuous and timing controlled output stream. Clients may utilise these functionality through the ''play service'' interface.
!provided services
* creating a PlayProcess
* managing existing play processes
* convenience short-cuts for //performing// several kinds of high-level model objects
!creating a play process
This is the core service provided by the player subsystem. The purpose is to create a collaboration between several entities, creating media output
;data producers
:a set of ModelPort elements to ''pull'' for generating output. They can either be handed in direcly, or resolved from a set of OutputDesignation elements
;data sinks
:to be able to create output, the PlayProcess needs to cooperate with [[output slots|OutputSlot]]
:physical outputs are never handled directly, rather, the playback or rendering needs an OutputManager to resolve the output designations into output slots
;controller
:when provided with these two prerequisites, the play service is able to build a PlayProcess.
:for clients, this process can be accessed and maintained through a PlayController, which acts as (copyable) handle and front-end.
;engine
:the actual processing is done by the RenderEngine, which in itself is a compound of several services within [[Backend]] and Proc-Layer
:any details of this processing remain opaque for the clients; even the player subsystem just accesses the EngineFaçade
Within Lumiera, »Player« is the name for a [[Subsystem]] responsible for organising and tracking //ongoing playback and render processes.// → [[PlayProcess]]
The player subsystem does not perform or even manage any render operations, nor does it handle the outputs directly.
Yet it addresses some central concerns:
;uniformity
:all playback and render processes are on equal footing, handled in a similar way.
;integration
:the player cares for the necessary integration with the other subsystems
:it consults the OutputManagement, retrieves the necessary information from the [[Session]] and coordinates the forwarding of [[Backend]] calls.
;time quantisation
:the player translates continuous time values into discrete frame counts.
:to perform this [[quantisation|TimeQuant]], the help of the session for building a TimeGrid for each output channel is required.
!{{red{WIP 5/2011}}} under construction
The player subsystem is currently about to be designed and built up; some time ago, __Joel Holdsworth__ and __Ichthyo__ did a design study with a PlayerDummy, which is currently hooked up with the TransportControl in the Lumiera GUI. Starting from these experiences, and the general requirements of an NLE, the [[design of the Player subsystem|DesignPlayerSubsystem]] is being worked out.
__Joelholdsworth__ and __Ichthyo__ created this player mockup in 1/2009 to find out about the implementation details regarding integration and colaboration between the layers. There is no working render engine yet, thus we use a ~DummyImageGenerator for creating faked yuv frames to display. Within the GUI, there is a ~PlaybackController hooked up with the transport controls on the timeline pane.
# first everything was contained within ~PlaybackController, which spawns a thread for periodically creating those dummy frames
# then, a ~PlayerService was factored out, now implemented within Proc-Layer (probably to be relocated into the backend for the final version). A new LayerSeparationInterface called ''~DummyPlayer'' was created and set up as a [[Subsystem]] within main().
# the next step was to support multiple playback processes going on in parallel. Now, the ~PlaybackController holds an smart-handle to the ~PlayProcess currently generating output for this viewer, and invokes the transport control functions and the pull frame call on this handle.
# then, also the tick generation (and thus the handling of the thread which pulls the frames) was factored out and pushed down into the mentioned ~PlayProcess. For this to work, the ~PlaybackController now makes a display slot available on the public GUI DisplayFacade interface, so the ~PlayProcessImpl can push up the frames for display within the GUI
[img[Overview to the dummy player operation|draw/PlayerArch1.png]]
!when playing...
As a prerequisite, a viewer has to be prepared within the GUI. A XV video display widget is wired up to a sigc++ signal slot, using the Glib::Dispatcher to forward calls from the play process thread to the GTK main event loop thread. All of this wiring actually is encapsulated as a DisplayerSlot, created and registered with the DisplayService.
When starting playback, the display slot handle created by these preparations is used to create a ~PlayProcess on the ~DummyPlayer interface. Actually
* a ~PlayProcessImpl object is created down within the player implementation
* this uses the provided slot handle to actually //allocate// the display slot via the Display facade. Here, //allocating// means registering and preparing it for output by //one single// ~PlayProcess. For the latter, this allocation yields an actually opened display handle.
* moreover, the ~PlayProcessImpl aquires an TickService instance, which is still trotteling (not calling the periodic callback)
* probably, a real player at this point would initiate a rendering process, so he can fetch the actual output frames periodically.
* on the "upper" side of the ~DummyPlayer facade, an lib::Handle object is created to track and manage this ~PlayProcesImpl instance
The mentioned handle is returned to the ~PlaybackController within the GUI, which uses this handle for all further interactions with the Player. The handle is ref counting and has value semantics, so it can be stored away, passed as parameter and so on. All such handles corresponding to one ~PlayProcess form a family; when the last goes out of scope, the ~PlayProcess terminates and deallocates any resources. Conceptually, this corresponds to pushing the "stop" button. Handles can be deliberately disconnected by calling {{{handle.close()}}} — this has the same effect as deleting a handle (when all are closed or deleted the process ends).
All the other play control operations are simply forwarded via the handle and the ~PlayProcessImpl. For example, "pause" corresponds to setting the tick frequency to 0 (thus temporarily discontinuing the tick callbacks). When allocating the display slot in the course of creating the ~PlayProcessImpl, the latter only accesses the Display facade. It can't access the display or viewer directly, because the GUI lives within an plugin; lower layers aren't allowed to call GUI implementation functions directly. Thus, within the Display facade a functor (proxy) is created to represent the output sink. This (proxy) Displayer can be used within the implementation of the perodic callback function. As usual, the implementation of the (proxy) Displayer can be inlined and doesn't create runtime overhead. Thus, each frame output call has to pass though two indirections: the function pointer in the Display facade interface, and the Glib::Dispatcher.
!rationale
There can be multiple viewer widgets, to be connected dynamically to multiple play-controllers. (the latter are associated with the timeline(s)). Any playback can require multiple playback processes to work in parallel. The playback controller(s) should not be concerned with managing the play processes, which in turn should neither care for the actual rendering, nor manage the display frequency and synchronisation issues. Moreover, the mentioned parts live in different layers and especially the GUI needs to remain separated from the core. And finally, in case of a problem within one play process, it should be able to unwind automatically, without interfering with other ongoing play processes.
Playlist is a sequence of individual Render Engine Processors able to render a segment of the timeline. So, together these Processors are able to render the whole timeline (or part of the timeline if only a part has to be rendered).
//Note, we have yet to specify how exactly the building and rendering will work together with the backend. There are several possibilities how to structure the Playlist//
Open issues, Things to be worked out, Problems still to be solved...
!!Parameter Handling
The requirements are not quite clear; obviously Parameters are the foundation for getting automation right and for providing effect editing interfaces, so it seems to me we need some sort of introspection, i.e. Parameters need to be discovered, enumerated and described at runtime. (→ see [[tag:automation|automation]])
''Automation Type'': Directly connected is the problem of handling the //type// of parameters sensible, including the value type of automation data. My first (somewhat naive) approach was to "make everything a double". But this soon leads into quite some of the same problems haunting the automation solution implemented in the current Cinelerra codebase. What makes the issue difficult is the fact we both need static diversity as well as dynamic flexibility. Usually, when combining hierarchies and templates, one has to be very careful; so I just note the problem down at the moment and will revisit it later, when I have a more clear understanding of the demands put onto the [[ProcNode]]s
!!Treatment of Time (points) and Intervals
At the moment we have no clear picture what is needed and what problems we may face in that domain.
From experience, mainly with other applications, we can draw the following conclusions
* drift and rounding errors are dangerous, because time in our context usually is understood as a fixed grid (Frames, samples...)
* fine grained time values easily get very large
* Cinelerra currently uses the approach of simply counting natural values for each media type separately. In an environment mixing several different media types freely, this seems a bit too simplistic (because it actually brings in the danger of rounding errors, just think at drop frame TC)
!!Organizing of Output Channels
How to handle the simultaneous rendering of several output streams (video, audio channels). Shall we treat the session as one entity containing different output channels, or should it rather be seen as a composite of several sub-sessions, each for only one output channel? This decision will be reflected in the overall structure of the network of render nodes: We could have a list of channel-output generating pipelines in each processor (for every segment), or we could have independently segmented lists of Processors for every output channel/type. The problem is, //it is not clear what approach to prefer at the moment// because we are just guessing.
!!Tracks, Channels, Layers
Closely related to this is the not-so-obvious problem how to understand the common global structures found in most audio and video editing applications. Mostly, they stem from imitating hardware recording and editing solutions, thus easing the transition for professionals grown up with analogue hardware based media. But as digital media are the de-facto standard nowadays, we could rethink some of this accidental complexity introduced by sticking to the hardware tool metaphor.
* is it really necessary to have fixed global tracks?
* is it really helpful to feed "source tracks" into global processing busses/channels?
Users accustomed with modern GUI applications typically expect that //everything is a object// and can be pulled around and manipulated individually. This seems natural at start, but raises the problem of providing a efficient workflow for handling larger projects and editing tasks. So, if we don't have a hard wired multitrack+bus architecture, we need some sort of templating to get the standard editing use case done efficiently.
!!Compound and Multiplicity
Simple relations can be hard wired. But, on the contrary, it would be as naive to define a Clip as having a Video track and two audio tracks, as it would be naive to overlook the problem of holding video and corresponding audio together. And, moreover, the default case has to be processed in a //straight forward// fashion, with as few tests and decisions as possible. So, basically each component participating in getting the core processing done has to mirror the structure pattern of the other parts, so that processing can be done without testing and forking. But this leaves us with the problem where to put the initial knowledge about the structural pattern used for building up the compound structures and — especially — the problem how to treat different kinds of structural patterns, how to detect the pattern to be applied and how to treat multiple instances of the same structural pattern.
One example of this problem is the [[handling of multichannel media|MultichannelMedia]]. Following the above reasoning, we end with having a [["structural processing pattern"|ProcPatt]], typically one video stream with MPEG decoder and a pair of audio streams which need either to be routed to some "left" and "right" output pipes, or have to be passed through a panning filter accordingly. Now the problem is: //create a new instance of this structure for each new media, or detect which media to subsume under a existing pattern instance.//
!!Parallelism
We need to work out guidelines for dealing with operations going on simultaneously. Certainly, this will divide the application in several different regions. As always, the primary goal is to avoid multithread problems altogether. Typically, this can be achieved by making matters explicit: externalizing state, make the processing subsystems stateless, queue and schedule tasks, use isolation layers.
* the StateProxy is a key for the individual render processes state, which is managed in separate [[StateFrame]]s in the backend. The [[processing network|ProcNode]] is stateless.
* the [[Fixture]] provides an isolation layer between the renderengine and the Session / high-level model
* all EditingOperations are not threadsafe intentionally, because they are [[scheduled|ProcLayerScheduler]]
The current Lumiera architecture separates functionality into three Layers: __GUI__, __Proc__ and __Backend__.
While the Backend is responsible for Data access and management and for carrying out the computation intensive media opteratons, the middle Layer or ~Proc-Layer contains [[assets|Asset]] and [[Session]], i.e. the user-visible data model and provides configuration and behaviour for these entities. Besides, he is responsible for [[building and configuring|Builder]] the [[render engine|RenderEngine]] based on the current Session state.
All Assets of kind asset::Proc represent //processing algorithms// in the bookkeeping view. They enable loading, browsing and maybe even parametrizing all the Effects, Plugins and Codecs available for use within the Lumiera Session.
Besides, they provide an __inward interface__ for the [[ProcNode]]s, enabling them to dispatch the actual processing call while rendering. Actually, this interface is always accessed via an ~Effect-MObject; mostly it is investigated and queried in the build process when creating the corresponding processor nodes. → see EffectHandling for details
{{red{todo: the naming scheme??}}}
[img[Asset Classess|uml/fig131077.png]]
{{red{Note 3/2010}}} it is very unlikely we'll organise the processing nodes as a class hierarchy. Rather it looks like we'll get several submodules/special capabilities configured in within the Builder
The middle Layer of our current Architecture plan, i.e. the layer managing all processing and manipulation, while the actual data handling is done in the backend and the user interaction belongs to the GUI Layer.
→ see the [[Overview]]
The Render Engine is the part of the application doing the actual video calculations. Utilising system level services and retrieving raw audio and video data through [[Lumiera's Backend|backend.html]], its operations are guided by the objects and parameters edited by the user in [[the session|Session]]. The middle layer of the Lumiera architecture, known as the Proc-Layer, spans the area between these two exteremes, providing the the (abstract) edit operations available to the user, the representation of [["editable things"|MObjects]] and the translation of those into structures and facilities allowing to [[drive the rendering|Rendering]].
!About this wiki page
|background-color:#E3F3F1;width:96ex;padding:2ex; This TiddlyWiki is the central location for design, planning and documentation of the Lumiera Proc-Layer. Some parts are used as //extended brain// — collecting ideas, considerations and conclusions — while other tiddlers contain the decisions and document the planned or implemented facilities. The intention is to move over the more mature parts into the emerging technical documentation section on the [[Lumiera website|http://www.lumiera.org]] eventually. <br/><br/>Besides cross-references, content is largely organised through [[Tags|TabTags]], most notably <br/><<tag overview>> · <<tag def>> · <<tag decision>> · <<tag Concepts>> <br/> <<tag Model>> · <<tag SessionLogic>> · <<tag GuiIntegration>> · <<tag Builder>> · <<tag Rendering>> · <<tag Player>> · <<tag Rules>> · <<tag Types>> |
!~Proc-Layer Summary
When editing, the user operates several kinds of //things,// organized as [[assets|Asset]] in the AssetManager, like media, clips, effects, codecs, configuration templates. Within the context of the [[Project or Session|Session]], we can use these as »[[Media Objects|MObjects]]« — especially, we can [[place|Placement]] them in various kinds within the session and relative to one another.
Now, from any given configuration within the session, we create sort or a frozen- and tied-down snapshot, here called »[[Fixture|Fixture]]«, containing all currently active ~MObjects, broken down to elementary parts and made explicit if necessary. This Fixture acts as a isolation layer towards the Render Engine. We will hand it over to the [[Builder]], which in turn will transform it into a network of connected [[render nodes|ProcNode]]. This network //implements// the [[Render Engine|OverviewRenderEngine]].
The system is ''open'' inasmuch every part mirrors the structure of corresponding parts in adjacent subsystems, and the transformation of any given structure from one subsystem (e.g. Asset) to another (e.g. Render Engine) is done with minimal "magic". So the whole system should be able to handle completely new structures mostly by adding new configurations and components, without much need of rewriting basic workings.
!!see also
→ [[Overview]] of Subsystems and Components, and DesignGoals
→ [[An Introduction|WalkThrough]] discussing the central points of this design
→ [[Overview Session (high level model)|SessionOverview]]
→ [[Overview Render Engine (low level model)|OverviewRenderEngine]]
→ BuildProcess and RenderProcess
→ [[Two Examples|Examples]] (Object diagrams)
→ how [[Automation]] works
→ [[Problems|ProblemsTodo]] to be solved and notable [[design decisions|DesignDecisions]]
→ [[Concepts, Abstractions and Formalities|Concepts]]
→ [[Implementation Details|ImplementationDetails]] {{red{WIP}}}
A data processing node within the Render Engine. Its key feature is the possibility to pull from it one (freely addressable) [[Frame]] of calculated data. Further, each ~ProcNode has the ability to be wired with other nodes and [[Parameter Providers|ParamProvider]]
!! {{red{open questions}}}
* how to address a node
* how to type them
* how to discover the number and type of the ports
* how to discover the possible parameter ports
* how to define and query for additional capabilities
→ see also the [[open design process draft|http://www.pipapo.org/pipawiki/Lumiera/DesignProcess/DesignRenderNodesInterface]]
→ see [[mem management|ManagementRenderNodes]]
→ see RenderProcess
This special type of [[structural Asset|StructAsset]] represents information how to build some part of the render engine's processing nodes network. Processing patterns can be thought of as a blueprint or micro program for construction. Most notably, they are used for creating nodes reading, decoding and delivering source media material to the render network, and they are used for building the output connection via faders, summation or overlay nodes to the global pipes (busses). Each [[media Asset|MediaAsset]] has associated processing patterns describing the codecs and other transformations needed to get at the media data of this asset. (and because media assets are typically compound objects, the referred ~ProcPatt will be compound too). Similarily, for each stream kind, we can retrieve a processing pattern for making output connections. Obviously, the possibilities opened by using processing patterns go far beyond.
Technically, a processing pattern is a list of building instructions, which will be //executed// by the [[Builder]] on the render node network under construction. This implies the possibility to define further instruction kinds when needed in future; at the moment the relevant sorts of instructions are
* attach the given sequence of nodes to the specified point
* recursively execute a nested ~ProcPatt
More specifically, a sequence of nodes is given by a sequence of prototypical effect and codec assets (and from each of them we can create the corresponding render node). And the point to attach these nodes is given by an identifier — in most cases just "{{{current}}}", denoting the point the builder is currently working at, when treating some placed ~MObject which in turn yielded this processing pattern in question.
Like all [[structural assets|StructAsset]], ~ProcPatt employs a special naming scheme within the asset name field, which directly mirrors its purpose and allows to bind to existing processing pattern instances when needed. {{red{TODO: that's just the general idea, but really it will rather use some sort of tags. Yet undefined as of 5/08}}} The idea is letting all assets in need of a similar processing pattern refer to one shared ~ProcPatt instance. For example, within a MPEG video media asset, at some point there will be a ~ProcPatt labeled "{{{stream(mpeg)}}}". In consequence, all MPEG video will use the same pattern of node wiring. And, of course, this pattern could be changed, either globally, or by binding a single clip to some other processing pattern (for making a punctual exception from the general rule)
!!defining Processing Patterns
The basic working set of processing patterns can be expected to be just there (hard wired or default configuration, mechanism for creating an sensible fallback). Besides, the idea is that new processing patterns can be added via rules in the session and then referred to by other rules controlling the build process. Any processing pattern is assembled by adding individual build instructions, or by including another (nested) processing pattern.
!!retrieving a suitable Processing Pattern
For a given situation, the necessary ProcPatt can be retrieved by issuing a [[configuration query|ConfigQuery]]. This query should include the needed capabilities in predicate form (technically this query is a Prolog goal), but it can leave out some pieces of information by just requesting "the default" → see DefaultsManagement
!!how does this actually work?
Any processing pattern needs the help of a passive holder tool suited for a specific [[building situation|BuilderPrimitives]]; we call these holder tools [[building moulds|BuilderMould]]. Depending on the situation, the mould has been armed up by the builder with the involved objects to be connected and extended. So, just by issuing the //location ID// defined within the individual build instruction (in most cases simply {{{"current"}}}), the processing pattern can retrieve the actual render object to use for building from the mould it is executed in.
!!errors and misconfiguration
Viewed as a micro program, the processing patterns are ''weak typed'' — thus providing the necessary flexibility within an otherwise strong typed system. Consequently, the builder assumes they are configured //the right way// — and will just bail out when this isn't the case, marking the related part of the high-level model as erroneous.
→ see BuilderErrorHandling for details
a given Render Engine configuration is a list of Processors. Each Processor in turn contains a Graph of ProcNode.s to do the acutal data processing. In order to cary out any calculations, the Processor needs to be called with a StateProxy containing the state information for this RenderProcess
The Quantiser implementation works by determining the grid interval containing a given raw time.
These grid intervals are denoted by ordinal numbers (frame numbers), with interval #0 starting at the grid's origin and negative ordinals allowed.
!frame quantisation convention
Within Lumiera, there is a fixed convention how these frame intervals are to be defined (⇒ [[time handling RfC|http://lumiera.org/documentation/devel/rfc/TimeHandling.html]])
[img[Lumiera's frame quantisation convention|draw/FramePositions1.png]]
Especially, this convention is agnostic of the actual zero-point of the scale and allows direct length calculations and seamless sequences of intervals.
The //nominal coordinate// of an interval is also the starting point -- for automation keys frames we'll utilise special provisions.
!range limitation problems
because times are represented as 64bit integers, the time points addressable within a given scale grid can be limited, compared with time points addressable through raw (internal) time values. As an extreme example, consider a time scale with origin at {{{Time::MAX}}} -- such a scale is unable to represent any of the original scale's value above zero, because the resulting coordinates would exceed the range of the 64bit integer. Did I mention that 64bit micro ticks can represent about 300000 years?
Now the actual problem is that using 64bit integers already means pushing to the limit. There is no easy escape hatch, like using a larger scale data type for intermediaries -- it //is// the largest built-in type. Basically we're touching the topic of ''safe integer arithmetics'' here, which is frequently discussed as a security concern. The situation is as follows:
* every programmer promises "I'll do the checks when necessary" -- just to forget doing so in practice then.
* for C programming, the situation is hopeless. Calling functions for simple arithmetics is outright impractical -- that won't happen volountarily
* it is possible to build a ~SafeInt datatype in C++ though. While theoretically fine, in practice this also creates a host of problems:
** actually detecting all cases and coding the checks is surprisingly hard and intricate.
** the attempt to create a smooth integration with the built-in data types drives us right into one of the most problematic areas of C++
** the performance hit is considerable (factor 2 - 4) -- again luring people into "being clever".
There is an existing [[SafeInt class by David LeBlanc|http://safeint.codeplex.com/]], provided by Microsoft through the ~CodePlex platform. It is part of the libraries shipped with ~VisualStudio and extensively used in Office 2010 and Windows, but also provided under a somewhat liberal license (you may use it but any derived work has to reproduce copyright and usage terms). Likely this situation also hindered the further [[development|http://thread.gmane.org/gmane.comp.lib.boost.devel/191010]] of a comparable library in boost (→ [[vault|https://svn.boost.org/trac/boost/wiki/LibrariesUnderConstruction#Boost.SafeInt]]).
!!!solution possibilities
;abort operation
:an incriminating calculation need to be detected somehow (see below).
:the whole usage context gets aborted by excaption in case of an alarm, similar to an out of memory...
:thus immediate corruption is avoided, but the user has to realise and avoid the general situation.
;limit values
:provide a limiter to kick in after detecting an alarm (see below).
:time values will just stick to the maximum/minimum boundary value....
:the rest of the application needs to be prepared for timing calculations to return degenerate results
;~SafeInt
:base ~TimeValue on a ~SafeInt<gavl_time_t>
:this way we could easily guarantee for detecting any //situation.//
:of course the price is the performance hit on all timing calculations.
:Moreover -- if we want to limit values instead of raising an exception, we'd need to write our own ~SaveInt.
;check some operations
:time values are mostly immutable, thus it would likely be sufficient only to check some strategically important points
:#parsing or de-serialising
:#calculations in ~TimeVar
:#quantisation
:#build Time from framecount
;limit time range
:because the available time range is so huge, it wouldn't hurt to reduce it by one or two decimals
:this way both the implementation of the checks could be simplified and the probability of an overflow reduced
;ignore the problem
:again, because of the huge time range, the problem could generally be deemed irrelevant
:we could apply a limiter to enforce a reduced time range just in the quantisation and evaluation of timecodes
A combination of the last approaches seems to be most appropriate here:
Limit the officially allowed time range and perform simple checks when quantising a time value and when applying a scale factor.
We limit {{{Time::MAX}}} by factor 1/30 and define the minimum //symmetrically to this.// This still leaves us with ''±9700 years allowed timerange''.
{{red{WIP as of 10/09}}}...//brainstorming about the first ideas towards a query subsystem//
!use case: discovering the contents of a container in the HighLevelModel
In the course of shaping the session API, __joel__ and __ichthyo__ realised that we're moving towards some sort of discovery or introspection. This gives rise to the quest for a //generic// pattern how to issue and run these discovery operations. The idea is to understand such a discovery as running a query — using this specific problem to shape the foundation of a query subsystem to come.
* a ''query'' is a polymorphic, noncopyable, non-singleton type; a query instance corresponds to one distinctly issued query
* issuing a query yields a result set, which is hidden within the concrete query implementation.
* the transactional behaviour needs still to be defined: how to deal with concurrent modifications? COW?
* the query instance remains property of the entity exposing the query capability.
* client code gets a result iterator, which can be explored //only once until exhaustion.//
* the handed out result iterator is used to manage the allocation for the query result set by sideeffect (smart handle). → Ticket #353
For decoupling the query invocation from the facility actually processing the query, we need to come up with common pattern. In 10/09, there is an immediate demand for such a solution pattern for implementing the QueryFocus and PlacementScope framework, which is crucial for contents discovery in general on the session interface. → QueryResolver was shaped to deal with this situation, but has the potential to evolve into a general solution for issuing queries.
!use case: retrieving object to fulfil a condition
The requirement is to retrieve one (or multiple) objects of a specific kind, located within a scope, and fulfilling some additional condition. For example: find a sub-track with {{{id(blubb)}}}.
This is a special case of the general discovery (described in the previous use case), but it is also a common situation in a general ConfigQuery (⇒ an object of a specific type and with additional capabilities...). The tricky question seems to be how to specify and resolve these additional conditions or capabilities.
* in a generic query handled by a resolution engine, we might represent these capabilities as a nested //goal.//
* if we approach the problem as a filter pipeline, then the condition becomes a functor (or closure). → QueryResolver
On second consideration, these two approaches don't contradict each other, because they live in different contexts and levels of abstraction. Performance-wise, both are bad and degenerate on large models, because both effectively cause a full table scan. Only specialised search functions for hardcoded individual properties could improve that situation, by backing them with an additional sub-index.
__Conclusion__: no objection against providing the functor/filter solution right now, even on the QueryFocus API —
notwithstanding the fact we need a better solution later.
----
See also the notes on
→ QueryImplProlog
→ QueryRegistration
When querying contents of the session or sub-containers within the session, the QueryFocus follows the current point-of-query. As such queries can be issued to explore the content of container-like objects holding other MObjects, the focus is always attached to a container, which also acts as [[scope|PlacementScope]] for the contained objects. QueryFocus is an implicit state (the current point of interrest). This sate especially remembers the path down from the root of the HighLevelModel, which was used to access the current scope. Because this path constitutes a hierarchy of scopes, it can be relevant for querying and resolving placement properties. (→ SessionStructureQuery)
!provided operations
* attach to a given scope-like object. Causes the current focus to //navigate//
* open a new focus, thereby pushing the existing focus onto a [[focus stack|QueryFocusStack]]
* return (pop) to the previous focus
* get the current scope, represented by the "top" Placement of this scope
* get the current ScopePath from root (session globals) down to the current scope
* (typed) content discovery query on the current scope
[>img[Scope Locating|uml/fig136325.png]]
→ [[more|SessionContentsQuery]] regarding generic scope queries
!!!relation to Scope
There is a tight integration with PlacementScope through the ScopeLocator, which establishes the //current focus.// But while the [[scope|PlacementScope]] just decorates the placement defining a scope (called //»scope top«//), QueryFocus is more of a //binding// — it links or focusses the current state into a specific scope with a ScopePath in turn depending on this current state. Thus, while Scope is just a passive container allowing to locate and navigate, QueryFocus by virtue of this binding allows to [[Query]] at this current location.
!implementation notes
we provide a static access API, meaning that there is a singleton (the ScopeLocator) behind the scenes, which holds the mentioned scope stack. The current focus stack top, i.e. the current ScopePath is managed through an ref-counting handle embedded into each QueryFocus instance. Thus, effectively QueryFocus is an frontend object for accessing this state. Moreover, embedded into ScopeLocator, there is an link to the current session. But this link is kept opaque; it works by the current session exposing an [[query service|QueryResolver]], while QueryFocus doesn't rely on knowledge about the session, allowing the focus to be unit tested.
The stack of scopes must not be confused with the ScopePath. Each single frame on the stack can be seen and accessed as a QueryFocus and as such relates to a current ScopePath. The purpose of the stack is to make the scope handling mostly transparent; especially this stack allows to write dedicated query functions directed at a given object: they work by pushing and then navigating to the object to use as starting point for the query, i.e. the //current scope.//
!!!simplifications
The full implementation of this scope navigation is tricky, especially when it comes to determining the relation of two positions. It should be ''postponed'' and replaced by a ''dummy'' (no-op) implementation for the first integration round.
The ScopeLocator uses a special stack of ScopePath »frames« to maintain the //current focus.//
What is the ''current'' QueryFocus and why is it necessary? There is a state-dependent part involved, inasmuch the effective ScopePath depends on how the invoking client has navigated the //current location// down into the HighLevelModel structures. Especially, when a VirtualClip is involved, there can be discrepancies between the paths resulting when descending down through different paths. (See → BindingScopeProblem).
Thus, doing something with the current location, and especially descending or querying adjacent scopes can modify this current path state. Thus we need a means of invoking a query in a way not interfering with the current path state, otherwise we wouldn't be able to provide side-effect free query operations accessible on individual objects within the model.
!maintaining the current QueryFocus
As long as client code is just interested to use the current query location, we can provide a handle referring to it. But when a query needs to be run without side effect on the current location, we //push// it aside and start using a new QueryFocus on top, which starts out at a new initial location. Client code again gets a handle (smart-ptr) to this location, and additionally may access the new //current location.// When all references are out of scope and gone, we'll drop back to the focus put aside previously.
!implementation of ref-counting and clean-up
Actually, client code should use QueryFocus instances as frontend to access this »current focus«. Each ~QueryFocus instance incorporates a smart-ptr. But as in this case we're not managing objects allocated somewhere, we use an {{{boost::intrusive_ptr}}} and maintain the ref-count immediately within the target objects to be managed. These target objects are ScopePath instances and are living within the QueryFocusStack, which in turn is managed by the ScopeLocator singleton (see the UML diagram →[[here|QueryFocus]]). We use an (hand-written) stack implementation to ensure the memory locations of these ScopePath »frames« remain valid (and also to help with providing strong exception guarantees). The stack is aware of these ref-count and takes it into account on performing the {{{pop_unused()}}} operation: any unused frame on top will be evicted, stopping at the first frame still in use (which may be even just the old top). This cleanup also happens automatically when accessing the current top, re-initialising an potentially empty stack with a default-constructed new frame if necessary. This way, just accessing the stack top always yields the ''current focus location'', which thereby is //defined as the most recently used focus location still referred.//
!concurrency
This concept deliberately ignores parallelism. But, as the current path state is already encapsulated (and ref-counting is in place), the only central access point is to reach the current scope. Instead of using a plain-flat singleton here, this access can easily be routed through thread local storage.
{{red{As of 10/09 it is not clear if there will be any concurrent access to this discovery API}}} — but it seems not unlikely to happen...
//obviously, getting this one to work requires quite a lot of technical details to be planned and implemented.// This said...
The intention is to get much more readable ("declarative") and changeable configuration as by programming the decision logic literately within the implementation of some object.
!Draft
As an example, specifying how a Track can be configured for connecting automatically to some "mpeg" bus (=pipe)
{{{
resolve(O, Cap) :- find(O), capabilities(Cap).
resolve(O, Cap) :- make(O), capabilities(Cap).
capabilities(Q) :- call(Q).
stream(T, mpeg) :- type(T, track), type(P, pipe), resolve(P, stream(P,mpeg)), placed_to(P, T).
}}}
Then, running the goal {{{:-resolve(T, stream(T,mpeg)).}}} would search a Track object, try to retrieve a pipe object with stream-type=mpeg and associate the track with this pipe. This relies on a predicate "stream(P,mpeg)" implemented (natively) for the pipe object. So, "Cap" is the query issued from calling code — here {{{stream(T,mpeg)}}}, the type guard {{{type(T, track)}}} will probably be handled or inserted automatically, while the predicate implementations for find/1, make/1, stream/2, and placed_to/2 are to be provided by the target types.
* __The supporting system__ had to combine several code snippets into one rule system to be used for running queries, with some global base rules, rules injected by each individual participating object kind and finally user provided rules added by the current session. The actual query is bound to "Cap" (and consequently run as a goal by {{{call(Q)}}}). The implementation needs to provide a symbol table associating variable terms (like "T" or "P") to C/C++ object types, enabling the participating object kinds to register their specific predicate implementations. This is crucial, because there can be no general scheme of object-provided predicates (for each object kind different predicates make sense, e.g. [[pipes|PipeHandling]] have other possibilities than [[wiring requests|WiringRequest]]). Basically, a query issues a Prolog goal, which in turn evaluates domain specific predicates provided by the participating objects and thus calls back into C/C++ code. The supporting system maintains the internal connection (via the "type" predicate) such that from Prolog viewpoint it looks as if we were binding Variables directly to object instances. (there are some nasty technical details because of the backtracking nature of Prolog evaluations which need to be hidden away)
* Any __participating object kind__ needs a way to declare domain specific predicates, thus triggering the registration of the necessary hooks within the supporting system. Moreover, it should be able to inject further prolog code (as shown in the example above with the {{{strem(T, mpeg)}}} predicate. For each of these new domain specific predicates, there needs to be a functor which can be invoked when the C implementation of the predicate is called from Prolog (in some cases even later, when the final solution is "executed", e.g. a new instance has been created and now some properties need to be set).
!!a note on Plugins
In the design of the Lumiera Proc Layer done thus far, we provide //no possibility to introduce a new object kind// into the system via plugin interface. The system uses a fixed collection of classes intended to cover all needs (Clip, Effect, Track, Pipe, Label, Automation, ~Macro-Clips). Thus, plugins will only be able to provide new parametrisations of existing classes. This should not be any real limitation, because the whole system is designed to achieve most of its functionality by freely combining rather basic object kinds. As a plus, it plays nicely with any plain-C based plugin interface. For example, we will have C++ adapter classes for the most common sorts of effect plugin (pull system and synchronous frame-by-frame push with buffering) with a thin C adaptation layer for the specific external plugin systems used. Everything beyond this point can be considered "configuration data" (including the actual plugin implementation to be loaded)
//Querying for some suitable element,// instead of relying on hard wired dependencies, is considered a core pattern within Lumiera.
But we certainly can't expect to subsume every query situation to one single umbrella interface, so we need some kind of '''query dispatch''' and a registration mechanism to support this indirection. Closely related is the → TypedQueryProblem
!Plans and preliminary implementation
As of 6/10, the intention is to be able just to //pose queries eventually.// Behind the scenes, a suitable QueryResolver should then be picked to process the query and yield a resultset. Thus the {{{Goal}}} and {{{Query<TY>}}} interfaces are to become the access point to a generic dispatching service and a bundle of specialised resolution mechanisms.
But implementing this gets a bit involved, for several reasons
* we don't know the kinds of queries, their frequency and performance requirements
* we don't know the exact usage pattern with respect to memory management of the resultsets.
* we can't asses the relevance of //lock contention,// created by using a central dispatcher facility.
We might end up with a full blown subsystem, and possibly with a hierarchy of dispatchers.
But for now the decision is to proceed with isolated and specialised QueryResolver subclasses, and to pass a basically suitable resolver to the query explicitly when it comes to retrieving results. This resolver should be obtained by some system service suitable for the concrete usage situation, like e.g. the ~SessionServiceExploreScope, which exposes a resolver to query session contents through the PlacementIndex. Nonetheless, the actual dispatch mechanism is already implemented (by using an ~MultiFact instance), and each concrete resolution mechanism is required to do an registration within the ctor, by calling the inherited {{{QueryResolver::installResolutionCase(..)}}}.
Within the Lumiera Proc-Layer, there is a general preference for issuing [[queries|Query]] over hard wired configuration (or even mere table based configuration). This leads to the demand of exposing a //possibility to issue queries// — without actually disclosing much details of the facility implementing this service. For example, for shaping the general session interface (in 10/09), we need a means of exposing a hook to discover HighLevelModel contents, without disclosing how the model is actually organised internally (namely by using an PlacementIndex).
!Analysis of the problem
The situation can be decomposed as follows.[>img[QueryResolver|uml/fig137733.png]]
* first off, we need a way to state //what kind of query we want to run.// This includes stipulations on the type of the expected result set contents
* as the requirement is to keep the facility actually implementing the query service hidden behind an interface, we're forced to erase specific type information and pass on an encapsulated version of the query
* providing an iterator for exploring the results poses the additional constraint of having an fairly generic iterator type, while still being able to communicate with the actual query implementation behind the interface.
!!!Difficulties
*the usage pattern is not clear — mostly it's just //planned//
*# client might create a specific {{{Query<TY>}}} and demand resolution
*# client might create just a goal, which is then translated into a specific query mechanism behind the invocation interface
*# client issues a query and expect it just to be handled by //some// suitable resolver
* thus it's difficult to determine, //what// part of the issued query needs automatic management. More specifically, is it possible for the client to dispose the query after issuing it, but keeping and exploring the iterator obtained as result of the query?
* and then there is the notorious problem of re-gaining the specifically typed context //behind// the invocation interface. Especially, the facility processing the query needs to know both the expected result type and details about the concrete query and its parametrisation. <br/>→ TypedQueryProblem
!!!Entities and Operations
The //client// (code using query-resolver.hpp) either wants a ''goal'' or ''query'' to be resolved; the former is just implicitly typed and usually given in predicate logic from ({{red{planned as of 11/09}}}), while the latter may be a specialised subclass templated to yield objects of a specific type as results. A ''query resolver'' is an (abstracted) entity capable of //resolving// such a goal. Actually, behind the scenes there is s