| State | Idea |
| Date | Fri Jul 23 19:34:29 2010 |
| Proposed by | Christian Thaeter <ct@pipapo.org> |
From the beginning on we planned some kind of profiling to adapt dynamically to workload and machine capabilities. I describe here how statistic data can be gathered in a generic way. This will later work together with other components tuning the system automatically.
I just introduce some ideas about the planned profiling framework here, nothing is defined/matured yet this is certainly subject for futher discussion and refinement.
Profiling should be sufficiently abstracted to have a single set of datastructures and algorithms to work on a broad range of subjects being profiled. Moreover the profiling core just offers unitless counters, semantic will be added on top of that on a higher level.
Profiling itself must not cost much, it must not block and should avoid expensive operations. Simple integer arithmetic without divisions is suggested.
We may sample data on in stochastic way to reduce the overhead, nevertheless data which gets sampled must be accurately stored and processed without rounding losses and drifts.
It’s quite common that some values can be far off either in maximum or in minimum direction, the system should adapt to this and recover from such false alarms. Workload also changes over time we need to find some way to measure the current/recent workload an grand total over the whole application runtime is rather uninteresting. While it is also important that we adapt slow enough not to get into some osccilating cycle.
Profiling can be only passive collecting data and let it be analyzed by some other component or active triggering some action when some limits are reached. I am yet a bit undecided and keep it open for both.
typedef int64_t profile_value; struct profile { ProfileVTable vtable; /* Using trylock for sampling makes it never contend on the lock but some samples are lost. Should be ok. */ mutex_t lock; /* with trylock? */ /* statistics / running averages */ /* n being a small number 2-5 or so */ profile_value max[n]; /* n maximum values seen so far, decreased by recovery */ profile_value min[n]; /* n minimum values seen so far, increased by recovery */ /* store sum & count, but average calculation implies a division and will be only done on demand */ profile_value count; /* count profile calls */ profile_value sum; /* sum up all calls, average = sum/count */ /* current is the sampled value to be integrated */ /* trend is caclulated before theb new run_average */ profile_value trend; /* trend = (trend + (run_average-current))>>1 */ /* we may need some slower diverging formula for running average */ profile_value run_average; /* run_average = (run_average + current)>>1) */ /* active limits, define whats good and whats bad, calls back to vtable when limit is hit */ profile_value max_limit; profile_value min_limit; /* do we want limits for trends too? */ /* we count how often we hit limits, a hit/miss ratio will give a good value for optimization */ profile_value hit_cnt; profile_value high_miss_cnt; profile_value low_miss_cnt; /* recovery state */ int rec_init; int rec_current; int rec_percent; void* extra; };
viable:
conclusions about its process: