How do I interpret the ElectricAccelerator cluster manager server load report?
One of the lines in our server load report is:
Cluster,2014-04-16 17:29:27 PDT,272,263,166, 63,34,24 m 38 s,16
We have 272 total agents, a demand for 263 agents, 166 agents in use, a cluster shortage of 63, and a license shortage of 34.
Since we have 166 agent licenses (we temporarily have way more agent node machines than we actually need), I would expect that we'd have a cluster shortage of 97 and a license shortage of 97, as cluster shortage is defined as "The difference between the maximum number of agents requested by all builds and the number of agents that were assigned" and license shortage is defined as "The difference between the maximum request for agents by all builds and the number of agents the license allows". I do not know how to reconcile this with the 63 and 34 numbers in the report.
What conclusions should we draw from these metrics?
Best Answer
Starting with Accelerator 11.0 the cluster manager metrics have been redesigned to more accurately describe peak and average load. If you are using a cluster manager older than 11.0 it is recommended that you upgrade in order to enjoy the benefits of these new metrics.
Generally I prefer to use the sealevel report when trying to understand cluster load. I think it's easier to understand and gives a better view into cluster usage than the server load report.
As for the server load report, I think that the agents-in-use, cluster shortage, and license shortage metrics are flawed and probably will not give you actionable information. Fundamentally they are trying to distill a very complex algorithm -- agent allocation -- into a few simple numbers. I don't think this is really possible. Those numbers conceal all of the many factors that affect agent allocation, like platform, resource constraints, agent allocation policy, build priorities, min/max agent settings, and more. As if that were not enough, the values shown are actually rolling averages over a 5-minute interval, which means there is a dampening effect on peaks/troughs in the actual data. Finally, the number of samples per window is not fixed, because the metrics are updated whenever any agent allocation event occurs, which in turn is dependent on the specific set of builds active at a given moment.
In addition to all of that, you have to understand that the license limitation is the _very last restriction_ applied to agent allocation. That means that if the allocation is not fully satisfied for _any other reason_, like priorities or allocation policy, then it's not counted as a license shortage by this metric. For example, suppose you have a cluster with 30 agents, a license for 20, and a resource "foo" that includes 10 agents. A build runs specifying resource "foo" and requests 30 agents. It is allocated only 10 agents, because there are only 10 matching the resource constraint. The agent demand for the cluster as a whole would be 30, agents-in-use would be 10, cluster shortage would be _zero_ and license shortage would be _zero_. Crazy, right?
If you're still not convinced, I think there's a defect in the way that the cluster shortage metric is computed in all versions of Accelerator prior to and including 7.2.x. Its value probably should be the sum of the value currently reported and the license shortage currently reported (coincidentally, that gives the 97 you were expecting to see, although I think more rightfully the cluster shortage for your example should be zero, since you have strictly more agents than the agent demand, at least according to the averages, but then this is the kind of nuttiness you get from trying to make sense out of these numbers).
The only way these metrics have a hope of working out sensibly is if you are not using resources, your cluster is all of one platform, and you are using the _shared_ allocation policy. Then, maybe, the numbers will make some sense. But really, you're better off using the sealevel report.
Hmm. It would seem to be that since the language of the definitions says "maximum number of agents requested" the calculation is based on demand, not total agents. It's telling you that though it needs 97 more agents assigned to meet demand, but there's only enough licenses to add 63 more. To fully meet demand you'd need 34 more licenses.
