Is the Cluster Manager queue limited to 10 builds at a time?
I am seeing some interesting behavior from EA and wanted to understand if this is normal.
Because the Windows box we are using for agents is not as powerful as the Linux agent box, the build script that launches emake repeatedly (I have yet to write a single makefile for the entire build, there's too much hoop-jumping in the build script right now) for each component of the build runs faster than the agents. What is interesting is that the script is running a consistent 10 builds ahead of the agent box, and if an 11th build is going to be added to the queue, the call to emake does not return until the build at the head of the queue finishes, and a space opens up at the end.
Is this normal? Is the behavior configurable?
What do you mean by "the call to emake does not return until the build at the head of the queue finishes"? I would not expect "emake" to exit until the build has finished. There is no limit to the number of builds that may be queued.
In the builds that we are doing, if there are 10 builds queued (the agents are working on build 617, for example, and emake is trying to create a build 627), the call to emake will not exit until build 617 finishes and the agents start working on build 618. We are able to consistently and repeatedly duplicate this behavior in sessions that create 40+ consecutive builds. The machine that is running the script that runs emake will not move to build N of the script until the N-10 build has finished.
Why would you expect the call to emake for build 627 to exit before build 627 has finished?
Because the call to emake for build 617 exited and emake went on to start the builds for 618 through 626 and exit while the agents were still building 617? When emake says it is starting build 627, then it pauses until the agents are done with 617, then continues to the next, etc.
OK, it sounds more like you have 10 _active_ builds (making progress), and 1 build "queued" (waiting for agents), rather than 10 queued builds. There is no limit to the number of builds which may be queued (waiting), but there may be limits on how many concurrent builds your cluster can support, determined by the size of the cluster and the agent allocation policies.
Best Answer
You're using "queued" in a strange way here, I think: usually when we talked about _queued_ builds we mean builds that are not making any progress because they are waiting for agents, but you seem to be using it to mean builds which are active, making progress. There is no limit on the number of queued (waiting) builds with ElectricAccelerator, but the number of active builds is effectively limited by the size of your cluster and your agent allocation policy configuration.
In this case, you probably don't have enough agents to service the number of concurrent builds you're running, for the agent allocation policies you've configured. In particular, emake must have at least one agent available in order to make any progress, and it may require _more_ than one if your _minimum agents_ setting is greater than one. The default minimum agents value is _two_. You can change the value on the "Agent Policies" subtab of the "Agents" tab in the cluster manager.
If you haven't got enough agents, your options are either to add more agents, or wait until agents become available to service additional builds (what's happening to you now).
"In particular, emake must have at least one agent available in order to make any progress, and it may require more than one if your minimum agents setting is greater than one."
Ok, but this doesn't match the behavior I am seeing. For example, the first build will use all the agents and emake will exit. The second build will exit out of emake and be queued even though there are no agents available. That will repeat up to nine more times, and on the 10th then it stop and waits for the oldest build to finish and release the agents.
What would you need to see in order to see the behavior I am talking about? Which logs and/or annotation files?
Your description doesn't make sense. Are you saying that the second build terminates without doing any work? Are you running these builds as background processes or something? When you invoke "emake" it will not exit until it has performed all the work that is part of the build that you have told it to run. If you're seeing different behavior then there must be more to the story than you've said. Can you post (part of) the script that you are using to invoke emake as an edit to the original question?
I don't have permission to post it, but essentially it is a PERL script that sets up the environment and repeatedly calls emake over and over again through a PERL system call in a build that has dozens of modules. It is walking a directory structure based on a list of modules in a library that it needs to build.
In contrast, there is another PERL script that calls emake to process a Makefile which then internally cd's to the subfolder for each module and calls emake again to process the Makefile in each subfolder. The behavior in that case is exactly as you describe.
I suspect that the difference here is the PERL system call that launches emake over and over again as opposed to the main Makefile launching emake to process the Makefile's in the subfolders.
Juan, you can surely post some Perl code that shows what this script is doing. It need not be a cut-and-paste from the actual script, just a simple script that has a similar implementation so we can understand the behavior.
