Why does the ElectricAccelerator Cluster Manager report a longer build duration than expected for timed out builds?
With ElectricAccelerator 7.2.2 on our RHE6.5 systems, we started a build at 10:20 AM. At 11:30, this build timed out, and we could see this in the log with EC3102.
Later we noticed the cluster manager reports a build duration of 1h 53m 8s. This doesn't seem to make sense since 1h10m of real time had elapsed at the time the build was marked timed out.
We're looking into the timeout issue, but we wish to know - why does the cluster manager report a build duration longer than what seems to be possible?
When did you observe the build duration on the CM? Was that immediately after the timeout, or later?
Best Answer
We found the problem. All machines are configured to use NTP, and this was an unconfirmed assumption. We just checked, and the time is off by an hour. The team is adjusting the CM time.
Yes. This is another example of "trust but verify."
It became clear we were off when we saw a build time of 0 minutes. Somehow, the NTP service was not set up and/or wonky on all of these three newly-provisioned machines. The incorrect assumption is they were just fine.
Check the clocks on the cluster manager, build machine and agents. Perhaps they are not synchronized? I would suggest making sure all the boxes and VM's are using the same NTP service to set their clocks.
We checked - all the machines here use a time service. They have other systems that rely on synchronized clocks, and we think we're covered on *that* front.
Ok, then scratch that one and moving right along to Laurent's suggestion. :)
Do you have conflicts in your build? I've seen cases, mostly in the ElectricSimulator report where timing is incorrect due to those.
Yes, we have conflicts. This was a brand-new, clean, learning build.
OK so solve your conflicts and come back to us with the results
It sounds like the "timed out" emake process eventually woke up and contacted the cluster manager again, possibly even sending the end-of-build handshake message. This would cause the cluster manager to reassess its idea of how long the build was running. You could simulate this by invoking emake, suspending the process with CTRL-Z for long enough that the cluster manager regards it as timed out, then resuming the emake process.
