We've upgraded our test environment from ElectricCommander 4.2 to 7.3 and are currently running test before we upgrade our Pro environment.
One user is having trouble viewing logs from jobs running on their agents. Clicking on the step logs we see:
The agent returned error 404. Error contacting [their resource]: org.apache.http.ConnectionClosedException: Connection closed unexpectedly
The logs are there in their workspace and can be viewed manually, but not from the Web UI.
Details about our setup:
For testing, I exported/imported their existing project, resources, and pools from our current Pro instance (4.2) to our test instance (7.3). Everything is working on the 4.2 instance.
I've already confirmed the server names are correct in the server and apache config files. I can also view remote logs on another agent in one of my test projects, so it looks like its specific with this user's resources.
Their jagent.log shows the request to get the log file, but it's showing a "file not found" error when trying to fetch the file.
Something to note about their workspace: They are on a windows vm with a fileserver mounted on a separate drive for their workspace. We had to change firewall rules to connect to their lab, and file-sharing rules are hard to get approved, so this was the route we took. From ElectricCommander/ElectricFlow's perspective, the workspace is local to the agent. And this works fine on our 4.2 version.
Answer by huafeng · May 27, 2019 at 08:45 AM
When investigating an issue like this one, it's very helpful if we understand how a step log file is found and fetched. [Local Workspaces](http://docs.electric-cloud.com/eflow_doc/9_0/User/HTML/FlowUserGuide_9_0.htm#eflow_user/workspace.htm#Local_workspaces) explains this in details.
The next step is checking the logs from the relevant nodes (the Flow server, the agent that run the step and the agent on the web server).
From the server log, we can see that when a step is created (requested to run on an agent), everything seems correct, please notice "winDrive" in the following request:
2019-05-13T18:52:11.216 | DEBUG | Agent-002 | | | | AgentRequestFactoryImpl | onSubmit: request[id=run-265b6ad0-75b0-11e9-83af-0acc96f90ee4-26e052bd-75b0-11e9-bd66-0acc96f90ee4,agent[id=00000000-0000-0000-0000-000000001925,dfe-vmbuild-3.idp.boi.rd.hpicorp.net:0,secure,state=alive,version=4.2.4.71224,protocol=6,os=Microsoft Windows [Version 10.0.14393],platform=windows,secure,serial=1790,signature=jrW1c14u3vY9chyvkac2SQ==,zone[name=default,id=00000000-0000-0000-0000-000000000001,TRANSIENT]]] 0 y:\DFE_Build_Library-8.0.0_MASTER_Build-TEST-17303983 DFE_workspace D:/workspace/dfe-component-build-tool_63613 build.bat BUILD-PHASE exec-build-tool- BUI.26e052bd-75b0-11e9-bd66-0acc96f90ee4.log
Also, in the agent log, we can confirm that the data is correct in the request (a different one but uses the same workspace) that the agent receives to run a step/command:
2019-05-23T10:38:23.897 | DEBUG | AgentHttp-007 | | | n-0e101315-7d79-11e9-8d6f-0645b93d2238-0e1da77a-7d79-11e9-bc46-0645b93d2238 | AgentProtocolHandler | Received agent request: 0 Y:\DFE_Build-dfe-component-build_current-core-8.0-18453944 DFE_workspace ectool setProperty /myCall/resourceToUse dfe-vmbuild-4 ectool setProperty /myCall/BUILD "true" init-electric-cloud.0e1da77a-7d79-11e9-bc46-0645b93d2238.log init-electric-cloud.0e1da77a-7d79-11e9-bc46-0645b93d2238-postp.log
However, when the server is asked to resolve a step log file, the data from the server seems wrong. Please notice "windowsJobDirectory" in the following response, "" is unexpected:
2019-05-23T16:38:33.636 | DEBUG | Api-004 | 70446 | 0e101315-7d79-11e9-8d6f-0645b93d2238 | resolveFile | XmlApiRequestHandler | Response for id=14086: artifacts https://dfe-vmbuild-4.idp.boi.rd.hpicorp.net:7800/ init-svn-change-chec.0e1e1cac-7d79-11e9-bc46-0645b93d2238.log Y:DFE_Build-dfe-component-build_current-core-8.0-18453944
When the agent receives such a request to fetch a file like the following, it fails to find the file as the value of "windowsJobDirectory" is wrong:
2019-05-23T10:38:33.808 | DEBUG | AgentHttp-008 | | | fetchFile-268 | AgentProtocolHandler | Received agent request: Y:?DFE_Build-dfe-component-build_current-core-8.0-18453944 artifacts init-svn-change-chec.0e1e1cac-7d79-11e9-bc46-0645b93d2238.log init-svn-change-chec.0e1e1cac-7d79-11e9-bc46-0645b93d2238.log 2019-05-23T10:38:33.808 | DEBUG | AgentHttp-008 | | | fetchFile-268 | AgentProtocolHandler | final path of file to retrieve: Y:?DFE_Build-dfe-component-build_current-core-8.0-18453944\init-svn-change-chec.0e1e1cac-7d79-11e9-bc46-0645b93d2238.log 2019-05-23T10:38:33.808 | DEBUG | AgentHttp-008 | | | fetchFile-268 | FileTransferHelper | Preparing to transfer Y:\\?DFE_Build-dfe-component-build_current-core-8.0-18453944\init-svn-change-chec.0e1e1cac-7d79-11e9-bc46-0645b93d2238.log 2019-05-23T10:38:33.808 | DEBUG | AgentHttp-008 | | | fetchFile-268 | ResponseHelper | Sending error response: File Y:?DFE_Build-dfe-component-build_current-core-8.0-18453944\init-svn-change-chec.0e1e1cac-7d79-11e9-bc46-0645b93d2238.log not found.
So, in this case, we can confirm that the culprit is in the server side. Deleting the workspace and recreating it or editing it to use some path like "c:/path/in/c/drive" may help.
Viewing PROPERTY_REFERENCE_ERROR logs 1 Answer
How do I temporary disable a procedure step ? 1 Answer
How to display a workspace dir listing in the web UI 2 Answers
How to get unix workspace path 3 Answers
How does ElectricCommander handle a crash of the command shell on the agent when running a job step? 1 Answer
Electric Cloud powers Continuous Delivery, helping organizations developing deliver better software faster by automating and accelerating build, test, and deployment processes at scale. Industry leaders like Qualcomm, SpaceX, Cisco, GE, Gap, and E*TRADE use Electric Cloud solutions to boost DevOps productivity and Agile throughput.