Dashboard reports incorrect completion state

[gidden]

See http://198.101.154.53:8080/dashboard for Issue #666, whose batlab result is:
http://submit-1.batlab.org/nmi/results/details?runID=215388

[gidden]

I'll ping this issue again with a current example. See http://cycamore-ci.fuelcycle.org/dashboard issue 175. It is reported as successful, but if you go to the batlab run, you'll notice it failed.

[scopatz]

This is because either _NMI_STEP_FAILED is the wrong thing to test or has the incorrect value. You can see it's usage here. @zwelchWI, how do we get the return status of the remote nodes? We want them all to be true for the post_all to report success.

[zwelchWI]

It looks like the problem is that |_NMI_STEP_FAILED| is being evaluated
on the polyphemus server. I went and looked at the post all script for
issue 175 on the submit server and it looks like this :

#!/bin/bash

polyphemus post_all callbacks

if [ -z ]
then
curl --form
status='{"status":"success","number":175,"description":"build and test
completed successfully"}'
http://cycamore-ci.fuelcycle.org:80/batlabstatus
else
curl --form
status='{"status":"failure","number":175,"description":"build and test
failed"}' http://cycamore-ci.fuelcycle.org:80/batlabstatus
fi

which will always be true!

On Mon Jan 20 10:59:42 2014, Anthony Scopatz wrote:

This is because either |_NMI_STEP_FAILED| is the wrong thing to test
or has the incorrect value. You can see it's usage here
https://github.com/polyphemus-ci/polyphemus/blob/master/polyphemus/batlabrun.py#L37.
@zwelchWI https://github.com/zwelchWI, how do we get the return
status of the remote nodes? We want them all to be true for the
post_all to report success.

—
Reply to this email directly or view it on GitHub
#5 (comment).

Zach Welch
Graduate Student
Dept of Computer Sciences
University of Wisconsin - Madison

[gidden]

So I'm looking into this again, and it might actually be on batlab's end... take a look at the Result field of this run: http://submit-1.batlab.org/nmi/results/details?runID=226876

[gidden]

The overall batlab result also claimed success in the original run I linked to at the start of the issue

[gidden]

Ok, I've checked the postall script for pr #25, and it's correct. this is batlab's problem.

[gidden]

I assume @zwelchWI can handle it on batlab's end, if that's ok?

[gidden]

So, one last comment (sorry). Are we sure the post_all script is doing what we think it's doing? If the step it's checking failure against is the post_all step, then it will definitely succeed.

[zwelchWI]

I'm reworking how we detect failure in post_all, because $_NMI_STEP_FAILED is clearly not working like we want it to.

[scopatz]

Thank @zwelchWI.

[zwelchWI]

GOOD NEWS: I have fixed our batlab stuff so it correctly sends FAIL
messages to polyphemus.

BAD NEWS: Polyphemus crashes when it gets a FAIL.

Continuing to work on this

On Thu Feb 6 17:27:34 2014, Anthony Scopatz wrote:

Thank @zwelchWI https://github.com/zwelchWI.

—
Reply to this email directly or view it on GitHub
#5 (comment).

Zach Welch
Graduate Student
Dept of Computer Sciences
University of Wisconsin - Madison

[scopatz]

I'd like to think that this happens because Polyphemus gets so distraught over a failed batlab run that he just can't take it anymore.