Push result status on Hiptest


#1

Hi !

When I execute a BeHat Test from hiptest-publisher generated features and I push results on Hiptest (junit format),I have only 2 status Success / Failed.

  • Skipped status are marked as Success,
  • Other status like Failed on PendingException are marked as Failed vs Pending status.

It’s an issue ? or I make a Mistake ?

Thanks a lot for your Feedback.

Eric.


#2

Hi Eric,

Normally the skipped tests should be recognised and marked as skipped in Hiptest too.
For the other statuses (like pending), we do not yet support them indeed (only pass/fail/skip is currently handled).

Would it be possible for you to send us one report containing a skipped test so I can have a look and fix this issue ?

Best regards,
Vincent


#3

Thanks for your feedback.

I see you upgrade hiptest-publisher to 1.9 >> Same results.

I have not a “real” generated report with skipped status but I make one from a failed and the result is same

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="elec">
  <testsuite name="Accessibilité" tests="0" skipped="2" failures="0" errors="0">
    <testcase name="Accès direct à une page (uid:5e9e0054-a892-4a1e-a53d-c893a79a7cff) #1" status="skipped"></testcase>
    <testcase name="Accès direct à une page (uid:5e9e0054-a892-4a1e-a53d-c893a79a7cff) #2" status="skipped"></testcase>
  </testsuite>
</testsuites>

#4

Ok I get the problem.
Normally, a JUnit report for a skipped test would look like this:

      <testcase classname="somewhere.over" name="Third scenario" file="/somewhere/over/the/rainbow" time="16.64">
        <skipped />
      </testcase>

Those ones are recognized as “Skipped” when pushed back. I’m not sure there are reporters that mark the slipped tests this way, we might have to dig to JUnit reporters (but as there’s no real “norm” around this format, this might be tricky)


#5

Ok.

I use behat with command line

--format=junit --out=./results/junit

Have you got any way to push results to Hiptest with correct status from Behat ?

Thanks a lot.


#6

Hi,

I just did a test with hps-behat (https://github.com/hiptest/hps-behat) by adding a scenario where steps are not implemented.
What I get as a result is:

<testsuite name="Serve coffee" tests="2" failures="0" skipped="0" errors="0" time="0.002">
  <testcase name="Simple use" time="0.001" status="passed"/>
  <testcase name="Not implemented" time="0.001" status="undefined"/>
</testsuite>

Are you sure the skipped tests get a status=“skipped” tag in the JUnit report ?
I could add the support to mark tests as skipped when finding status=“skipped” or status=“undefined” (although this second one seems a bit too generic for me to be frank).

Could you give me some more inputs on how to get a test marked as skipped so I can ensure getting the correct XML output to handle ?
By the way, which version of Behat do you use ? (that could explain the difference).
Also which formatter do you use for generating JUnit ? (and which version too)


#7

Hi !

I launch my test with behat 3.3 and have same results after update to behat last version 3.4.1

I use Selenium GRID over MinkExtention for Behat and Chrome Automation.

I can get “Skipped” Status when Selenium GRID is not correctly configured :
Chrome was updated and the current webdriver I became obsolete.

Exception thrown org.openqa.selenium.WebDriverException: disconnected: unable to connect to renderer.

Behat give me a JUNIT report with skipped status:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="elec">
   <testsuite name="01 - Initialisation" tests="2" skipped="2" failures="0" errors="0">
     <testcase name="Création des opérations (uid:9f729fd5-747d-4160-8071-d3925f07c608) #1" status="skipped"></testcase>
    <testcase name="Création des opérations (uid:9f729fd5-747d-4160-8071-d3925f07c608) #2" status="skipped"></testcase>
  </testsuite>
</testsuites>

(real output this time)


#8

Ok, I’ll add the support for this syntax in Hiptest :slight_smile:
(main problem with JUnit XML is that there’s no real norm, so every framework has more or less its own output :confused: )

I keep you informed as soon as it is deployed


#9

Hi Eric,

The fix has been deployed on Hiptest, you should now be able to import your tests marked as skipped :slight_smile:

Best regards,
Vincent


#10

Hi Vincent,

That’s good ! Je skipped status was pushed on Hiptest.

Thanks a lot for your action !

Eric.


#11

Hi !

I have another status that is not correctly pushed on Hiptest from Behat : Pending status expected but Failed status displayed.

Here an example of Behat JUNIT result file :

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="test">
  <testsuite name="Projet Test" tests="2" skipped="0" failures="0" errors="2">
    <testcase name="Scénario 1 (uid:ee4ded54-79b5-4d77-aab0-75cbada31e0c)" status="pending">
      <error message="When I am on &quot;http://www.google.fr&quot;: The actionword &quot;iAmOnUrl&quot; was not implemented." type="pending"></error>
    </testcase>
    <testcase name="Scénario 2 (uid:54bf2fd5-ebba-4ca5-9fa1-21c5abd77e9a)" status="pending">
      <error message="When I am on &quot;http://www.google.fr&quot;: The actionword &quot;iAmOnUrl&quot; was not implemented." type="pending"></error>
    </testcase>
  </testsuite>
</testsuites>

Thanks a lot.