skip_testcase_on_fail Does not skip the remaning test case on failure. #3939

gravityvi · 2023-10-19T08:55:59Z

Description of the bug/issue

Context

There are test-suite use cases that can achieved through a combination of abortOnFailure and skip_test_on_fail.
abortOnFailure is to determine whether to end the execution of the test case at that node or continue the execution of the test case. The flag does not affect marking a test as a pass or execution of subsequent test cases. Whereas skip_testcase_on_failure will allow us to decide to skip the following test cases in the test suite if a failure occurs in the previous test case.

The bug is created when abortOnFailure is false and skip_testcase_on_fail is set to true. Upon failure, the rest of the test cases are not skipped. This can be reproduced with the example shown below.

Steps to reproduce

Run the test suite shown below with nightwatch.

Sample test

module.exports = {
  'wait for not': () => {
    browser.url('https://www.nightwatchjs.org');
    browser.waitForElementNotVisible('#header', 1000, 100, false) //failure
    browser.pause(1);
  },
  'wait for': () => {
    browser.url('https://www.nightwatchjs.org');
    browser.setAttribute('nav.navigation', 'hidden', 'true');
    browser.waitForElementVisible('nav.navigation', 1000, 100, false); 
  }
};

Command to run

npx nightwatch example.js --env chrome --verbose

Verbose Output

Starting ChromeDriver on port 9515...
   Request POST /session  
   {
     capabilities: {
       firstMatch: [ {} ],
       alwaysMatch: {
         browserName: 'chrome',
         webSocketUrl: true,
         'goog:chromeOptions': {
           w3c: true,
           args: [ '--remote-debugging-port=0' ],
           binary: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
         }
       }
     }
⠋ Starting ChromeDriver on port 9515...
   Response 200 POST /session (1546ms)
   {
     value: {
       capabilities: {
         acceptInsecureCerts: false,
         browserName: 'chrome',
         browserVersion: '118.0.5993.70',
         chrome: {
           chromedriverVersion: '118.0.5993.70 (e52f33f30b91b4ddfad649acddc39ab570473b86-refs/branch-heads/5993@{#1216})',
           userDataDir: '/var/folders/n9/8h4bw85j2y95c2rw67sjjdmr0000gn/T/.org.chromium.Chromium.VXtmxY'
         },
         'fedcm:accounts': true,
         'goog:chromeOptions': { debuggerAddress: 'localhost:58181' },
         networkConnectionEnabled: false,
         pageLoadStrategy: 'normal',
         platformName: 'mac',
         proxy: {},
         setWindowRect: true,
         strictFileInteractability: false,
         timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },
         unhandledPromptBehavior: 'dismiss and notify',
         webSocketUrl: 'ws://localhost:9515/session/26feabcc566d0e35e8545d6ead0a09ef',
         'webauthn:extension:credBlob': true,
         'webauthn:extension:largeBlob': true,
         'webauthn:extension:minPinLength': true,
         'webauthn:extension:prf': true,
         'webauthn:virtualAuthenticators': true
       },
       sessionId: '26feabcc566d0e35e8545d6ead0a09ef'
     }
ℹ Connected to ChromeDriver on port 9515 (1714ms).
  Using: chrome (118.0.5993.70) on MAC.

 Received session with ID: 26feabcc566d0e35e8545d6ead0a09ef

 → Running [before]:
 → Completed [before].



  Running wait for not:
───────────────────────────────────────────────────────────────────────────────────────────────────
 → Running [beforeEach]:
 → Completed [beforeEach].
 
 → Running command: url ('https://www.nightwatchjs.org')
  ⠋ Loading url: https://www.nightwatchjs.org
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/url  
  ⠙ Loading url: https://www.nightwatchjs.org
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/url (2559ms)
  ℹ Loaded url https://www.nightwatchjs.org in 2563ms
 → Completed command: url ('https://www.nightwatchjs.org') (2570ms)
 
 → Running command: waitForElementNotVisible ('#header', 1000, 100, false)
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/elements  
   { using: 'css selector', value: '#header' }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/elements (17ms)
   {
     value: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (18ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (17ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (13ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (19ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (20ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (18ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (12ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (18ms)
   { value: true }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28',
         ELEMENT: 'B13B688FFFF3CAE1BC5C0EFA7A9EA08C_element_28'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (18ms)
   { value: true }
  ✖ NightwatchAssertError
   Timed out while waiting for element <#header> to not be visible for 1000 milliseconds. - expected "not visible" but got: "visible" (1003ms)

    Error location:
    /Users/ravi/Documents/Projects/nightwatch/examples/tests/ecosia.js:
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
     2 |   'wait for not': () => {
     3 |     browser.url('https://www.nightwatchjs.org');
     4 |     browser.waitForElementNotVisible('#header', 1000, 100, false) 
     5 |     browser.pause(1);
     6 |   },
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

 → Completed command: waitForElementNotVisible ('#header', 1000, 100, false) (1011ms)
 
 → Running command: pause (1)
 → Completed command: pause (1) (2ms)
 → Running [afterEach]:
 → Completed [afterEach].

  FAILED: 1 assertions failed (3.594s)



  Running wait for:
───────────────────────────────────────────────────────────────────────────────────────────────────
 → Running [beforeEach]:
 → Completed [beforeEach].
 
 → Running command: url ('https://www.nightwatchjs.org')
  ⠋ Loading url: https://www.nightwatchjs.org
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/url  
  ⠙ Loading url: https://www.nightwatchjs.org
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/url (142ms)
  ℹ Loaded url https://www.nightwatchjs.org in 145ms
 → Completed command: url ('https://www.nightwatchjs.org') (145ms)
 
 → Running command: setAttribute ('nav.navigation', 'hidden', 'true')
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/elements  
   { using: 'css selector', value: 'nav.navigation' }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/elements (11ms)
   {
     value: [
       {
         'element-6066-11e4-a52e-4f735466cecf': '66491573582C4293AF6020DBE7C7A88E_element_36'
       }
     ]
  }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: 'var passedArgs = Array.prototype.slice.call(arguments,0); return (function (e, a, v) { try {\n' +
       "                        if (e && typeof e.setAttribute == 'function') {\n" +
       '                            e.setAt... (480 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': '66491573582C4293AF6020DBE7C7A88E_element_36'
       },
       'hidden',
       'true'
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (8ms)
   { value: true }
 → Completed command: setAttribute ('nav.navigation', 'hidden', 'true') (24ms)
 
 → Running command: waitForElementVisible ('nav.navigation', 1000, 100, false)
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/elements  
   { using: 'css selector', value: 'nav.navigation' }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/elements (7ms)
   {
     value: [
       {
         'element-6066-11e4-a52e-4f735466cecf': '66491573582C4293AF6020DBE7C7A88E_element_36'
       }
     ]
  }
   Request POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync  
   {
     script: '/* isDisplayed */return (function(){return (function(){var k=this||self;function aa(a){return"string"==typeof a}function ba(a,b){a=a.split(".");var c=k;a[0]in c||"undefined"==typeof c.execScript||c.ex... (44044 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': '66491573582C4293AF6020DBE7C7A88E_element_36',
         ELEMENT: '66491573582C4293AF6020DBE7C7A88E_element_36'
       }
     ]
  }
   Response 200 POST /session/26feabcc566d0e35e8545d6ead0a09ef/execute/sync (10ms)
   { value: true }
  ✔ Element <nav.navigation> was visible after 19 milliseconds.
 → Completed command: waitForElementVisible ('nav.navigation', 1000, 100, false) (20ms)
 → Running [afterEach]:
 → Completed [afterEach].

  ✨ PASSED. 1 assertions. (192ms)
 → Running [after]:
 → Completed [after].
 
 → Running command: end (true)
 
 → Running command: session ('delete', [Function])
   Request DELETE /session/26feabcc566d0e35e8545d6ead0a09ef  

   Response 200 DELETE /session/26feabcc566d0e35e8545d6ead0a09ef (77ms)
   { value: null }
 → Completed command: session ('delete', [Function]) (82ms)
 Wrote log file to: /Users/ravi/Documents/Projects/nightwatch/logs/ecosia_chromedriver.log
 → Completed command: end (true) (93ms)

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  ️TEST FAILURE (6.08s):  
   - 1 assertions failed; 1 passed

   ✖ 1) ecosia

   – wait for not (3.594s)

   → ✖ NightwatchAssertError
   Timed out while waiting for element <#header> to not be visible for 1000 milliseconds. - expected "not visible" but got: "visible" (1003ms)

    Error location:
    /Users/ravi/Documents/Projects/nightwatch/examples/tests/ecosia.js:
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
     2 |   'wait for not': () => {
     3 |     browser.url('https://www.nightwatchjs.org');
     4 |     browser.waitForElementNotVisible('#header', 1000, 100, false) 
     5 |     browser.pause(1);
     6 |   },
    ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––


 ChromeDriver process closed.
 Wrote HTML report file to: /Users/ravi/Documents/Projects/nightwatch/tests_output/nightwatch-html-report/index.html

 Wrote JSON report file to: /Users/ravi/Documents/Projects/nightwatch/tests_output/CHROME_118.0.5993.70__ecosia.json
 Wrote Rerun Json report file to: /Users/ravi/Documents/Projects/nightwatch/tests_output/minimal_report.json
 Wrote XML report file to: /Users/ravi/Documents/Projects/nightwatch/tests_output/CHROME_118.0.5993.70__ecosia.xml

Nightwatch Configuration

No response

Nightwatch.js Version

3.2.1

Node Version

16

Browser

No response

Operating System

No response

Additional Information

No response

The text was updated successfully, but these errors were encountered:

Kartikeya-09 · 2024-02-23T14:35:37Z

@gravityvi This issue is still open or not..

BishalJena · 2024-08-02T18:27:18Z

I'd like to work on this issue.
@gravityvi

gravityvi · 2024-08-04T09:54:05Z

Hey @BishalJena 👋🏻 , sure go ahead. I have not contributed since a while now so I might have lost context but @garg3133 can definitely help here.

gravityvi mentioned this issue Oct 19, 2023

abortOnFailure=false for waitForElementVisible/NotVisible does not seem to prevent failures #3931

Closed

gravityvi added bug complexity: high p2 labels Oct 19, 2023

ac-mmi mentioned this issue Aug 21, 2024

Fix: skip_testcase_on_fail Does not skip the remaning test case on failure. #4254

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

skip_testcase_on_fail Does not skip the remaning test case on failure. #3939

skip_testcase_on_fail Does not skip the remaning test case on failure. #3939

gravityvi commented Oct 19, 2023

Kartikeya-09 commented Feb 23, 2024

BishalJena commented Aug 2, 2024

gravityvi commented Aug 4, 2024

skip_testcase_on_fail Does not skip the remaning test case on failure. #3939

skip_testcase_on_fail Does not skip the remaning test case on failure. #3939

Comments

gravityvi commented Oct 19, 2023

Description of the bug/issue

Context

Steps to reproduce

Sample test

Command to run

Verbose Output

Nightwatch Configuration

Nightwatch.js Version

Node Version

Browser

Operating System

Additional Information

Kartikeya-09 commented Feb 23, 2024

BishalJena commented Aug 2, 2024

gravityvi commented Aug 4, 2024