When using Cucumber as a runner a NightwatchAssertError thrown in Page Objects exits node process abruptly

[memihai]

Description of the bug/issue

When the runner is Cucumber if an assertion fails in a Page Object the process exists abruptly with no stack trace.

Steps to reproduce

1. define a custom command containing a .waitForElementVisible() step
2. define a page object command that calls the custom command with and invalid selector (CSS or XPath)
3. use the page object command in a Cucumber step
4. Init Nightwatch with cucumber.js and chromedriver

Actual:
Node process exists with exit code 0 after the NightwatchAssertError is thrown.

Expected:
Test is failed and next test is run.

Sample test

// Using the Nightwatch example test installed with 'npm init nightwatch'

// customCommands/customWait
module.exports = class customWait {
  async command(selector , typeOfSelector='xpath') {
    await browser.waitForElementVisible('xpath', selector)
  }
}
//page_bojects/RijksmuseumHomepage.js
const xpath = {
  randomXpath: searchString =>`//body[contains(text(), ${searchString}]`,
};

module.exports = {

  elements: [
    {
      randomXpath:{
        selector : "//body[contains(text(),\"Random string\")]",
        locateStrategy: 'xpath'
      }

    },

  ],

  commands: [
   {
      waitForAnElementToBeVisible() {
        //this.expect.element('@randomXpath').to.be.visible;
        this.customWait(xpath.randomXpath('Random string'));
        return this;
      }
    }
  ],

};

// test/features/nightwatch/step_definitions/nightwatch.js
const {Given, Then, When, Before} = require('@cucumber/cucumber');

Given(/^I open the Rijksmuseum page$/,  function() {

  browser.navigateTo('https://www.rijksmuseum.nl/en');
  return browser.page.RijksmuseumHomepage().waitForAnElementToBeVisible();
});

// test/features/nightwatch/nightwatch.feature
Feature: Google Search

  @a
Scenario: Open URL and wait for element
  Given I open the Rijksmuseum page

  @a
  Scenario: Second scenario to Open URL and wait for element
    Given I open the Rijksmuseum page

Command to run

npx nightwatch --verbose

Verbose Output

Launching up to 1 concurrent test worker processes...

 Running:  default: features/nightwatch/step_definitions/nightwatch.js

┌ ────────────────── ✔  default: features/nightwatch/step_definitions/nightwatch.js  ──────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                                                                                                                                  │
│                                                                                                                                                                                  │
│    - Starting ChromeDriver on port 9515...                                                                                                                                       │
│    Starting ChromeDriver with server_path=/Users/mihai.merce/IdeaProjects/bug_nigtwatch_cucumber/node_modules/chromedriver/lib/chromedriver/chromedriver...                      │
│      Request POST /session                                                                                                                                                       │
│    {                                                                                                                                                                             │
│         capabilities: {                                                                                                                                                          │
│           firstMatch: [ {} ],                                                                                                                                                    │
│           alwaysMatch: { browserName: 'chrome', 'goog:chromeOptions': {} }                                                                                                       │
│         }                                                                                                                                                                        │
│      }                                                                                                                                                                           │
│      Response 200 POST /session (3226ms)                                                                                                                                         │
│    {                                                                                                                                                                             │
│         value: {                                                                                                                                                                 │
│           capabilities: {                                                                                                                                                        │
│             acceptInsecureCerts: false,                                                                                                                                          │
│             browserName: 'chrome',                                                                                                                                               │
│             browserVersion: '109.0.5414.119',                                                                                                                                    │
│             chrome: {                                                                                                                                                            │
│               chromedriverVersion: '109.0.5414.74 (e7c5703604daa9cc128ccf5a5d3e993513758913-refs/branch-heads/5414@{#1172})',                                                    │
│               userDataDir: '/var/folders/kr/_h84sd115676rkpk977vv36hb1_10l/T/.com.google.Chrome.snnSYA'                                                                          │
│             },                                                                                                                                                                   │
│             'goog:chromeOptions': { debuggerAddress: 'localhost:58411' },                                                                                                        │
│             networkConnectionEnabled: false,                                                                                                                                     │
│             pageLoadStrategy: 'normal',                                                                                                                                          │
│             platformName: 'mac os x',                                                                                                                                            │
│             proxy: {},                                                                                                                                                           │
│             setWindowRect: true,                                                                                                                                                 │
│             strictFileInteractability: false,                                                                                                                                    │
│             timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },                                                                                                          │
│             unhandledPromptBehavior: 'dismiss and notify',                                                                                                                       │
│             'webauthn:extension:credBlob': true,                                                                                                                                 │
│             'webauthn:extension:largeBlob': true,                                                                                                                                │
│             'webauthn:virtualAuthenticators': true                                                                                                                               │
│           },                                                                                                                                                                     │
│           sessionId: '2ced7bf75fa47285ae8d3bd4951874a6'                                                                                                                          │
│         }                                                                                                                                                                        │
│      }                                                                                                                                                                           │
│    ℹ Connected to ChromeDriver on port 9515 (3420ms).                                                                                                                            │
│    Using: chrome (109.0.5414.119) on MAC OS X.                                                                                                                                   │
│    Received session with ID: 2ced7bf75fa47285ae8d3bd4951874a6                                                                                                                    │
│                                                                                                                                                                                  │
│    .                                                                                                                                                                             │
│    .                                                                                                                                                                             │
│                                                                                                                                                                                  │
│     → Running command: navigateTo ('https://www.rijksmuseum.nl/en')                                                                                                              │
│      Request POST /session/2ced7bf75fa47285ae8d3bd4951874a6/url                                                                                                                  │
│    { url: 'https://www.rijksmuseum.nl/en' }                                                                                                                                      │
│      Response 200 POST /session/2ced7bf75fa47285ae8d3bd4951874a6/url (2813ms)                                                                                                    │
│    { value: null }                                                                                                                                                               │
│     → Completed command: navigateTo ('https://www.rijksmuseum.nl/en') (2816ms)                                                                                                   │
│                                                                                                                                                                                  │
│     → Running command: customWait ('//body[contains(text(), "Random string")]')                                                                                                  │
│                                                                                                                                                                                  │
│     → Running command: waitForElementVisible ('xpath', '//body[contains(text(), "Random string")]')                                                                              │
│      Request POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements                                                                                                             │
│    { using: 'xpath', value: '//body[contains(text(), "Random string")]' }                                                                                                        │
│      Response 200 POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements (20ms)                                                                                                 │
│    { value: [] }                                                                                                                                                                 │
│      Request POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements                                                                                                             │
│    { using: 'xpath', value: '//body[contains(text(), "Random string")]' }                                                                                                        │
│      Response 200 POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements (10ms)                                                                                                 │
│    { value: [] }                                                                                                                                                                 │
│      Request POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements                                                                                                             │
│    { using: 'xpath', value: '//body[contains(text(), "Random string")]' }                                                                                                        │
│      Response 200 POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements (8ms)                                                                                                  │
│    { value: [] }                                                                                                                                                                 │
│      Request POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements                                                                                                             │
│    { using: 'xpath', value: '//body[contains(text(), "Random string")]' }                                                                                                        │
│      Response 200 POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements (11ms)                                                                                                 │
│    { value: [] }                                                                                                                                                                 │
│      Request POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements                                                                                                             │
│    { using: 'xpath', value: '//body[contains(text(), "Random string")]' }                                                                                                        │
│      Response 200 POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements (10ms)                                                                                                 │
│    { value: [] }                                                                                                                                                                 │
│      Request POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements                                                                                                             │
│    { using: 'xpath', value: '//body[contains(text(), "Random string")]' }                                                                                                        │
│      Response 200 POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements (10ms)                                                                                                 │
│    { value: [] }                                                                                                                                                                 │
│      Request POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements                                                                                                             │
│    { using: 'xpath', value: '//body[contains(text(), "Random string")]' }                                                                                                        │
│      Response 200 POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements (10ms)                                                                                                 │
│    { value: [] }                                                                                                                                                                 │
│      Request POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements                                                                                                             │
│    { using: 'xpath', value: '//body[contains(text(), "Random string")]' }                                                                                                        │
│      Response 200 POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements (11ms)                                                                                                 │
│    { value: [] }                                                                                                                                                                 │
│      Request POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements                                                                                                             │
│    { using: 'xpath', value: '//body[contains(text(), "Random string")]' }                                                                                                        │
│      Response 200 POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements (9ms)                                                                                                  │
│       { value: [] }                                                                                                                                                              │
│      Request POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements                                                                                                             │
│    { using: 'xpath', value: '//body[contains(text(), "Random string")]' }                                                                                                        │
│      Response 200 POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements (10ms)                                                                                                 │
│    { value: [] }                                                                                                                                                                 │
│      Request POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements                                                                                                             │
│       { using: 'xpath', value: '//body[contains(text(), "Random string")]' }                                                                                                     │
│      Response 200 POST /session/2ced7bf75fa47285ae8d3bd4951874a6/elements (102ms)                                                                                                │
│       { value: [] }                                                                                                                                                              │
│     ✖ NightwatchAssertError                                                                                                                                                      │
│       Timed out while waiting for element <//body[contains(text(), "Random string")]> to be present for 5000 milliseconds. - expected "visible" but got: "not found" (5229ms)    │
│                                                                                                                                                                                  │
│        Error location:                                                                                                                                                           │
│        /Users/mihai.merce/IdeaProjects/bug_nigtwatch_cucumber/customCommands/customWait.js:                                                                                      │
│        –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––                                                                                     │
│         1 | module.exports = class customWait {                                                                                                                                  │
│         2 |   async command(selector , typeOfSelector='xpath') {                                                                                                                 │
│         3 |     await browser.waitForElementVisible('xpath', selector)                                                                                                           │
│         4 |   }                                                                                                                                                                  │
│         5 | }                                                                                                                                                                    │
│        –––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––                                                                                     │
│                                                                                                                                                                                  │
│     → Completed command: waitForElementVisible ('xpath', '//body[contains(text(), "Random string")]') (5241ms)                                                                   │
│      → Completed command: customWait ('//body[contains(text(), "Random string")]') (5244ms)        
      --- Process finished with exit code 0  at this point  ---                                                                         │└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Nightwatch Configuration

module.exports = {
  // An array of folders (excluding subfolders) where your tests are located;
  // if this is not specified, the test source must be passed as the second argument to the test runner.
  src_folders: ['test'],

  // See https://nightwatchjs.org/guide/concepts/page-object-model.html
  page_objects_path: ['page_objects'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-commands.html
  custom_commands_path: ['customCommands'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-assertions.html
  custom_assertions_path: [],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-plugins.html
  plugins: [],
  
  // See https://nightwatchjs.org/guide/concepts/test-globals.html
  globals_path: '',
  
  webdriver: {},

  test_workers: {
    enabled: true
  },

  test_settings: {
    default: {
      disable_error_log: false,
      launch_url: 'http://localhost',

      screenshots: {
        enabled: false,
        path: 'screens',
        on_failure: true
      },

      desiredCapabilities: {
        browserName: 'chrome'
      },
      
      webdriver: {
        start_process: true,
        server_path: ''
      },
      
      test_runner: {
        // set cucumber as the runner
        // For more info on using CucumberJS with Nightwatch, visit:
        // https://nightwatchjs.org/guide/writing-tests/using-cucumberjs.html
        type: 'cucumber',

        // define cucumber specific options
        options: {
          //set the feature path
          feature_path: 'test/features',

          // start the webdriver session automatically (enabled by default)
          // auto_start_session: true

          // use parallel execution in Cucumber
          // parallel: 2 // set number of workers to use (can also be defined in the cli as --parallel 2
        }
      }
    },
    
    chrome: {
      desiredCapabilities: {
        browserName: 'chrome',
        'goog:chromeOptions': {
          // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/
          //
          // w3c:false tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78)
          w3c: true,
          args: [
            //'--no-sandbox',
            //'--ignore-certificate-errors',
            //'--allow-insecure-localhost',
            //'--headless'
          ]
        }
      },

      webdriver: {
        start_process: true,
        server_path: '',
        cli_args: [
          // --verbose
        ]
      }
    },
    
  },
  
};

Nightwatch.js Version

2.6.11

Node Version

17.8.0

Browser

Chrome 109.0.5414.119

Operating System

No response

Additional Information

This reproduces with custom commands and expect statemetns like
this.expect.element('@randomslector').to.be.visible;

[asos-saveriocutrupi]

@memihai Check this issue, I think there might be an overlap, as it's the same kind of error.

[memihai]

@asos-saveriocutrupi I hit that error too but that one hangs the process indefinitely. I never logged it as I didn't know how to reproduce it.

There are 2 differences between that and the current issue:

1. The problem I am seeing is not related to custom commands as expect statements still fail if they are in a Page Object and the selector doesn't match an element.
2. The Node process exists abruptly (without stack trace obviously)
   I have no idea what blows up and I could find no workaround to this other than moving assertions into the step definitions, which is not an option for me.

But thanks for pointing that issue out! Good to know about it as I can now avoid it 👍 .

[memihai]

I played around with this today and realised that if I define the command like this

commands: [
   {
      async waitForAnElementToBeVisible() {
         //this.expect.element('@randomXpath').to.be.visible;
         await this.customWait(xpath.randomXpath('Random string'));
       }
     }
   ],

So it returns a promise then I use it like so in the step definition:

Given(/^I open the Rijksmuseum page$/, async function() {
 await browser.navigateTo('https://www.rijksmuseum.nl/en');
 await browser.page.RijksmuseumHomepage().waitForAnElementToBeVisible();

});

Then the tests behave as expected. After the first test failed the second one is run and the result is 2 out of 2 failures.

[AutomatedTester]

There is a simpler test case for this issue in https://discord.com/channels/618399631038218240/1072553726193184788/threads/1083290462011523142

describe('Ecosia', function() {

  // test() and specify() is also available

  it('demo test', function(browser) {
    browser
      .url('https://www.ecosia.org/%27)
      .setValue('input[type=search]', 'nightwatch')
      .click('button[type=submit]')
     // .assert.containsText('.mainline-results', 'Nightwatch.js')
    browser.waitForElementNotPresent('#dialog', 50000)
      .end();
  });
});

Updating priority. Hopefully we can get to this soon!

[ituradastra]

I can also reproduce this issue and it is one of the blockers for upgrading to v2.

[memihai]

The problem described here is still an issue in Nightwatch 3.0.1, I just tested it today.
@ituradastra FYI.

[memihai]

@gravityvi, @AutomatedTester Do you have any updates on this?
We want to resume our efforts to migrate to NW3 now and this is a blocker for us.

[gravityvi]

@memihai I don't think it's a bug while using cucumber the nightwatch commands should return a promise or a value in the example shown above the page command isn't returning a promise.

To fix the test just return the result from customWait. Let me know if I missed something here.

Actual Example:

 waitForAnElementToBeVisible() {
        //this.expect.element('@randomXpath').to.be.visible;
        this.customWait(xpath.randomXpath('Random string'));
        return this;
      }

Fixed command:

 waitForAnElementToBeVisible() {
        return this.customWait(xpath.randomXpath('Random string'));
      }

[memihai]

@gravityvi yes your solution works for that example but what if I want to click the element after waiting for it to be visible

waitForAnElementToBeVisible() {
         this.customWait(xpath.randomXpath('Random string'));
         return this.click(xpath.randomXpath('Random string');
      }

This will fail the same way I had pointed out in the issue.

Another thing I tried is to return the result from the command which should be a promise

// customCommands/customWait
module.exports = class customWait {
  command(selector , typeOfSelector='xpath') {
   return browser.waitForElementVisible('xpath', selector)
  }
}

and then chain the wait and click:

waitForAnElementToBeVisible() {
         return this.customWait(xpath.randomXpath('Random string')).click(xpath.randomXpath('Random string'));
      }

This will fail the same way I had pointed out in the issue.

I would expect both scenariops to work or at least the last one. Am I wrong in my assumptions?