Convert (or branch off) a node lib version of abao

[ProLoser]

@plroebuck I have found that abao just isn't powerful enough for me. It's essentially an obfuscation around mocha that doesn't really give you the ability to perform tests across requests (like testing the oauth dance) and other features.

What do you think about converting abao into a digestible node library so that we can simply write the chai tests ourselves and we use abao to convert raml into request objects with methods to perform validation checks against schema and headings, etc.

This way we can do things like writing the tests and working with the data ourselves, rather than having to bloat abao itself with features and configuration options.

[cybertk]

This way will binding abao to nodejs only. I think it will decrease the usage scope.

I think @ProLoser's topic is important. We need define what abao is and what features it should provide before enhancing it.

[ProLoser]

I just don't think what abao currently does is good enough

[plroebuck]

@ProLoser, I'm curious about your proposal, but would like to hear the design plans; not completely following what you're proposing -- digestable? For me, this tool had been about being able to test that my API [implementation] agreed with my RAML while being reasonably fast/thorough without much work on my part [perhaps like a pre-commit step]; think many other users would agree. Comprehensive testing (like what I think you're proposing) -- not sure how to discern intent just from the RAML itself (but sweet idea if we could write the "brains" to do so). That said, think it's beyond what abao represents IMO -- not that we couldn't expand on the original design, but think what you want is beyond its scope (based on what I've digested thus far). @cybertk, I disagree with some of the program's README assertions; for example, we don't verify Location header information specified in the RAML, but documentation implies we do. If we fixed the "multiple test per resource" issue, I think it would handle most people's needs just fine. But that's just my opinion...

[oharlem]

>> We need define what abao is ...

It does not make any sense to check against some file that calls itself "raml" without actual validation for RAML spec conformity.
OK, I got 200 response. Hurray! Success? Not exactly. The whole reason of "abao" existence is to provide RAML validation as follows from the home page statement: "...tool for testing API documentation ...". So the focus here is more on following the standard than making sure that API itself works. Did not work? oops. But not a primary concern. Maybe its just a bad definition, and not the API.

This brings us to a fork:

1. Either the statement is correct, and abao's primary purpose is to provide RAML file validation - then that's the direction of further development.
2. Or, maybe the statement is incorrect and the intent is to provide a test runner, that would just accept RAML definitions file to know what endpoints to expect. Fine.
   But here we need to:
   a. Either, again, provide automated validation in the end, otherwise response codes are not really helping.
   b. Or provide infrastructure for manual testing, i.e. add an assertions and mocking library (hint: it can use RAML's "examples" as mocks, as I'm already doing in my project - my Go app is reading example JSONs from RAML directory as mock responses from of my httptest servers).
   c. Fix the homepage description

[plroebuck]

@mpmlj, the README.md clearly states:

testing API documentation written in RAML format against its back-end implementation.

This tool creates tests to verify the RAML routes against server responses -- period.
We use another NodeJS module for parsing the RAML portion; our code makes use
of that parsed information, but we rely on the other module completely for that.

Think it's currently a mix of 2a and 2b; while there is hope to move more towards
the former as it grows, I don't see it writing the hooks for you coming any time soon.
If you get a response body, it conforms to the RAML-provided schema -- ensuring the
response makes any logical sense is up to you.

[ProLoser]

I don't really care what abao is or is not supposed to do. If this means making a different project, fine.

Here's the bottom line:

I need to test an api for regressions. Sometimes I'm testing response payloads, sometimes I test headers, sometimes these tests span multiple endpoints in succession (create, then list, then read, then delete, then list, then read with 404). Right now, abao generates the tests for me, and while it is nice, the tests it generates are not complex enough for what I need.

Instead of building more features and configuration options into abao, I'd rather just use abao to help me write the tests myself.

I don't see a benefit of doing:

abao.before('GET /api -> 200', fn(test) {
  test.foo = bar
})
abao.after('GET /api -> 200', fn(test) {
  if (test.bar === foo)
    // ...
})

over:

it('should respond with x', function(){
  // before logic
  options.foo = bar;
  abao.get('/api/step1', options).then(fn(response1){
    // after logic
    assert(response1.bar).is(foo);
    // response validation
    assert(response1.isValid()).is(true);

    abao.get('/api/step2', response1).then(...);
  })
})

My proposal is to use abao to assist in writing tests, not create a completely new testing library I have to use. If in fact what I propose ^ can be done out of the box with whatever node library it is that abao uses, then point me to it and I'll simply abandon abao because I feel like I'm starting to waste my time hacking on an obfuscation library.

The CLI features of abao are limited, I work with multiple raml files, and the project only has 100-200 stars so being paranoid about the number of users going down because we limit them to node seems silly to me considering the hook files are node.

[ProLoser]

I started working on this instead:

'use strict';

let request = require('request');

class Endpoint {
  constructor(path, data) {
    this.path = path;
    this.data = data;
    this.resource = this.getResource(path, data);
    if (!this.resource)
      throw Error('Resource not found', path, data);
  }

  getResource(path, data) {
    function iterate(resources, parent) {
      let length = resources.length;
      for (var i = 0; i < length; i++) {
        let resource = resources[i];
        parent = parent + resource.relativeUri;
        if (parent === path) {
          return resource;
        } else if (~path.indexOf(parent) && resource.resources) {
          return iterate(resource.resources, parent);
        }
      }
    }
    return iterate(data.resources, '');
  }

  tokenize(tokens, text) {
    for (var token of Object.keys(tokens)) {
      text = text.replace(`{${token}}`, tokens[token]);
    }
    return text;
  }

  request(options, callback) {
    let params = options.params || this.params || {};
    params.version = params.version || this.data.version;
    let url = this.data.baseUri + this.path;
    if (params)
      url = this.tokenize(params, url);
    options = Object.assign({
      url: url
    }, options);
    return request(options, (error, response, body) => {
      return response = new Response(response);
    });
  }
}


class Response {
  constructor(data) {
    Object.assign(this, data);
  }

  isValid() {
    return true;
  }
}

Usage:

let raml = require('raml-parser');

raml.loadFile('../abao/test/fixtures/simple.raml').then( function(data) {

  let endpoint = new Endpoint('/songs/{songId}', data);

  endpoint.params = {
    type: 'channels',
    reqPayload: 'abc123',
    version: 'v2',
    songId: '123'
  };

  endpoint.request({
    method: 'GET'
  }, function(err, res){
    response.isValid();
  }, function(err, res){
    response.isValid();
  });
});

[kevinduffey-vmware]

Crazy thought here.. why not look at using Postman for this sort of thing? Postman can import RAML (still a work in progress) and you could build complex multi-api endpoint requests in a collection format. You can use it as a node library (it has a tool called newman that runs as headless CLI) and I believe to some extent do what you are intending to do. ABAO as far as I know just takes each defined endpoint in the RAML file and tests it. You are looking to control the actual flow of API calls, which could also mean multiple RAML files (making up multiple API surfaces for example), which is outside the scope of ABAO. I would look to using Postman and figuring out how to utilize Postmans capabilities for this.

[ProLoser]

@kevinduffey-vmware what I am proposing is simplifying the library so that you can do more with it. What you are proposing is roping in 2 other tools (a commandline tool and a gui).

I'm not sure I understand the push-back. I understand that abao is somewhat convenient in that it runs rudimentary tests out-of-the-box but I still feel it's extremely limited and going to get up-its-own-asshole with proprietary options, api, and configuration. It's just sort of a PITA to pick up for someone new due to the mixed documentation around it.

[plroebuck]

@ProLoser, seems like @kevinduffey-vmware was just suggesting Postman as an alternative to abao that might meet your needs better. Not sure what you mean about "proprietary options, api, and configuration". I've been trying to improve both its documentation and what it does. If either is lacking, please specify...

[ProLoser]

I just don't like having to do before and after hooks, and I don't like the approach of how you must specify an endpoint to be tested (that large string version of the endpoint), and I don't like that you must change the string name to have random notes concatenated to the end in order to test an endpoint in multiple ways.

This ended up being a decent portion of my hook files:

var hooks = require('hooks');
var reqPayload = process.env.REQUEST_PAYLOAD;
var endpoint = '/browse/items/{browse_type}/filter/{filter_type}/sort/{sort_method}/offset/{offset}/size/{size}';

function before(params, debug) {
  var name = 'GET '+endpoint+' -> 200 '+JSON.stringify(params);
  hooks.before(name, function(test, done) {
    test.request.headers.reqPayload = reqPayload;
    // Optional default params
    test.request.params = Object.assign({
      filter_type: 'all',
      offset: 0,
      size: 3
    }, params);
    done();
  });
  if (debug) {
    hooks.after(name, function(test, done) {
      console.log(test.response.body);
      done();
    });
  }
}

before({
  browse_type: 'now_playing',
  sort_method: 'popular'
});
before({
  browse_type: 'favorites',
  sort_method: 'numeric'
});
before({
  browse_type: 'recently_watched',
  sort_method: 'numeric'
});
before({
  browse_type: 'recommended',
  sort_method: 'numeric'
});
before({
  browse_type: 'featured',
  sort_method: 'featured'
});

It just doesn't feel very elegant. I'd rather write normal unit tests in chai or mocha or whatever and just use abao to simply execute the queries and provide me with a method to validate the schemas and nothing more.