Loopback and pagination on listView

[F3L1X79]

Hi again,
I got a question related to #414
I'm actually using loopback and the customParams to manage pagination on my listViews:

RestangularProvider.addFullRequestInterceptor(function(element, operation, what, url, headers, params) {
    if (operation === "getList") {
        // custom pagination params
        if (params._page) {
            params['filter[limit]'] = params._perPage;
            params['filter[skip]'] = (params._page - 1) * params._perPage; //skip is the same os offset
            delete params._page;
            delete params._perPage;
        }
        return {params: params};
    }
});

And my listView is defined like this:

    post.listView()
        .title('Posts List')
        .perPage(10)
        .infinitePagination(false)
        ...
        .listActions(['show', 'edit', 'delete']);

I have access to all Posts while browsing all pages one by one : http://[...]/bower_components/#/Posts/list?page=2, http://[...]/bower_components/#/Posts/list?page=3, etc.
But my problem is that the pagination always shows 1 - 10 on 10 on all pages (and not 1-10 on 25 as I expected) - and the buttons of the other pages are not displaying.
-> I guess the total-items count must not be equal to the per-page count in this case, but i'm not able to change this in any ways.

If I remove the .perPage(10) attribute, I can see all my 25 Posts (1 - 25 on 25).
Do you think it is a problem with loopback, or the latest branch of ng-admin - or should I search for another solution?
Help is greatly appreciated :)
Thanks in advance,
Felix.

[jpetitcolas]

I've succeeded in reproducing the issue using Loopback Example App and the following configuration file:

/*global angular*/
(function () {
    "use strict";

    var app = angular.module('myApp', ['ng-admin']);

    app.config(['NgAdminConfigurationProvider', 'RestangularProvider', function (NgAdminConfigurationProvider, RestangularProvider) {
        var nga = NgAdminConfigurationProvider;

        RestangularProvider.addFullRequestInterceptor(function(element, operation, what, url, headers, params) {
            if (operation === "getList") {
                // custom pagination params
                if (params._page) {
                    params['filter[limit]'] = params._perPage;
                    params['filter[skip]'] = (params._page - 1) * params._perPage; //skip is the same os offset
                    delete params._page;
                    delete params._perPage;
                }
                return {params: params};
            }
        });

        var admin = nga.application('ng-admin backend demo')
            .baseApiUrl('http://localhost:3500/api/');

        var cars = nga.entity('cars');
        admin.addEntity(cars);

        cars.listView()
            .perPage(10) // <-- not required to reproduce issue
            .fields([
                nga.field('id'),
                nga.field('make'),
                nga.field('model')
            ]);

        nga.configure(admin);
    }]);
})();

Looking at it.

[jpetitcolas]

The issue here is that Loopback use a method we don't deal to return total number of records. In admin-config, we can see we use:

• Response totalCount attribute,
• X-Total-Count header
• Retrieved data length

In this case, as we don't have the two firsts, we fallback on the last one. As API returns only 10 records, we consider we get only 10 records, and thus, there is no need of pagination.

Loopback returns total number of records via a /count URL. You may try run an Ajax request in a response interceptor, something like:

app.run(['Restangular', '$http', function(RestangularProvider, $http) {
    RestangularProvider.addResponseInterceptor(function(data, operation, what, url, response, deferred) {
        if (operation !== 'getList') {
            return deferred.resolve(data);
        }

        $http.get(url + '/count')
            .success(function(countData) {
                response.totalCount = countData.count;
                return deferred.resolve(data);
            })
            .error(function(error) {
                return deferred.reject(error);
            });
    });
}]);

Yet, I wasn't able to make promises work with Restangular. I opened an issue on their repository: Asynchronous task in response interceptor?.

With Loopback, isn't it possible to move the total count in a totalCount field?

[F3L1X79]

Finally found a solution ! Not best practice but it works (did it in app.config):

app.config(['NgAdminConfigurationProvider', 'RestangularProvider', function (NgAdminConfigurationProvider, RestangularProvider) {

    function Get(yourUrl){
            var Httpreq = new XMLHttpRequest();
            Httpreq.open("GET",yourUrl,false);
            Httpreq.send(null);
            return Httpreq.responseText;          
        };

        RestangularProvider.addResponseInterceptor(function(data, operation, what, url, response, deferred) {
            if (operation === 'getList') {
                var Result = JSON.parse(Get(url + '/count'));
                response.totalCount = Result.count;
            }
            return data;
        });
}

[jpetitcolas]

Asynchronous is the issue, you did synchronous. That works, indeed. :)

Closing the issue, as no activity on Restangular issue.

[emresebat]

Hi

I know this is closed but there is a better solution here loopback#1411 by @abovegradesoftware

Just needs a small addition from your FAQ, adding the CORS header like below

 // Set X-Total-Count for all search requests
  remotes.after('*.find', function (ctx, next) {
    //do this only for ng-admin
    if (ctx.req.headers.appid === 'ng-admin') {
      var filter;
      if (ctx.args && ctx.args.filter) {
        filter = JSON.parse(ctx.args.filter).where;
      }

      if (!ctx.res._headerSent) {
        this.count(filter, function (err, count) {
          //fix for CORS
          ctx.res.set('Access-Control-Expose-Headers', 'x-total-count');
          ctx.res.set('X-Total-Count', count);
          next();
        });
      } else {
        next();
      }
    } else {
      next();
    }
  });

[F3L1X79]

Hi @emresebat ,
Thank you for this answer but I don't understand where you can put that hook inside the ng-admin config file?

In case somebody needs it, I actually upgraded my code to make it work with the filtering:

app.config(['NgAdminConfigurationProvider', 'RestangularProvider', function (NgAdminConfigurationProvider, RestangularProvider) {

function Get(url) {
    var Httpreq = new XMLHttpRequest();
    Httpreq.open("GET", url, false);
    Httpreq.send(null);
    return Httpreq.responseText;
}

RestangularProvider.addResponseInterceptor(function(data, operation, what, url, response, deferred) {
                if (operation === 'getList') {
                    var result;
                    var filters = "";
                    _.forEach(response.config.params, function (value, entry) {
                        if(entry !== 'filter[limit]' && entry !== 'filter[skip]' && entry !== 'filter[include]' && entry !== 'filter[order]'){
                            if(filters === ""){
                                filters = "/count?" + entry.replace("filter[where]", "where") + "=" + value;
                            } else {
                                filters += "&" + entry.replace("filter[where]", "where") + "=" + value;
                            }
                        }
                    });
                    result = JSON.parse(Get(url + filters));
                    response.totalCount = result.count;
                }

            return data;
        });

[emresebat]

Hi @F3L1X79

Sorry for the confusion the code I posted is for loopback :) Your solution is also a good choice if you cannot change the loopback backend and just makes a second REST call.

[sm-g]

@emresebat
What is your ctx.args.filter object? I got "{"limit":30,"skip":0,"offset":0,"order":[{"key":"id","reverse":1}]}" when load http://localhost:3000/index.html#/entity/list so that code does not work.

[BenLeroy]

@emresebat
Your solution works for me!!
Standard solution doesn't take filters in consideration so it just send the total count of objects not the filtered one and also gives as much blank pages as it shall exists on total count....With yours I have a correct count and number of pages.

[emresebat]

@sm-g
ctx.args.filter is the same as yours, but you don't provide a where clause so this.count will return all your records. According to here count should work with or without a where clause.

@BenLeroy
Happy to here it's working for you :)

[pertschuk]

Got

{"error":{"name":"SyntaxError","status":500,"message":"Unexpected token o"}}

In order to get the hook to work I had to remove the JSON.parse from @emresebat's solution running on heroku in case anyone else comes across this issue:

module.exports = function (app) {
  var remotes = app.remotes();

  // Set X-Total-Count for all search requests
  remotes.after('*.find', function (ctx, next) {
    var filter;
    if (ctx.args && ctx.args.filter) {
      filter = ctx.args.filter.where;
    }

    if (!ctx.res._headerSent) {
      this.count(filter, function (err, count) {
        ctx.res.set('Access-Control-Expose-Headers', 'x-total-count');
        ctx.res.set('X-Total-Count', count);
        next();
      });
    } else {
      next();
    }
  });
};

[anagrath]

@F3L1X79 did not work after I added authentication. When you add ACL, it is virtually impossible to get the authorization token to send in the header and then the count commands return 401. I did @emresebat solution above (put into my server.js file in loopback) and it seems to be working.

[F3L1X79]

@anagrath :
Actually I store the token in the cookies after user login, then I get them in the full request interceptor like this:

app.run(['Restangular', '$rootScope',  '$cookieStore',
        function (RestangularProvider, $rootScope,  $cookieStore) {
            $rootScope.globals = $cookieStore.get('globals') || {};
            if ($rootScope.globals && $rootScope.globals.currentUser) {
                $http.defaults.headers.common['AppId'] = 'ng-admin'; // jshint ignore:line
                $http.defaults.headers.common['Authorization'] = 'Basic ' + $rootScope.globals.currentUser.authdata; // jshint ignore:line
            }
            RestangularProvider.addFullRequestInterceptor(function (element, operation, what, url, headers, params) {
                params.access_token = $rootScope.globals.currentUser.token;
                return {params: params};
            });
        }]);

But clearly, @emresebat 's solution is better when you can modify your Loopback's sources.

[akki-ng]

To solve pagination, loopback uses skip or offset. To make pagination faster we can use something as _id : {$gt: ...}. To achieve this I've created a mixin as PaginatedFind with options

"mixins": {
    "PaginatedFind": {
      "maxLimit": 100,
      "defaultLimit": 50
    }
  }

To whichever model I need pagination, I include mixin(options options) and this mixin overrides the find method of the model and gives result in below format

{
  "perPage": 100,
  "total": 4912,
  "data": [.......],
  "paging": {
    "cursors": {
      "before": "59d266eee9bb1ac50c2364bc",
      "after": "59d768360e6439745e3cecdb"
    }
  }
}

Here total respects the filter passed and gives count of data subset based on filter. I want to extend it to generate the next_url to fetch by auto creating filter and appending in query. Is it possible? And to get count also requires another DB hit, Is there any better way to do this?