Analyse how to implement pagination and implement

[bergos]

As described in #123 pagination should be implemented. It should be analysed how to implement it on the backend and fronted.

Accept:
There is a concept for pagination.

[bergos]

Maybe the TREE ontology could be used to create triples for the client.

[cristianvasquez]

Test data here: https://ld.zazuko.com/test/people/

[cristianvasquez]

I've tested a very straightforward solution: update the container query to produce some additional (hydra) controls + hydra:Collection.

I guess the renderer needs to handle collections as a list when they are marked as such

Example of the data: https://ld.zazuko.com/test/people/

[l00mi]

Open questions to me:

• Do we page triples per-se, or do we look at triples with the same Subject for pagination?
• How to define starting which amount entries we do pagination?
• How is sorting handled if we do pagination?
• How can we jump to a specific page in pagination?
• Is it possible to implement "continuous scroll" (like zack-search) with the underlying pagination mechanism?

[cristianvasquez]

Hi!

Thank you for the good questions

What would help is to define some example collections, for example, 'datasets' or 'cubes,' and iterate with them. How typical is this use case? What attributes to add? all of them? Some common ones? from a collection description?

[cristianvasquez]

Do we page triples per-se, or do we look at triples with the same Subject for pagination?

This example is paginating the triples, only showing a triple + label

?something a ?class + a label.

Which is configured in the containerGraphQuery in a modified version of the trifid-handler-SPARQL middleware, which triggers each time the URL ends with a slash '/'

Like: https://ld.zazuko.com/test/people/?page=4

    PREFIX schema: <http://schema.org/>
    PREFIX hydra: <http://www.w3.org/ns/hydra/core#>
    
    CONSTRUCT { 
      <${iri}> a hydra:Collection ;
         hydra:member ?s ;
         ${prev}
         hydra:first <${iri}> ;
         hydra:next <${iri}?page=${currentPage + 1}> .
      ?s a ?o. 
      ?s schema:name ?label.
    } WHERE { 
      ?s a ?o. 
      OPTIONAL {?s schema:name ?label. } .
      FILTER REGEX(STR(?s), "${iri}") .
      
    } limit ${pageSize} offset ${currentPage * pageSize}
    

I don't know if this is the best approach for the use case, but today this works out of the box.

To add more attributes for the items in the collection, one can update the underlying SPARQL in the configuration; for example, to add 'schema:Place', one needs to add an OPTIONAL in the where.

WHERE { 
      ?s a ?o. 
      OPTIONAL {?s schema:Place ?o. } .
      OPTIONAL {?s schema:name ?label. } .
      FILTER REGEX(STR(?s), "${iri}") .
    } 

[cristianvasquez]

How to define starting which amount entries we do pagination?

The example uses two parameters: currentpage and pageSize

[cristianvasquez]

How is sorting handled if we do pagination?

I don't know; it's difficult to tell without knowing some things.

For example:

• What attributes should a collection have while paginating?
  • Should this live in the configuration of SPARQL-Handler?
  • Should this live in the data, in a SHACL, or similar?
  • Most attributes by default?
• Should different entities, such as /people/ or /projects/ show different attributes?

But a priori, I think it could be done through a parameter.

``?sort=schema:name?order=ascending`

[cristianvasquez]

How can we jump to a specific page in pagination?

In the current example, it's done using the attribute 'page'

https://ld.zazuko.com/test/people/?page=3, which is the resource that describes page 3 of the paginated collection

[cristianvasquez]

Is it possible to implement "continuous scroll" (like zack-search) with the underlying pagination mechanism?

I think in a standalone application, this is straightforward,

but I'm unsure of how this would work on top of the trifid templating mechanism (#100)

How do you think this could work @bergos, @ludovicm67

[l00mi]

@cristianvasquez Ah, now I understand your example. I think here is a high-level missunderstanding. It is not about specific resource to paginate, but much more to paginate any kind of triples in case there are e.g. 1000+ triples on a resource. This is also only an HTML UI problem. All other content negotations, need still to send all triples.

[cristianvasquez]

I see,

If one wants to paginate the triples in the UI then one could use parameters currentpage and pageSize for the web component.

One doesn't paginate triples but rows.

Consider this toy example:

If one is not in 'compact mode', without grouping parameters or values, this corresponds to paginating the triples.

This will be simpler to implement than the collections :)

[l00mi]

But it will still load all triples of the resourcE?

[cristianvasquez]

Yes, now it's working like this:

• Intercept the quads from previous middlewares (which can be in any format)
• transform them into something the renderer understands (the quads)
• Group the quads by value and object to produce the rows
• Generate the HTML to send to the client.

[cristianvasquez]

To show quads as they arrive and provide a low-latency experience, we need to structure our trifid instances differently.

The middlewares need to return quads as rdf-js quads, (no turtle, no json-ld). And something like a single-page application that consumes this on top and reacts according to the quads that arrive.

[ludovicm67]

The way to solve this will be to:

• create a new endpoint that is dedicated to serve requested resources (paginated) ; like a standard API endpoint, let's say at /api/entity for example
• in the template, render the first page server-side like we are doing now (this offers results that can be fetched without JavaScript enabled), and dynamically loads the other pages from that template in background and complete the rendered page

[l00mi]

@cristianvasquez the solution with pageSize as mentioned above is already implemented as of now?

[cristianvasquez]

The way to solve this will be to:

* create a new endpoint that is dedicated to serve requested resources (paginated) ; like a standard API endpoint, let's say at `/api/entity` for example

* in the template, render the first page server-side like we are doing now (this offers results that can be fetched without JavaScript enabled), and dynamically loads the other pages from that template in background and complete the rendered page

As I understood recently, this is not about returning paginated resources but triples for one entity.

[cristianvasquez]

@cristianvasquez the solution with pageSize as mentioned above is already implemented as of now?

No, I've just understood the problem, I think we need an example.

[cristianvasquez]

I've added some entities with lots of triples, to see how this behaves

One entity with 500 distinct triples

Fair enough.

https://ld.zazuko.com/test/many/1

One entity with 1000 distinct triples

slow, still aceptable.

https://ld.zazuko.com/test/many/2

One entity with 5000 distinct triples

too slow.

https://ld.zazuko.com/test/many/3

[MartinaElsawy]

Discussion cleared several points, level of implementation is still ongoing