Skip to content

Commit

Permalink
Major rework in order to support embedded properties and a protobuf s…
Browse files Browse the repository at this point in the history
…erializer
  • Loading branch information
p3t committed Apr 26, 2024
1 parent 86b7693 commit 6f4b661
Showing 25 changed files with 568 additions and 198 deletions.
5 changes: 5 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -34,6 +34,11 @@ jobs:
runs-on: ubuntu-latest
permissions:
contents: read
outputs:
maven_artifactId: ${{ steps.maven_artifactId.outputs.artifactId }}
maven_groupId: ${{ steps.maven_groupId.outputs.groupId }}
maven_version: ${{ steps.maven_version.outputs.version }}
cache_key: ${{ steps.cache.outputs.cache_key }}

steps:
- uses: actions/checkout@v4
1 change: 1 addition & 0 deletions .github/workflows/publish-githubpkg.yml
Original file line number Diff line number Diff line change
@@ -8,6 +8,7 @@
name: Package & Deploy to GitHub

on:
workflow_dispatch:
release:
types: [created]

61 changes: 48 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -4,31 +4,32 @@
Library supporting cursor based paging for Spring Data Repositories.

# Introduction
Cursor based paging is an alternative to the Page/Pagerequest based paging provided by Spring.
It eliminates the need to provide an offset or a page-number which can cause a lot of load on a database in case of very large amount of records in a table.

Cursor based paging is an alternative to the page/offset based paging provided by Spring.
It eliminates the need to provide an offset or a page-number which can cause a lot of load on a database in case of very
large amount of records in a table. It also avoids the often not needed total count query per page.

# Considered Requirements
- The implementation should follow the repository concept from spring.
- Ordering by arbitrary columns should be possible.
- A filtering mechanism should be provided
- Total count of records is not part of the page response and not executed while retrieving the page
- No SQL limit/offset and no DB-cursor should be used
- State is send to the client and returned to the server for the next page
- State is/can send to the client and returned to the server for the next page (stateless behaviour)

# Quickstart / how to use it
under construction / todo
- [ ] Describe how to include dependency in maven.pom / build.gradle
- [ ] Different options to use repository interface(s)

Until done, please check the testapp sourcecode.
Please check the testapp sourcecode for latest examples and usage.

## Include the cursorpaging library in you maven pom / build.gradle

TODO (not yet published)

## Generate the JPA meta-model

The cursorpaging library is using the JPA meta-model to generate the queries.
The cursorpaging library is easier to use, when the JPA meta-model is generate to define the available attributes.
This is done by the `hibernate-jpamodelgen` annotation processor (in case you are using eclipse-link or another ORM
there should be a similar one available.

@@ -66,13 +67,20 @@ dependencies {
}
```

### Not using the JPA metamodel

The definition of the attributes should be possible as name/type combination. It might be a help to use
lombok's `@FieldNameConstants` annotation to get the attribute names as constants. Still the attributes type information
has to be added manually.

Currently, this has not really been tested, so there might be places which need adoption to support this fully.

## Register the CursorPageRepositoryFactoryBean

In order to use the repository interface an modified `JpaRepFactoryBean` is needed.
This is done via `@EnableJpaRepositories` annotation in the Spring Boot Application class.

```java

@SpringBootApplication
@EnableJpaRepositories( repositoryFactoryBeanClass = CursorPageRepositoryFactoryBean.class )
public class TestApplication {
@@ -83,7 +91,7 @@ public class TestApplication {
}
```

This implementation checks, whether the a fragment interface of the repository to be instantiated is
This implementation checks, whether there is a fragment interface of the repository to be instantiated is
a `CursorPageRepository` and if so, it will create a `CursorPageRepositoryImpl` instead of the default.

An alternative, which works without an extra factory implementation, would be to derive for each entity an additional
@@ -114,7 +122,7 @@ There are various shortcut APIs available to make the creation of the requests a

```java
public void queryData() {
final PageRequest<DataRecord> request = PageRequest.attributeAsc( DataRecord_.id );
final PageRequest<DataRecord> request = PageRequest.firstAsc( DataRecord_.id );
final Page<DataRecord> page = dataRecordRepository.findPage( request );
page.forEach( System.out::println );

@@ -137,12 +145,14 @@ public void queryData() {
}
```

### Example: Use creation time, order records descending
### Example: Multiple order defintions, sort by embedded entity attributes

```java
public void queryData() {
final PageRequest<DataRecord> request = PageRequest.create( b -> b.pageSize( 100 )
.attributeDesc( DataRecord_.createdAt ).attributeAsc( DataRecord_.id ) );
final PageRequest<DataRecord> request = PageRequest.create( b -> b.pageSize( 5 )
.desc( Attribute.path( DataRecord_.auditInfo, AuditInfo_.createdAt ) )
.asc( Attribute.path( DataRecord_.auditInfo, AuditInfo_.modifiedAt ) )
.asc( DataRecord_.id ) );

final Page<DataRecord> page = dataRecordRepository.findPage( request );
page.forEach( System.out::println );
@@ -160,7 +170,8 @@ attribute, if you want to get the records ordered e.g. by a name or creation dat
```java
public void queryData() {
final PageRequest<DataRecord> request = PageRequest.create( b -> b.pageSize( 100 )
.attributeDesc( DataRecord_.createdAt ).attributeAsc( DataRecord_.id )
.desc( Attribute.path( DataRecord_.auditInfo, AuditInfo_.createdAt ) )
.asc( DataRecord_.id )
.filter( Filter.attributeIs( DataRecord_.name, "Alpha" ) ) );

final Page<DataRecord> page = dataRecordRepository.findPage( request );
@@ -187,6 +198,30 @@ public long queryCount() {
}
```

## Passing the cursor to a client

(under construction/unfinished implementation)

The information in the page request(s) needs to be serialized and encrypted in order to be passed to a client.
Encryption is needed to avoid un-wanted insights to the implementation and to protects from injection attacks.
Within the sub-project `cursorpaging-jpa-serial` there is a serializer available generated respective page-links, e.g.
for a web-client.

```java
// TODO base64 encoding - Demo APP Controller implementation...
public String getNextLink( PageRequest<DataRecord> request ) {
final var serializer = Serializer.of( DataRecord.class, Encrypter.getInstance() )
.use( DataRecord_.name )
.use( DataRecord_.auditInfo )
.use( AuditInfo_.createdAt )
.use( AuditInfo_.modifiedAt );
final var serializedRequest = serializer.toBytes( pageRequest );

return "http://localhost:8080/datarecords/next?cursor=" + URLEncoder.encode( serializedRequest,
StandardCharsets.UTF_8 );
}
```

# Background: Concept description
## Basic idea
A Cursor is nothing elsa than a position in a list of records.
Original file line number Diff line number Diff line change
@@ -19,6 +19,7 @@ dependencyManagement {
imports {
mavenBom("org.springframework.boot:spring-boot-dependencies:3.2.4")
}

}
tasks.named<Jar>("bootJar") {
enabled = false
@@ -29,13 +30,16 @@ spotbugs {

extra["hibernate.version"] = "6.4.4.Final"


dependencies {
val lombokVersion: String by extra("1.18.30")
val junitVersion: String by extra("5.10.2")
val assertjVersion: String by extra("3.25.3")


// Load BOM for Spring Boot.
implementation(platform("org.springframework.boot:spring-boot-dependencies:3.2.3"))

implementation(platform(org.springframework.boot.gradle.plugin.SpringBootPlugin.BOM_COORDINATES))
compileOnly("org.projectlombok:lombok:${lombokVersion}")
compileOnly("org.hibernate:hibernate-jpamodelgen:6.4.4.Final")

18 changes: 11 additions & 7 deletions cursorpaging-jpa-serial/build.gradle.kts
Original file line number Diff line number Diff line change
@@ -34,22 +34,26 @@ tasks {
(this as CoreJavadocOptions).addStringOption("Xdoclint:none", "-quiet")
}
}
processResources {
// On GitHub actions the proto file seems to be copied twice
duplicatesStrategy = DuplicatesStrategy.EXCLUDE
}
}
protobuf {
protoc {
// Download from repositories
artifact = "com.google.protobuf:protoc:4.26.1"
}
generateProtoTasks {
// all().configureEach { task ->
// task.builtins {
// java {
//// option "lite"
// }
// }
// }
java{
}
}
}

sourceSets {
main {
proto {
srcDir("src/main/proto")
}
}
}
Original file line number Diff line number Diff line change
@@ -5,12 +5,7 @@
import io.vigier.cursor.Order;
import io.vigier.cursor.PageRequest;
import io.vigier.cursor.Position;
import io.vigier.cursorpaging.jpa.serial.dto.DtoCursor.DtoAttribute;
import io.vigier.cursorpaging.jpa.serial.dto.DtoCursor.DtoFilter;
import io.vigier.cursorpaging.jpa.serial.dto.DtoCursor.DtoPageRequest;
import io.vigier.cursorpaging.jpa.serial.dto.DtoCursor.DtoPosition;
import io.vigier.cursorpaging.jpa.serial.dto.DtoCursor.DtoValue;
import jakarta.persistence.metamodel.SingularAttribute;
import io.vigier.cursorpaging.jpa.serial.dto.Cursor;
import java.time.Instant;
import java.util.Collection;
import java.util.HashMap;
@@ -21,16 +16,16 @@
@RequiredArgsConstructor( staticName = "of" )
public class FromDtoMapper<E> {

private final DtoPageRequest request;
private final Map<String, SingularAttribute<E, ? extends Comparable<?>>> attributesByName = new HashMap<>();
private final Cursor.PageRequest request;
private final Map<String, Attribute> attributesByName = new HashMap<>();

public PageRequest<E> map() {
return PageRequest.<E>builder().positions( positions() ).filters( filters() )
return PageRequest.<E>builder().positions( positions() ).filters( filters() ).pageSize( request.getPageSize() )
.build();
}

public FromDtoMapper<E> using( final Collection<SingularAttribute<E, ? extends Comparable<?>>> attributes ) {
attributes.forEach( a -> attributesByName.put( a.getName(), a ) );
public FromDtoMapper<E> using( final Map<String, Attribute> attributes ) {
attributesByName.putAll( attributes );
return this;
}

@@ -42,19 +37,19 @@ private Collection<Filter> filters() {
return request.getFiltersList().stream().map( this::filterOf ).toList();
}

private Filter filterOf( final DtoFilter filter ) {
private Filter filterOf( final Cursor.Filter filter ) {
final var attribute = attributeOf( filter.getAttribute() );
return Filter.create(
b -> b.attribute( attribute ).values( valueListOf( attribute, filter.getValuesList() ) ) );
}

private List<? extends Comparable<?>> valueListOf( final io.vigier.cursor.Attribute attribute,
final List<DtoValue> valuesList ) {
private List<? extends Comparable<?>> valueListOf( final Attribute attribute,
final List<Cursor.Value> valuesList ) {
return valuesList.stream().map( v -> valueOf( attribute, v ) ).toList();
}

private Comparable<?> valueOf( final Attribute attribute, final DtoValue value ) {
if ( "".equals( value.getValue() ) ) {
private Comparable<?> valueOf( final Attribute attribute, final Cursor.Value value ) {
if ( value.getValue().isEmpty() ) {
return null;
}
if ( attribute.type().equals( Boolean.class ) ) {
@@ -78,7 +73,7 @@ private Comparable<?> valueOf( final Attribute attribute, final DtoValue value )
throw new IllegalArgumentException( "Unsupported attribute type: " + attribute.type().getName() );
}

private Position positionOf( final DtoPosition position ) {
private Position positionOf( final Cursor.Position position ) {
final var attribute = attributeOf( position.getAttribute() );

return Position.create( b -> b.attribute( attribute ).value( valueOf( attribute, position.getValue() ) )
@@ -89,12 +84,12 @@ private Position positionOf( final DtoPosition position ) {
} ) );
}

private io.vigier.cursor.Attribute attributeOf( final DtoAttribute attribute ) {
final var singularAttribute = attributesByName.get( attribute.getName() );
if ( singularAttribute == null ) {
private Attribute attributeOf( final Cursor.Attribute attribute ) {
final var cursorAttribute = attributesByName.get( attribute.getName() );
if ( cursorAttribute == null ) {
throw new IllegalArgumentException( "No attribute found for name: " + attribute.getName() );
}
return io.vigier.cursor.Attribute.of( singularAttribute );
return cursorAttribute;
}

}
Original file line number Diff line number Diff line change
@@ -1,35 +1,70 @@
package io.vigier.cursorpaging.jpa.serial;


import io.vigier.cursor.Attribute;
import io.vigier.cursor.PageRequest;
import io.vigier.cursorpaging.jpa.serial.dto.DtoCursor.DtoPageRequest;
import jakarta.persistence.metamodel.SingularAttribute;
import java.util.Collection;
import java.util.LinkedList;
import lombok.RequiredArgsConstructor;
import io.vigier.cursorpaging.jpa.serial.dto.Cursor;
import java.util.Base64;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Consumer;
import lombok.Builder;
import lombok.SneakyThrows;

@RequiredArgsConstructor( staticName = "of" )
public class Serializer<E> {
@Builder
public class Serializer {

private final Class<E> entityType;
private final Collection<SingularAttribute<E, ? extends Comparable<?>>> attributes = new LinkedList<>();
@Builder.Default
private Map<String, Attribute> attributes = new ConcurrentHashMap<>();

private final Encrypter encrypter;
@Builder.Default
private final Encrypter encrypter = Encrypter.getInstance();

public Serializer<E> use( final SingularAttribute<E, ? extends Comparable<?>> attribute ) {
attributes.add( attribute );
return this;
public static class SerializerBuilder {

public SerializerBuilder use( final Attribute attribute ) {
if ( this.attributes$value == null ) {
this.attributes$value = new ConcurrentHashMap<>();
}
this.attributes$value.put( attribute.name(), attribute );
attributes$set = true;
return this;
}
}

public static Serializer create( final Consumer<SerializerBuilder> c ) {
final var builder = builder();
c.accept( builder );
return builder.build();
}

public byte[] toBytes( final PageRequest<E> page ) {
final DtoPageRequest dtoRequest = ToDtoMapper.of( page ).map();
public static Serializer create() {
return create( b -> {
} );
}

public <E> byte[] toBytes( final PageRequest<E> page ) {
updateAttributes( page );
final Cursor.PageRequest dtoRequest = ToDtoMapper.of( page ).map();
return encrypter.encrypt( dtoRequest.toByteArray() );
}

private void updateAttributes( final PageRequest<?> page ) {
page.positions().forEach( p -> attributes.putIfAbsent( p.attribute().name(), p.attribute() ) );
page.filters().forEach( f -> attributes.putIfAbsent( f.attribute().name(), f.attribute() ) );
}

public String toBase64( final PageRequest<?> page ) {
return new String( Base64.getUrlEncoder().encode( toBytes( page ) ) );
}

@SneakyThrows
public PageRequest<E> toPageRequest( final byte[] data ) {
final var request = DtoPageRequest.parseFrom( encrypter.decrypt( data ) );
public <E> PageRequest<E> toPageRequest( final byte[] data ) {
final var request = Cursor.PageRequest.parseFrom( encrypter.decrypt( data ) );
return FromDtoMapper.<E>of( request ).using( attributes ).map();
}

public <E> PageRequest<E> toPageRequest( final String base64 ) {
return toPageRequest( Base64.getUrlDecoder().decode( base64 ) );
}
}
Loading

0 comments on commit 6f4b661

Please sign in to comment.