Skip to content

Commit

Permalink
Support correlations for data entries in JPA view (#1006)
Browse files Browse the repository at this point in the history
* feat(#1001): add correlations to jpa and build base for dataentry with correlation queries

* feat(1001): support flag for including correlations

* feat(1001): document new view

* feat(#1001): add tests and documentation for data entry correlation
- closes #1001

* chore: make view great again in test migration

* fix(#1001): Fixed test after converting table to view

* open java modules for urefire and failsafe

* made fields immutable, combined SQL scripts

---------

Co-authored-by: Michael von Bargen <[email protected]>
Co-authored-by: Tim Holzke <[email protected]>
Co-authored-by: Simon Zambrovski <[email protected]>
  • Loading branch information
4 people authored Jun 18, 2024
1 parent f007e8e commit 9769d03
Show file tree
Hide file tree
Showing 17 changed files with 346 additions and 47 deletions.
4 changes: 2 additions & 2 deletions bom/parent/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -530,7 +530,7 @@
</excludes>
<!-- Sets the VM argument line used when unit tests are run. -->
<!-- prevent the annoying ForkedBooter process from stealing window focus on Mac OS -->
<argLine>-Djava.awt.headless=true @{surefireArgLine} -XX:+StartAttachListener</argLine>
<argLine>-Djava.awt.headless=true @{surefireArgLine} -XX:+StartAttachListener --add-opens java.base/java.lang.invoke=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED</argLine>
<systemPropertyVariables>
<jgiven.report.dir>${project.build.directory}/jgiven-reports</jgiven.report.dir>
</systemPropertyVariables>
Expand All @@ -557,7 +557,7 @@
<!-- Sets the VM argument line used when integration tests are run. -->
<!-- prevent the annoying ForkedBooter process from stealing window focus on Mac OS -->
<!-- the add-opens is required for XStream -->
<argLine>-Djava.awt.headless=true @{failsafeArgLine} -XX:+StartAttachListener --add-opens java.base/java.util=ALL-UNNAMED</argLine>
<argLine>-Djava.awt.headless=true @{failsafeArgLine} -XX:+StartAttachListener --add-opens java.base/java.lang.invoke=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED</argLine>
</configuration>
</plugin>

Expand Down
25 changes: 24 additions & 1 deletion docs/reference-guide/components/view-jpa.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ configuration of this indexing process by the following configuration options:
polyflow.view.jpa:
stored-items: task, data-entry, process-instance, process-definition
payload-attribute-level-limit: 2
include-correlated-data-entries-in-data-entry-queries: false
data-entry-filters:
include: myProperty2.myOtherEmbeddedProperty3, myProperty2.myOtherEmbeddedProperty2
# exclude: myProperty
Expand All @@ -74,9 +75,14 @@ In addition, the `stored-items` property is holding a set of items to be persist
stored items are: `task`, `data-entry`, `process-instance` and `process-definition`. By setting this property, you can disable
storage of items not required by your application and save space consumption of your database. The property defaults to `data-entry`.

The `include-correlated-data-entries-in-data-entry-queries` flag controls whether a data entry query (`DataEntriesForUserQuery` or `DataEntriesQuery`) considers
the payload of correlated data entries. The data entry attributes (such as `entry_type`, `state.state`, ...) of correlated data entries are not considered.
*Note:* Only one level of correlation depth is considered here and there is no option yet to change the depth.

The attributes `data-entry-filters` and `task-filters` hold `include` / `exclude` lists of property paths which will be taken in
consideration during the search index creation.


!!! note
Please make sure you understand that the **payload enrichment** performed during collection and **indexing for search** are two different
operations. It is perfectly fine to have a large JSON payload attached to the task, but it makes no sense to make the entire payload searchable,
Expand Down Expand Up @@ -116,11 +122,12 @@ The JPA View uses several tables to store the results. These are:
* `PLF_TASK_CORRELATIONS`: table for user task correlation information
* `PLF_TASK_PAYLOAD_ATTRIBUTES`: table for user task attribute search index
* `PLF_VIEW_TASK_AND_DATA_ENTRY_PAYLOAD`: view for convenient taskWithDataEntry queries execution
* `PLF_DATA_ENTRY_PAYLOAD_ATTRIBUTES`: view for convenient data entry queries with correlations
* `TRACKING_TOKEN`: table for Axon Tracking Tokens

If you are interested in DDLs for the view, feel free to generate one using the following call of Apache Maven
`mvn -Pgenerate-sql -f view/jpa`. Currently, DDLs for the databases H2, MSSQL and PostgreSQL are generated into `target/` directory.
The DDL for the `PLF_VIEW_TASK_AND_DATA_ENTRY_PAYLOAD` cannot be auto-generated, therefore you need to use the following statement to create it:
The DDL for the `PLF_VIEW_TASK_AND_DATA_ENTRY_PAYLOAD` and `PLF_DATA_ENTRY_PAYLOAD_ATTRIBUTES` cannot be auto-generated, therefore you need to use the following statements to create them:
```
create view PLF_VIEW_TASK_AND_DATA_ENTRY_PAYLOAD as
((select pc.TASK_ID, dea.PATH, dea.VALUE
Expand All @@ -129,3 +136,19 @@ create view PLF_VIEW_TASK_AND_DATA_ENTRY_PAYLOAD as
union
select * from PLF_TASK_PAYLOAD_ATTRIBUTES);
```

```
create view PLF_VIEW_DATA_ENTRY_PAYLOAD as (
select *
from PLF_DATA_ENTRY_PAYLOAD_ATTRIBUTES
union
(select ec.OWNING_ENTRY_ID as ENTRY_ID,
ec.OWNING_ENTRY_TYPE as ENTRY_TYPE,
ep.path as PATH,
ep.value as VALUE
from PLF_DATA_ENTRY_CORRELATIONS ec
join PLF_DATA_ENTRY_PAYLOAD_ATTRIBUTES ep
on
ec.ENTRY_ID = ep.ENTRY_ID and ec.ENTRY_TYPE = ep.ENTRY_TYPE)
)
```
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ class JpaPolyflowViewDataEntryService(

val authorizedPrincipals: Set<AuthorizationPrincipal> = setOf(user(query.user.username)).plus(query.user.groups.map { group(it) })
val criteria: List<Criterion> = toCriteria(query.filters)
val specification = criteria.toDataEntrySpecification()
val specification = criteria.toDataEntrySpecification(polyflowJpaViewProperties.includeCorrelatedDataEntriesInDataEntryQueries)
val pageRequest = pageRequest(query.page, query.size, query.sort)

val page = dataEntryRepository.findAll(specification.and(isAuthorizedFor(authorizedPrincipals)), pageRequest)
Expand All @@ -87,7 +87,7 @@ class JpaPolyflowViewDataEntryService(
override fun query(query: DataEntriesQuery, metaData: MetaData): QueryResponseMessage<DataEntriesQueryResult> {

val criteria: List<Criterion> = toCriteria(query.filters)
val specification = criteria.toDataEntrySpecification()
val specification = criteria.toDataEntrySpecification(polyflowJpaViewProperties.includeCorrelatedDataEntriesInDataEntryQueries)
val pageRequest = pageRequest(query.page, query.size, query.sort)

val page = dataEntryRepository.findAll(specification, pageRequest)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,12 @@ data class PolyflowJpaViewProperties(
* Filters for the path for indexing.
*/
@NestedConfigurationProperty
private val taskFilters: PayloadAttributeFilterPaths = PayloadAttributeFilterPaths()
private val taskFilters: PayloadAttributeFilterPaths = PayloadAttributeFilterPaths(),

/**
* Controls if DataEntryQueries should consider the payload attributes of correlated data entries. Defaults to "false".
*/
val includeCorrelatedDataEntriesInDataEntryQueries: Boolean = false

) {
/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import io.holunda.polyflow.view.Task
import io.holunda.polyflow.view.filter.*
import io.holunda.polyflow.view.jpa.data.DataEntryEntity
import io.holunda.polyflow.view.jpa.data.DataEntryRepository.Companion.hasDataEntryPayloadAttribute
import io.holunda.polyflow.view.jpa.data.DataEntryRepository.Companion.hasDataEntryPayloadAttributeIncludingCorrelations
import io.holunda.polyflow.view.jpa.data.DataEntryRepository.Companion.hasEntryId
import io.holunda.polyflow.view.jpa.data.DataEntryRepository.Companion.hasEntryType
import io.holunda.polyflow.view.jpa.data.DataEntryRepository.Companion.hasProcessingType
Expand All @@ -24,8 +25,8 @@ import io.holunda.polyflow.view.jpa.task.TaskRepository.Companion.hasFollowUpDat
import io.holunda.polyflow.view.jpa.task.TaskRepository.Companion.hasFollowUpDateBefore
import io.holunda.polyflow.view.jpa.task.TaskRepository.Companion.hasPriority
import io.holunda.polyflow.view.jpa.task.TaskRepository.Companion.hasProcessName
import io.holunda.polyflow.view.jpa.task.TaskRepository.Companion.hasTaskPayloadAttribute
import io.holunda.polyflow.view.jpa.task.TaskRepository.Companion.hasTaskOrDataEntryPayloadAttribute
import io.holunda.polyflow.view.jpa.task.TaskRepository.Companion.hasTaskPayloadAttribute
import io.holunda.polyflow.view.jpa.task.TaskRepository.Companion.likeBusinessKey
import io.holunda.polyflow.view.jpa.task.TaskRepository.Companion.likeDescription
import io.holunda.polyflow.view.jpa.task.TaskRepository.Companion.likeName
Expand All @@ -43,10 +44,10 @@ import java.time.Instant
/**
* Creates a JPQL specification out of predicate wrapper.
*/
fun List<Criterion>.toDataEntrySpecification(): Specification<DataEntryEntity> {
fun List<Criterion>.toDataEntrySpecification(includeCorrelatedDataEntries: Boolean): Specification<DataEntryEntity> {

val attributeSpec = toDataEntryAttributeSpecification()
val payloadSpec = toDataEntryPayloadSpecification()
val payloadSpec = toDataEntryPayloadSpecification(includeCorrelatedDataEntries)

return where(attributeSpec).and(payloadSpec)
}
Expand Down Expand Up @@ -187,11 +188,11 @@ internal fun List<Criterion>.toDataEntryAttributeSpecification(): Specification<
/**
* Specification on payload.
*/
internal fun List<Criterion>.toDataEntryPayloadSpecification(): Specification<DataEntryEntity> {
internal fun List<Criterion>.toDataEntryPayloadSpecification(includeCorrelatedDataEntries: Boolean): Specification<DataEntryEntity> {
val relevant = this.filterIsInstance<Criterion.PayloadEntryCriterion>()
// compose criteria with same name with OR and criteria with different names with AND
val relevantByName = relevant.groupBy { it.name }
val orComposedByName = relevantByName.map { (_, criteria) -> criteria.toOrDataEntrySpecification() }
val orComposedByName = relevantByName.map { (_, criteria) -> criteria.toOrDataEntrySpecification(includeCorrelatedDataEntries) }

return composeAnd(orComposedByName)
}
Expand Down Expand Up @@ -325,12 +326,16 @@ internal fun Criterion.DataEntryCriterion.toTaskCorrelatedDataEntrySpecification
* Creates JPA Specification for query of payload attributes based on JSON paths. All criteria must have the same path
* and will be composed by the logical OR operator.
*/
internal fun List<Criterion.PayloadEntryCriterion>.toOrDataEntrySpecification(): Specification<DataEntryEntity> {
internal fun List<Criterion.PayloadEntryCriterion>.toOrDataEntrySpecification(includeCorrelatedDataEntries: Boolean): Specification<DataEntryEntity> {
require(this.isNotEmpty()) { "List of criteria must not be empty." }
require(this.all { it.operator == EQUALS }) { "JPA View currently supports only equals as operator for filtering of payload attributes." }
require(this.distinctBy { it.name }.size == 1) { "All criteria must have the same path." }

return hasDataEntryPayloadAttribute(this.first().name, this.map { it.value })
return if (includeCorrelatedDataEntries) {
hasDataEntryPayloadAttributeIncludingCorrelations(this.first().name, this.map { it.value })
} else {
hasDataEntryPayloadAttribute(this.first().name, this.map { it.value })
}
}


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import io.holunda.polyflow.view.jpa.auth.AuthorizationPrincipal.Companion.user
import io.holunda.polyflow.view.jpa.auth.AuthorizationPrincipalType.GROUP
import io.holunda.polyflow.view.jpa.auth.AuthorizationPrincipalType.USER
import io.holunda.polyflow.view.jpa.payload.PayloadAttribute
import org.camunda.bpm.engine.variable.Variables

/**
* Converts the entity into API type.
Expand All @@ -30,6 +31,7 @@ fun DataEntryEntity.toDataEntry(objectMapper: ObjectMapper) =
authorizedUsers = this.authorizedPrincipals.asUsernames(),
authorizedGroups = this.authorizedPrincipals.asGroupnames(),
payload = this.payload.toPayloadVariableMap(objectMapper),
correlations = Variables.fromMap(this.correlations.associate { it.entryType to it.entryId })
)

/**
Expand Down Expand Up @@ -86,6 +88,7 @@ fun DataEntryCreatedEvent.toEntity(objectMapper: ObjectMapper, revisionValue: Re
},
authorizedPrincipals = AuthorizationChange.applyUserAuthorization(mutableSetOf(), this.authorizations).map { user(it).toString() }
.plus(AuthorizationChange.applyGroupAuthorization(mutableSetOf(), this.authorizations).map { group(it).toString() }).toMutableSet(),
correlations = this.correlations.toMutableMap().map { entry -> DataEntryId(entryType = entry.key, entryId = entry.value.toString()) }.toMutableSet()
).apply {
this.protocol = this.protocol.addModification(this, this@toEntity.createModification, this@toEntity.state)
}
Expand Down Expand Up @@ -188,4 +191,4 @@ fun DataEntryDeletedEvent.toEntity(
}
}.apply {
this.protocol = this.protocol.addModification(this, this@toEntity.deleteModification, this@toEntity.state)
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import io.holunda.polyflow.view.jpa.payload.PayloadAttribute
import jakarta.persistence.*
import org.hibernate.annotations.Fetch
import org.hibernate.annotations.FetchMode
import org.hibernate.annotations.Immutable
import java.time.Instant

/**
Expand Down Expand Up @@ -62,6 +63,22 @@ class DataEntryEntity(
@OneToMany(mappedBy = "dataEntry", orphanRemoval = true, cascade = [CascadeType.ALL], fetch = FetchType.EAGER)
@Fetch(FetchMode.SELECT)
var protocol: MutableList<ProtocolElement> = mutableListOf(),
@ElementCollection(fetch = FetchType.EAGER)
@CollectionTable(
name = "PLF_DATA_ENTRY_CORRELATIONS",
joinColumns = [
JoinColumn(name = "OWNING_ENTRY_TYPE", referencedColumnName = "ENTRY_TYPE"),
JoinColumn(name= "OWNING_ENTRY_ID", referencedColumnName = "ENTRY_ID"),
]
)
var correlations: MutableSet<DataEntryId> = mutableSetOf(),
@Immutable
@OneToMany(fetch = FetchType.LAZY)
@JoinColumns(
JoinColumn(name = "ENTRY_TYPE", referencedColumnName = "ENTRY_TYPE", insertable = false, updatable = false),
JoinColumn(name = "ENTRY_ID", referencedColumnName = "ENTRY_ID", insertable = false, updatable = false)
)
var payloadAndCorrelatedPayloadAttributes: MutableSet<DataEntryPayloadAttributeEntity> = mutableSetOf(),

@Column(name = "PAYLOAD")
@Lob
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
package io.holunda.polyflow.view.jpa.data

import jakarta.persistence.EmbeddedId
import jakarta.persistence.Entity
import jakarta.persistence.Table
import org.hibernate.annotations.Immutable

/**
* Entity that holds the combined payload attributes of the correlated DataEntries.
*/
@Entity
@Immutable
@Table(name = "PLF_VIEW_DATA_ENTRY_PAYLOAD")
class DataEntryPayloadAttributeEntity(
@EmbeddedId
var id: DataEntryPayloadAttributeEntityId,
) {
constructor(entryType: String, entryId: String, path: String, value: String) : this(
DataEntryPayloadAttributeEntityId(
entryType = entryType,
entryId = entryId,
path = path,
value = value
)
)

override fun toString(): String = "DataEntryPayloadAttributeEntity(id=$id)"
}

Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
package io.holunda.polyflow.view.jpa.data

import jakarta.persistence.Column
import jakarta.persistence.Embeddable
import org.hibernate.annotations.Immutable
import java.io.Serializable

/**
* Id class that holds the combined payload attributes of the correlated DataEntries.
*/
@Embeddable
@Immutable
class DataEntryPayloadAttributeEntityId(
@Column(name = "ENTRY_TYPE", length = 64, nullable = false, updatable = false, insertable = false)
var entryType: String,
@Column(name = "ENTRY_ID", length = 64, nullable = false, updatable = false, insertable = false)
var entryId: String,
@Column(name = "PATH", nullable = false, updatable = false, insertable = false)
var path: String,
@Column(name = "VALUE", nullable = false, updatable = false, insertable = false)
var value: String
) : Serializable {

override fun equals(other: Any?): Boolean {
if (this === other) return true
if (javaClass != other?.javaClass) return false

other as DataEntryPayloadAttributeEntityId

if (entryType != other.entryType) return false
if (entryId != other.entryId) return false
if (path != other.path) return false
if (value != other.value) return false

return true
}

override fun hashCode(): Int {
var result = entryType.hashCode()
result = 31 * result + entryId.hashCode()
result = 31 * result + path.hashCode()
result = 31 * result + value.hashCode()
return result
}

override fun toString(): String {
return "DataEntryPayloadAttributeEntityId(entryType='$entryType', entryId='$entryId', path='$path', value='$value')"
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,31 @@ interface DataEntryRepository : CrudRepository<DataEntryEntity, DataEntryId>, Jp
builder.and(pathEquals, valueAnyOf)
}

/**
* Specification for checking the payload attributes of a data entry (including payload attributes from correlated data entries).
* If multiple values are given, one of them must match. payload.name = ? AND (payload.value = ? OR payload.value = ? OR ...)
*/
fun hasDataEntryPayloadAttributeIncludingCorrelations(name: String, values: List<String>): Specification<DataEntryEntity> =
Specification { dataEntry, query, builder ->
query.distinct(true)
val join = dataEntry.join<DataEntryEntity, Set<DataEntryPayloadAttributeEntity>>(DataEntryEntity::payloadAndCorrelatedPayloadAttributes.name)
val pathEquals = builder.equal(
join.get<DataEntryPayloadAttributeEntityId>(DataEntryPayloadAttributeEntity::id.name)
.get<String>(DataEntryPayloadAttributeEntityId::path.name),
name
)

val valueAnyOf = values.map {
builder.equal(
join.get<DataEntryPayloadAttributeEntityId>(DataEntryPayloadAttributeEntity::id.name)
.get<String>(DataEntryPayloadAttributeEntityId::value.name),
it
)
}.let { builder.or(*it.toTypedArray()) }

builder.and(pathEquals, valueAnyOf)
}

/**
* Specification for checking authorization of multiple principals.
*/
Expand Down
Loading

0 comments on commit 9769d03

Please sign in to comment.