Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cleanup money #3535

Merged
merged 60 commits into from
Oct 2, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
e688d98
restructures ResultType and decouples printing into ResultPrinters:
awildturtok Jul 31, 2024
84ad761
fixes null resultType in Test
awildturtok Jul 31, 2024
18a0bd8
fix Printer signature
awildturtok Aug 1, 2024
061c9ac
minor cleanup of naming
awildturtok Aug 1, 2024
92ba8d0
fixes ResultTypeTest, brings back localized mapping of Resolution and…
awildturtok Aug 1, 2024
bab8bab
adds missing ResultType to satisfy default constructor
awildturtok Aug 1, 2024
a3a0a42
unwrap ListPrinters in Arrow and Excel renderer
awildturtok Aug 1, 2024
56950fd
fixes type of MappableSingleColumnSelect, and fixing type selection f…
awildturtok Aug 7, 2024
7645cf1
Merge branch 'develop' into feature/decouple-printing-from-ResultTyoe
awildturtok Aug 7, 2024
f152160
fix TypedSelectDummy holder
awildturtok Aug 14, 2024
d2a9986
adds ListPrinter for ConceptColumnSelect
awildturtok Aug 14, 2024
31de3f2
fixes DateRange header and SecondaryIdResultInfo
awildturtok Aug 14, 2024
8b2ee30
Early binds PrintSettings to avoid passing it into Printers at every …
awildturtok Aug 20, 2024
0eecef6
fixes not using Printer object for ConceptIds. Also migrate all print…
awildturtok Aug 20, 2024
cb0c47e
fixes StoredQueriesProcessorTest
awildturtok Aug 20, 2024
3bd7407
cleanup superfluous c'tor
awildturtok Aug 20, 2024
d4692bf
cleanup superfluous PrintSettings c'tor
awildturtok Aug 20, 2024
e95bb29
Cleanup Localized Printers
awildturtok Aug 22, 2024
6d03233
Code Style Issues and cleanup
awildturtok Aug 22, 2024
d8cceb5
Merge branch 'develop' into feature/decouple-printing-from-ResultTyoe
awildturtok Aug 22, 2024
15237c9
reworks Money to no longer be "cent" based and instead just use it as…
awildturtok Aug 22, 2024
f6ff223
wip, trying to fix money filler
awildturtok Aug 22, 2024
7d53595
Reworks Printers:
awildturtok Aug 28, 2024
39608be
use printers in ExcelRenderer
awildturtok Aug 28, 2024
3a3f81c
Merge remote-tracking branch 'origin/develop' into feature/cleanup-MONEY
awildturtok Aug 29, 2024
a39c4f4
fixes MONEY resultType in ArrowResultGenerationTest.java
awildturtok Aug 29, 2024
d92b0a9
fixes MONEY resultType in ParquetResultGenerationTest.java
awildturtok Aug 29, 2024
4fdb5c2
cleanup ResultTestUtil
awildturtok Aug 29, 2024
84549da
adds missing Dataset
awildturtok Aug 29, 2024
ab633f9
cleanup DateRange vector Filler, annd fixes potential NPEs
awildturtok Aug 29, 2024
a769b16
cleanup usage of ChainingPrinter
awildturtok Aug 29, 2024
ce5f407
delay binding of PrintSettings to createPrinter. Remove PrintSettings…
awildturtok Aug 29, 2024
b97fc49
use DECIMAL in Arrow and Parquet for MONEY
awildturtok Aug 29, 2024
bbf9cb7
Code Style issues from Code Review
awildturtok Aug 29, 2024
e95f8f7
fix test comparison of BigDecmial
awildturtok Aug 29, 2024
4239f39
fix handling of Money in ArrowResultGenerationTest separately to Parq…
awildturtok Aug 29, 2024
8e9ed1e
cleanup structure of PrinterFactory.
awildturtok Sep 2, 2024
e0904a8
fix excel money cell style and test render format
thoniTUB Sep 3, 2024
88ce4fb
fix ResultTypeTest
thoniTUB Sep 3, 2024
b939bef
NumberFilterConverter now also fixes FilterValue
awildturtok Sep 3, 2024
ca94db8
removes problematic MONEY NUMBER_RANGE test
awildturtok Sep 3, 2024
a82a528
Merge branch 'develop' into feature/cleanup-MONEY
awildturtok Sep 3, 2024
d5ae999
deletes problematic Aggregator
awildturtok Sep 3, 2024
9ee0f68
introduce types for Printers
awildturtok Sep 3, 2024
b90891a
pull PrinterFactory out of PrintSettings, pass manually into createPr…
awildturtok Sep 3, 2024
23a6b14
removes deleted Select
awildturtok Sep 3, 2024
38e7833
Code Style issues
awildturtok Sep 3, 2024
03434fb
pull default for decimalShift from CurrencyConfig
awildturtok Sep 4, 2024
672cb42
fixes injection and deserialization test
awildturtok Sep 4, 2024
f8ffea9
semantics are immutable
awildturtok Sep 4, 2024
8bdfdd1
fixes missing null-Check
awildturtok Sep 4, 2024
6358f64
try fixing Money handling for preview
awildturtok Sep 4, 2024
65e701c
Revert "try fixing Money handling for preview"
awildturtok Sep 5, 2024
e9203fa
revert using Decimal for Arrow/Parquet: causes issues with jsArrow an…
awildturtok Sep 5, 2024
88224e5
fixes MoneyPrinter being fixed to BigDecimal causing issues with dese…
awildturtok Sep 5, 2024
efa20ba
skip rescaling of money values
awildturtok Sep 5, 2024
c65d0fe
fixes rescaling for Money
awildturtok Sep 16, 2024
484acf2
simplify printer usage
awildturtok Sep 16, 2024
32d30d1
Merge branch 'develop' into feature/cleanup-MONEY
awildturtok Sep 17, 2024
256b651
Merge branch 'develop' into feature/cleanup-MONEY
awildturtok Oct 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 69 additions & 28 deletions backend/src/main/java/com/bakdata/conquery/ResultHeaders.java
Original file line number Diff line number Diff line change
Expand Up @@ -9,64 +9,105 @@
import com.bakdata.conquery.models.query.PrintSettings;
import com.bakdata.conquery.models.query.resultinfo.FixedLabelResultInfo;
import com.bakdata.conquery.models.query.resultinfo.ResultInfo;
import com.bakdata.conquery.models.query.resultinfo.printers.ResultPrinters;
import com.bakdata.conquery.models.query.resultinfo.printers.Printer;
import com.bakdata.conquery.models.query.resultinfo.printers.PrinterFactory;
import com.bakdata.conquery.models.query.resultinfo.printers.common.LocalizedEnumPrinter;
import com.bakdata.conquery.models.types.ResultType;
import com.bakdata.conquery.models.types.SemanticType;
import lombok.experimental.UtilityClass;

@UtilityClass
public class ResultHeaders {
public static ResultInfo datesInfo(PrintSettings settings) {
final String label = C10nCache.getLocalized(ResultHeadersC10n.class, settings.getLocale()).dates();
public static ResultInfo datesInfo() {

final ResultType.ListT<Object> type = new ResultType.ListT<>(ResultType.Primitive.DATE_RANGE);

return new FixedLabelResultInfo(label, label, type, Set.of(new SemanticType.EventDateT()), settings, ResultPrinters.printerFor(type, settings));
return new FixedLabelResultInfo(type, Set.of(new SemanticType.EventDateT())){
@Override
public String userColumnName(PrintSettings printSettings) {
return C10nCache.getLocalized(ResultHeadersC10n.class, printSettings.getLocale()).dates();
}
};
}

public static ResultInfo historyDatesInfo(PrintSettings settings) {
final String label = C10nCache.getLocalized(ResultHeadersC10n.class, settings.getLocale()).dates();
public static ResultInfo historyDatesInfo() {

final ResultType.ListT<Object> type = new ResultType.ListT<>(ResultType.Primitive.DATE_RANGE);

return new FixedLabelResultInfo(label, label, type, Set.of(new SemanticType.EventDateT(), new SemanticType.GroupT()), settings, ResultPrinters.printerFor(type, settings));
return new FixedLabelResultInfo(type, Set.of(new SemanticType.EventDateT(), new SemanticType.GroupT())) {
@Override
public String userColumnName(PrintSettings printSettings) {
return C10nCache.getLocalized(ResultHeadersC10n.class, printSettings.getLocale()).dates();
}
};
}

public static ResultInfo sourceInfo(PrintSettings settings) {
final String label = C10nCache.getLocalized(ResultHeadersC10n.class, settings.getLocale()).source();

return new FixedLabelResultInfo(label, label, ResultType.Primitive.STRING, Set.of(new SemanticType.SourcesT(), new SemanticType.CategoricalT(), new SemanticType.GroupT()), settings, ResultPrinters.printerFor(ResultType.Primitive.STRING, settings));
public static ResultInfo sourceInfo() {
return new FixedLabelResultInfo(ResultType.Primitive.STRING, Set.of(new SemanticType.SourcesT(), new SemanticType.CategoricalT(), new SemanticType.GroupT())) {
@Override
public String userColumnName(PrintSettings printSettings) {
return C10nCache.getLocalized(ResultHeadersC10n.class, printSettings.getLocale()).source();
}
};
}

public static ResultInfo formContextInfo(PrintSettings settings) {
final String label = C10nCache.getLocalized(ResultHeadersC10n.class, settings.getLocale()).index();
public static ResultInfo formContextInfo() {

return new FixedLabelResultInfo(label, label, ResultType.Primitive.INTEGER, Set.of(), settings, ResultPrinters.printerFor(ResultType.Primitive.INTEGER, settings));
return new FixedLabelResultInfo(ResultType.Primitive.INTEGER, Set.of()) {
@Override
public String userColumnName(PrintSettings printSettings) {
return C10nCache.getLocalized(ResultHeadersC10n.class, printSettings.getLocale()).index();
}
};
}

public static ResultInfo formDateRangeInfo(PrintSettings settings) {
final String label = C10nCache.getLocalized(ResultHeadersC10n.class, settings.getLocale())
.dateRange();
public static ResultInfo formDateRangeInfo() {

return new FixedLabelResultInfo(label, label, ResultType.Primitive.DATE_RANGE, Set.of(), settings, ResultPrinters.printerFor(ResultType.Primitive.DATE_RANGE, settings));
return new FixedLabelResultInfo(ResultType.Primitive.DATE_RANGE, Set.of()) {
@Override
public String userColumnName(PrintSettings printSettings) {
return C10nCache.getLocalized(ResultHeadersC10n.class, printSettings.getLocale()).dateRange();
}
};
}

public static ResultInfo formResolutionInfo(PrintSettings settings) {
final String label = C10nCache.getLocalized(ResultHeadersC10n.class, settings.getLocale()).resolution();
public static ResultInfo formResolutionInfo() {

return new FixedLabelResultInfo(ResultType.Primitive.STRING, Set.of()) {
@Override
public Printer createPrinter(PrinterFactory printerFactory, PrintSettings printSettings) {
return new LocalizedEnumPrinter<>(printSettings, Resolution.class);
}

return new FixedLabelResultInfo(label, label, ResultType.Primitive.STRING, Set.of(), settings, new ResultPrinters.LocalizedEnumPrinter<>(settings, Resolution.class));
@Override
public String userColumnName(PrintSettings printSettings) {
return C10nCache.getLocalized(ResultHeadersC10n.class, printSettings.getLocale()).resolution();
}
};
}

public static ResultInfo formEventDateInfo(PrintSettings settings) {
final String label = C10nCache.getLocalized(ResultHeadersC10n.class, settings.getLocale())
.eventDate();
public static ResultInfo formEventDateInfo() {

return new FixedLabelResultInfo(label, label, ResultType.Primitive.DATE, Set.of(), settings, ResultPrinters.printerFor(ResultType.Primitive.DATE, settings));
return new FixedLabelResultInfo(ResultType.Primitive.DATE, Set.of()) {
@Override
public String userColumnName(PrintSettings printSettings) {
return C10nCache.getLocalized(ResultHeadersC10n.class, printSettings.getLocale()).eventDate();
}
};
}

public static ResultInfo formObservationScopeInfo(PrintSettings settings) {
final String label = C10nCache.getLocalized(ResultHeadersC10n.class, settings.getLocale()).observationScope();
public static ResultInfo formObservationScopeInfo() {

return new FixedLabelResultInfo(ResultType.Primitive.STRING, Set.of()) {
@Override
public Printer createPrinter(PrinterFactory printerFactory, PrintSettings printSettings) {
return new LocalizedEnumPrinter<>(printSettings, FeatureGroup.class);
}

return new FixedLabelResultInfo(label, label, ResultType.Primitive.STRING, Set.of(), settings, new ResultPrinters.LocalizedEnumPrinter<>(settings, FeatureGroup.class));
@Override
public String userColumnName(PrintSettings printSettings) {
return C10nCache.getLocalized(ResultHeadersC10n.class, printSettings.getLocale()).observationScope();
}
};
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@
import com.bakdata.conquery.models.query.queryplan.DateAggregationAction;
import com.bakdata.conquery.models.query.resultinfo.ResultInfo;
import com.bakdata.conquery.models.query.resultinfo.UniqueNamer;
import com.bakdata.conquery.models.query.resultinfo.printers.JavaResultPrinters;
import com.bakdata.conquery.models.query.statistics.ResultStatistics;
import com.bakdata.conquery.models.query.visitor.QueryVisitor;
import com.bakdata.conquery.models.types.SemanticType;
Expand Down Expand Up @@ -585,15 +586,23 @@ public ResultStatistics getResultStatistics(SingleTableResult managedQuery) {
new PrintSettings(true, locale, managedQuery.getNamespace(), config, null, null, decimalFormat, integerFormat);
final UniqueNamer uniqueNamer = new UniqueNamer(printSettings);

final List<ResultInfo> resultInfos = managedQuery.getResultInfos(printSettings);
final List<ResultInfo> resultInfos = managedQuery.getResultInfos();

final Optional<ResultInfo>
dateInfo =
resultInfos.stream().filter(info -> info.getSemantics().contains(new SemanticType.EventDateT())).findFirst();

final Optional<Integer> dateIndex = dateInfo.map(resultInfos::indexOf);

return ResultStatistics.collectResultStatistics(managedQuery, resultInfos, dateInfo, dateIndex, printSettings, uniqueNamer, config);
return ResultStatistics.collectResultStatistics(managedQuery,
resultInfos,
dateInfo,
dateIndex,
printSettings,
uniqueNamer,
config,
new JavaResultPrinters()
);
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
import com.bakdata.conquery.io.jackson.View;
import com.bakdata.conquery.models.identifiable.ids.specific.ManagedExecutionId;
import com.bakdata.conquery.models.query.DateAggregationMode;
import com.bakdata.conquery.models.query.PrintSettings;
import com.bakdata.conquery.models.query.QueryPlanContext;
import com.bakdata.conquery.models.query.QueryResolveContext;
import com.bakdata.conquery.models.query.Visitable;
Expand Down Expand Up @@ -93,17 +92,17 @@ public void collectRequiredQueries(Set<ManagedExecutionId> requiredQueries) {
}

@Override
public List<ResultInfo> getResultInfos(PrintSettings printSettings) {
public List<ResultInfo> getResultInfos() {
awildturtok marked this conversation as resolved.
Show resolved Hide resolved
final List<ResultInfo> resultInfos = new ArrayList<>();
ResultInfo dateInfo = ResultHeaders.datesInfo(printSettings);
ResultInfo dateInfo = ResultHeaders.datesInfo();

if(getResolvedDateAggregationMode() != DateAggregationMode.NONE){
// Add one DateInfo for the whole Query
resultInfos.add(0, dateInfo);
}
int lastIndex = resultInfos.size();

childQueries.forEach(q -> resultInfos.addAll(q.getResultInfos(printSettings)));
childQueries.forEach(q -> resultInfos.addAll(q.getResultInfos()));

if(!resultInfos.isEmpty()) {
// Remove DateInfo from each childQuery
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
import com.bakdata.conquery.io.cps.CPSBase;
import com.bakdata.conquery.io.cps.CPSType;
import com.bakdata.conquery.models.identifiable.ids.specific.ManagedExecutionId;
import com.bakdata.conquery.models.query.PrintSettings;
import com.bakdata.conquery.models.query.QueryExecutionContext;
import com.bakdata.conquery.models.query.QueryPlanContext;
import com.bakdata.conquery.models.query.QueryResolveContext;
Expand Down Expand Up @@ -79,7 +78,7 @@ public final Set<ManagedExecutionId> collectRequiredQueries() {
public abstract void collectRequiredQueries(Set<ManagedExecutionId> requiredQueries) ;

@JsonIgnore
public abstract List<ResultInfo> getResultInfos(PrintSettings settings);
public abstract List<ResultInfo> getResultInfos();

public void visit(Consumer<Visitable> visitor) {
visitor.accept(this);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@

import com.bakdata.conquery.io.cps.CPSType;
import com.bakdata.conquery.models.identifiable.ids.specific.ManagedExecutionId;
import com.bakdata.conquery.models.query.PrintSettings;
import com.bakdata.conquery.models.query.QueryPlanContext;
import com.bakdata.conquery.models.query.QueryResolveContext;
import com.bakdata.conquery.models.query.queryplan.ConceptQueryPlan;
Expand All @@ -33,7 +32,7 @@ public void collectRequiredQueries(Set<ManagedExecutionId> requiredQueries) {
}

@Override
public List<ResultInfo> getResultInfos(PrintSettings settings) {
public List<ResultInfo> getResultInfos() {
return Collections.emptyList();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@
import com.bakdata.conquery.io.jackson.View;
import com.bakdata.conquery.models.identifiable.ids.specific.ManagedExecutionId;
import com.bakdata.conquery.models.query.DateAggregationMode;
import com.bakdata.conquery.models.query.PrintSettings;
import com.bakdata.conquery.models.query.QueryExecutionContext;
import com.bakdata.conquery.models.query.QueryPlanContext;
import com.bakdata.conquery.models.query.QueryResolveContext;
Expand Down Expand Up @@ -80,16 +79,16 @@ public void resolve(QueryResolveContext context) {
}

@Override
public List<ResultInfo> getResultInfos(PrintSettings printSettings) {
public List<ResultInfo> getResultInfos() {
Preconditions.checkNotNull(resolvedDateAggregationMode);

final List<ResultInfo> resultInfos = new ArrayList<>();

if (resolvedDateAggregationMode != DateAggregationMode.NONE) {
resultInfos.add(ResultHeaders.datesInfo(printSettings));
resultInfos.add(ResultHeaders.datesInfo());
}

resultInfos.addAll(root.getResultInfos(printSettings));
resultInfos.addAll(root.getResultInfos());

return resultInfos;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@
import com.bakdata.conquery.models.execution.ManagedExecution;
import com.bakdata.conquery.models.identifiable.ids.specific.ManagedExecutionId;
import com.bakdata.conquery.models.query.ManagedQuery;
import com.bakdata.conquery.models.query.PrintSettings;
import com.bakdata.conquery.models.query.QueryPlanContext;
import com.bakdata.conquery.models.query.QueryResolveContext;
import com.bakdata.conquery.models.query.queryplan.QueryPlan;
Expand All @@ -39,7 +38,7 @@ public Set<ManagedExecutionId> collectRequiredQueries() {
}

@JsonIgnore
public abstract List<ResultInfo> getResultInfos(PrintSettings printSettings);
public abstract List<ResultInfo> getResultInfos();

@Override
public ManagedQuery toManagedExecution(User user, Dataset submittedDataset, MetaStorage storage, DatasetRegistry<?> datasetRegistry) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@
import com.bakdata.conquery.models.error.ConqueryError;
import com.bakdata.conquery.models.identifiable.ids.specific.ManagedExecutionId;
import com.bakdata.conquery.models.query.DateAggregationMode;
import com.bakdata.conquery.models.query.PrintSettings;
import com.bakdata.conquery.models.query.QueryExecutionContext;
import com.bakdata.conquery.models.query.QueryPlanContext;
import com.bakdata.conquery.models.query.QueryResolveContext;
Expand All @@ -27,7 +26,7 @@
import com.bakdata.conquery.models.query.queryplan.ConceptQueryPlan;
import com.bakdata.conquery.models.query.queryplan.SecondaryIdQueryPlan;
import com.bakdata.conquery.models.query.resultinfo.ResultInfo;
import com.bakdata.conquery.models.query.resultinfo.printers.SecondaryIdResultInfo;
import com.bakdata.conquery.models.query.resultinfo.SecondaryIdResultInfo;
import com.fasterxml.jackson.annotation.JsonView;
import jakarta.validation.constraints.NotNull;
import lombok.Getter;
Expand Down Expand Up @@ -125,12 +124,12 @@ public void resolve(final QueryResolveContext context) {
}

@Override
public List<ResultInfo> getResultInfos(PrintSettings printSettings) {
public List<ResultInfo> getResultInfos() {
final List<ResultInfo> resultInfos = new ArrayList<>();

resultInfos.add(new SecondaryIdResultInfo(secondaryId, printSettings));
resultInfos.add(new SecondaryIdResultInfo(secondaryId));

resultInfos.addAll(query.getResultInfos(printSettings));
resultInfos.addAll(query.getResultInfos());

return resultInfos;
}
Expand Down
Loading
Loading