Clickhouse mode of study view #11224

alisman · 2024-11-22T20:24:16Z

Fix # (see https://help.github.com/en/articles/closing-issues-using-keywords)

Describe changes proposed in this pull request:

a
b

Checks

The commit log is comprehensible. It follows 7 rules of great commit messages. We can fix this during merge by using a squash+merge if necessary
Has tests or has a separate issue that describes the types of test that should be created. If no test is included it should explicitly be mentioned in the PR why there is no test.
Is this PR adding logic based on one or more clinical attributes? If yes, please make sure validation for this attribute is also present in the data validation / data loading layers (in backend repo) and documented in File-Formats Clinical data section!
Make sure your PR has one of the labels defined in https://github.com/cBioPortal/cbioportal/blob/master/.github/release-drafter.yml

Any screenshots or GIFs?

If this is a new visual feature please add a before/after screenshot or gif
here with e.g. Giphy CAPTURE or Peek

Notify reviewers

Read our Pull request merging
policy. It can help to figure out who worked on the
file before you. Please use git blame <filename> to determine that
and notify them either through slack or by assigning them as a reviewer on the PR

alisman · 2024-11-22T20:25:36Z

src/main/resources/org/cbioportal/persistence/mybatis/ClinicalEventMapper.xml

@@ -181,7 +181,7 @@
                </if>
            </if>
        </where>
-        Group by clinical_event.EVENT_TYPE, patient.STABLE_ID
+        Group by clinical_event.EVENT_TYPE, patient.INTERNAL_ID


@haynescd this is one fix i did. i think we can keep it. the stable id is not unique across study

alisman · 2024-11-22T21:10:11Z

src/main/java/org/cbioportal/service/impl/AlterationCountServiceImpl.java

@@ -223,6 +263,197 @@ public Pair<List<CopyNumberCountByGene>, Long> getPatientCnaGeneCounts(List<Mole
        );
    }

+    @Override


@haynescd what's all this stuff?

This is the stuff for the new clickhouse implementation

alisman · 2024-11-22T21:16:33Z

src/main/resources/db-scripts/clickhouse/materialized_views.sql

@@ -0,0 +1,32 @@
+DROP TABLE IF EXISTS sample_list_columnstore;


@haynescd we can kill this file, right?

alisman · 2024-11-22T21:18:22Z

src/main/resources/org/cbioportal/persistence/mybatis/GenePanelMapper.xml

@@ -121,6 +121,7 @@
        <include refid="selectGenePanelData"/>
        <include refid="fromGenePanelData"/>
        WHERE
+        SAMPLE_ID IS NOT NULL AND 


@haynescd this is one we might want to undo because it changes profile counts. this is what allows the system to recover from incomplete sample_profile table issue.

alisman · 2024-11-22T21:19:11Z

src/main/resources/templates/index.html

@@ -32,7 +32,7 @@
        window.netlify = localStorage.netlify;

        if (window.localdev || window.localdist) {
-            window.frontendConfig.frontendUrl = "//localhost:3000/"
+            window.frontendConfig.frontendUrl = "https://localhost:3000/"


might possible break localdb tests

Yea, probably

* Add Columnar SQL file to init Clickhouse DB * Refactored Mapper xml to extract StudyViewFilterMapper

* ✅ Add Unit test for StudyViewMapper Clickhouse * ✅ Update db props to include mysql and clickhouse datasources to fix tests * Address comments * Rename package to clickhouse * Update to static final * Use bean name instead of qualifier

* Create new wide table sql file and rename package * Remove genomic_event view * Add AlterationFilter to mutated_genes endpoint * Add AlterationFilter to mutated-genes endpoint * Fix unit test * Fix sonar issues * Add test for mutation types and status * remove unused imports

* add missing poc clinical data binning function

* Add sample_mv materialized view and use it in mappers

* Add Support for TotalProfiledCase Counts for Mutated-genes endpoint. * Create sql files to create new tables * Add unit test for totalProfiledCount * Add matching gene panel ids * Add TotalProfiledCountsWithoutPanelData * Add profileCount for genes without gene panel data * Add Comments for SQL * Update matching Gene Panel Ids * Clean up code * Fix test * Add query to get correct Gene Panels * Fix unit test * Add comments

* working poc * refactor logic into service, so clean * refactor for parameters builder, simplify min max logic, streamline service call * remove unused services and imports * remove more unused imports

* Implement molecular profile count endpoint using Clickhouse * Cleanup

* ✨ Add CNA Gene Endpoint * 🐛 Fix StudyViewFilterMapper.xml to allow ability to filter on gene and alteration * Fix merge conflict * Address comments * Fix unit tests * Fix sonar issues

* ✨ Add StructuralVariant-genes endpoint * Fix sonar issues * Update MatchingGenePanel request to return list * Create and use sample_derive * Update where sample_derived is stored to fix unit test

Co-authored-by: Bryan Lai <[email protected]>

* use clinical_data_derived instead of sample_clinical_attribute_numeric_mv and patient_clinical_attribute_numeric_mv * use clinical_attribute_meta instead of sample_clinical_attribute_numeric_mv and patient_clinical_attribute_numeric_mv * remove unused clinical data count methods and SQL * fix numericalClinicalDataCountFilter * Move CategoricalClinicalAttributeFilter to repository * remove unused columns * Add override to methods --------- Co-authored-by: haynescd <[email protected]>

…0857) * Add patient_id column to genomic_event_derived * Update sql to convert list of patients to list of samples

* refactor to use clickhouse * filter out empty attr values * edit comment * fix sonarcloud issues * use parallel stream, shaves off 5s * use newer mapping annotation

* add api tests for circleci

This reverts commit 80ef8c9.

* add tests for profiled counts * fix profiled counts assertions --------- Co-authored-by: Bryan Lai <[email protected]>

#11209) * separate clickhouse tests into different files * change name of cna counts test file --------- Co-authored-by: Bryan Lai <[email protected]>

…inical data counts endpoint (#11192) Co-authored-by: alisman <[email protected]>

Use PR-specific image for circleci api tests

Co-authored-by: Bryan Lai <[email protected]>

haynescd · 2024-11-25T02:26:12Z

src/main/java/org/cbioportal/persistence/util/RedisCacheUtils.java

@@ -14,7 +14,9 @@
 import java.util.stream.Collectors;

 @Component
-@ConditionalOnProperty(name = "persistence.cache_type", havingValue = "redis")
+@ConditionalOnExpression(
+    "#{environment['persistence.cache_type'] == 'redis' or environment['persistence.cache_type_clickhouse'] == 'redis'}"


I think we should remove this and be enable caching or disable caching for the whole system... going forward

agreed. this is so that we can assess performance without cheating with cache. we need caching on for legacy because otherwise, the product is totally unusable!

and at the same time, when we deploy for demonstration purposes, we want initial load of studyview to compete with legacy, i.e. cache ON.

haynescd · 2024-11-25T02:33:06Z

src/main/java/org/cbioportal/service/impl/RedisCacheStatisticsServiceImpl.java

@@ -19,7 +20,9 @@
 import java.util.stream.Collectors;

 @Service
-@ConditionalOnProperty(name = "persistence.cache_type", havingValue = {"redis"})
+@ConditionalOnExpression(
+    "#{environment['persistence.cache_type'] == 'redis' or environment['persistence.cache_type_clickhouse'] == 'redis'}"


Same as above... Should we still separate these?

* Hide select properties of ClinicalDataFilter from frontend * Update swagger decorators on clickhouse controller

sonarcloud · 2024-11-25T23:11:53Z

Quality Gate failed

Failed conditions
4 New Bugs (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

alisman commented Nov 22, 2024

View reviewed changes

haynescd and others added 25 commits November 23, 2024 23:28

➕ Add infra to support multi db and Columnar DataSource Properties

bd2918f

Add Columnar support for Mybatis

86bd2cc

Add Columnar Study View Controller /api/column-store

9db2295

Feature/ch/update poc base branch (#10776)

b4a8a38

* Add Columnar SQL file to init Clickhouse DB * Refactored Mapper xml to extract StudyViewFilterMapper

use regular views instead of materialized views (#10793)

12b7b07

Add missing POC clinical data binning function (#10778)

63858cf

* add missing poc clinical data binning function

Create sample materialized view and use it (#10822)

b5fb29e

* Add sample_mv materialized view and use it in mappers

RFC80:density plot endpoint(#10837)

a03ec54

* working poc * refactor logic into service, so clean * refactor for parameters builder, simplify min max logic, streamline service call * remove unused services and imports * remove more unused imports

Fix clinical data counts query/filter using new derived table (#10833)

1051dea

Molecular-profile-counts service implementation (#10840)

ed26ca7

* Implement molecular profile count endpoint using Clickhouse * Cleanup

Fix clinical data scatterplot (point at new derived cd table) (#10846)

2015f69

Rfc80/add cna genes endpoint (#10845)

6e8e6c7

* ✨ Add CNA Gene Endpoint * 🐛 Fix StudyViewFilterMapper.xml to allow ability to filter on gene and alteration * Fix merge conflict * Address comments * Fix unit tests * Fix sonar issues

✨ Add StructuralVariant-genes endpoint (#10854)

5b0f2bd

* ✨ Add StructuralVariant-genes endpoint * Fix sonar issues * Update MatchingGenePanel request to return list * Create and use sample_derive * Update where sample_derived is stored to fix unit test

Implement study view sample-lists-counts endpoint in Clickhouse (#10849)

3cac3a2

Co-authored-by: Bryan Lai <[email protected]>

Fix class import (#10856)

3dd1e2a

Rfc80/fix patient level filtering for alteration counts endpoints (#1…

ab3000a

…0857) * Add patient_id column to genomic_event_derived * Update sql to convert list of patients to list of samples

🐛 Fix totalCount calculation when WES not available (#10858)

fc03406

Consolidate SQL files used to create CH Tables (#10867)

604568f

Demo rfc80 poc violin plot (#10863)

a569ec6

* refactor to use clickhouse * filter out empty attr values * edit comment * fix sonarcloud issues * use parallel stream, shaves off 5s * use newer mapping annotation

replace sample_mv references with sample_derived (#10880)

783b608

🔧 Update Sonar Cloud Token (#10888)

fc2f99d

fuzhaoyuan and others added 18 commits November 23, 2024 23:40

Update generic assay table schema (#11189)

5804a71

Do not used prepared statements for the studyid lists (#11191)

c56d3ea

Api Tests Config for CircleCI (#11188)

a4ced11

* add api tests for circleci

Revert "add custom driver filtering unit tests" (#11203)

74209d4

This reverts commit 80ef8c9.

Update Total profile Count to filter on genetic profile (#11198)

85329b6

Add tests for profiled counts (#11190)

01b72a9

* add tests for profiled counts * fix profiled counts assertions --------- Co-authored-by: Bryan Lai <[email protected]>

Remove test failing due to legacy miscount (#11205)

543f115

cast all data filter start and end values to float (#11207)

4e3dc8f

Split clickhouse integration tests into separate files for each method (

242ca4d

#11209) * separate clickhouse tests into different files * change name of cna counts test file --------- Co-authored-by: Bryan Lai <[email protected]>

Adding option for extracting study ids from sample identifiers for cl…

cad5ab8

…inical data counts endpoint (#11192) Co-authored-by: alisman <[email protected]>

Add missing DROP TABLE statements (#11211)

3174391

Use PR-specific image for circleci api tests (#11217)

db43621

Use PR-specific image for circleci api tests

Pretty print portal info in api-test logs

1e95a1c

Add a frontend prop for clickhouse mode (#11219)

ce3354d

require approval for restricted tests

a6ace50

rename approval job for circleci tests

ef25b13

add integration tests for cna-genes and sv-genes (#11221)

53c7126

Co-authored-by: Bryan Lai <[email protected]>

Fix Rebase issues with master

2b45019

haynescd force-pushed the demo-rfc80-poc branch from 4d4b302 to 2b45019 Compare November 25, 2024 02:21

Merge branch 'master' into demo-rfc80-poc

64fe5d0

haynescd reviewed Nov 25, 2024

View reviewed changes

zainasir and others added 2 commits November 25, 2024 10:34

Use upstream repo for api-tests

b44eb6e

Fix swagger decorators for rfc80 clickhouse controller (#11230)

b780409

* Hide select properties of ClinicalDataFilter from frontend * Update swagger decorators on clickhouse controller

alisman changed the title ~~Clickhouse api~~ Clickhouse mode of study view Nov 25, 2024

onursumer added 4 commits November 25, 2024 18:03

check for non-numerical values in numerical clinical data count filter

2aac75e

unify numerical data filters in one sql

9075f37

do not normalize attribute value if not NA

17a5dad

add unit tests

c97fc06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clickhouse mode of study view #11224

Clickhouse mode of study view #11224

alisman commented Nov 22, 2024

alisman Nov 22, 2024

haynescd Nov 25, 2024

alisman Nov 22, 2024

haynescd Nov 25, 2024

alisman Nov 22, 2024

haynescd Nov 25, 2024

alisman Nov 22, 2024

alisman Nov 22, 2024

haynescd Nov 25, 2024

haynescd Nov 25, 2024

alisman Nov 25, 2024

alisman Nov 25, 2024

haynescd Nov 25, 2024

sonarcloud bot commented Nov 25, 2024

		@@ -0,0 +1,32 @@
		DROP TABLE IF EXISTS sample_list_columnstore;

Clickhouse mode of study view #11224

Are you sure you want to change the base?

Clickhouse mode of study view #11224

Conversation

alisman commented Nov 22, 2024

Checks

Any screenshots or GIFs?

Notify reviewers

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Nov 25, 2024

Quality Gate failed