| Project | Release Date | Version |
|---|---|---|
| Lern | 18 Aug 22 | V 5.0.0 |
| Components | Tags |
|---|---|
| Batch Service | sunbird-course-service : release-5.0.0_RC6 cert-service : release-5.0.0_RC4 certficate-registry : release-5.0.0_RC1 data-pipeline : release-5.0.0_RC6 |
| User&Org Service | sunbird-lms-service : release-5.0.0_RC5 |
| Group Service | groups-service : release-5.0.0_RC3 |
| Discussion Forum Service | discussions-middleware : release-5.0.0_RC2 sunbird-nodebb : release-5.0.0_RC1 |
| Notification Service | sunbird-notification-service : release-5.0.0_RC6 |
| Data Products | data-products : release-5.0.0_RC4 |
- Migrating component repositories in to Lern organisation in GitHub as a first step in making the installation and setup easy for adopters and contributors
- Increasing code coverage and unit test cases of all the components in Lern as part of stabilising the components
- Refactoring of the provisioning and deployment scripts of Lern BB
- Making SB Lern Cloud agnostic
{% tabs %} {% tab title="User&Org Service" %}
Configurations:
Sunbird-lms-service:
isMultiDCEnabled={{cassandra_isMultiDCEnabled}}
sunbird_cassandra_consistency_level={{sunbird_cassandra_consistency_level}}
sunbird_user_cert_kafka_topic={{kafka_topic_lms_user_account}}
sunbird_cloud_service_provider={{cloud_service_provider}}
sunbird_account_name={{sunbird_public_storage_account_name}}
sunbird_account_key={{sunbird_public_storage_account_key}}
{% endtab %}
{% tab title="Batch Service" %}
| JIRA ID | Description |
|---|---|
| SB-30075 | Batch service - increase code coverage and unit test cases |
| SB-29843 | Batch service Dataproducts Migration to Lern |
| SB-30068 | Batch service - Deployment and Release processes |
| SB-29841 | assessment aggregate updater Migration to Lern |
| SB-29840 | activity aggregate updater Migration to Lern |
| SB-29839 | course-service migration to Lern |
| SB-29837 | enrolment-reconciliation Migration to Lern |
| SB-29836 | relation cache updater Migration to Lern |
| SB-29835 | colletion-certificate-generator Migration to Lern |
| SB-29834 | collection-cert-pre-processor Migration to Lern |
| SB-29833 | certificate processor Migration to Lern |
| SB-29832 | certificate-registry migration to Lern |
| SB-29831 | cert-service migration to Lern |
| LR-104 | Making SB Lern Cloud agnostic : Code changes to generalise CSP support in BatchService |
| LR-105 | Making SB Lern Cloud agnostic : Code changes to generalise CSP support in datapipeline |
| LR-106 | Making SB Lern Cloud agnostic : Code changes to generalise CSP support in data products |
| LR-108 | Signed URL Generation for old certificates - cert-service |
| LR-107 | RC deploy helm chart changes to upload Credential template, context files |
Configurations:
https://github.com/project-sunbird/sunbird-devops/blob/learn-bb/ansible/roles/stack-sunbird/templates/
Sunbird-course-service:
isMultiDCEnabled={{cassandra_isMultiDCEnabled}}
# Add proper cloud service provider (azure,aws,gcloud)
sunbird_cloud_service_provider={{cloud_service_provider}}
sunbird_account_name={{sunbird_public_storage_account_name}}
sunbird_account_key={{sunbird_public_storage_account_key}}
#deleted the below variable
sunbird_content_azure_storage_container=sunbird-content-dev
# Added below variable for supporting multiple cloud service providers
# Provide corresponding cloud service provider(azure,aws,gcloud) container name here
sunbird_content_cloud_storage_container=sunbird-content-dev
Cert-service:
CONTAINER_NAME={{cert_service_container_name}}
# Add proper cloud service provider (azure,aws,gcloud)
CLOUD_STORAGE_TYPE={{cloud_service_provider}}
PRIVATE_CLOUD_STORAGE_SECRET={{sunbird_private_storage_account_key}}
PRIVATE_CLOUD_STORAGE_KEY={{sunbird_private_storage_account_name}}
PUBLIC_CLOUD_STORAGE_KEY={{sunbird_public_storage_account_name}}
PUBLIC_CLOUD_STORAGE_SECRET={{sunbird_public_storage_account_key}}
Certificate-registry:
isMultiDCEnabled={{cassandra_isMultiDCEnabled}}
Data-pipeline:
isMultiDCEnabled={{cassandra_isMultiDCEnabled}}
# Add proper cloud service provider (azure,aws,gcloud)
cloud_storage_type :azure/aws/gcloud
data-products:
cloud_storage_type :azure/aws/gcloud
{% endtab %}
{% tab title="Discussion Forum" %}
| JIRA ID | Descriptions |
|---|---|
| SB-30070 | DF - Deployment and Release processes |
| SB-29820 | DF Migration to Lern and deployment setup |
| SB-30073 | DF - increase code coverage and unit test cases |
| {% endtab %} |
{% tab title="Group Service" %}
| JIRA ID | Description |
|---|---|
| SB-30074 | Group service - increase code coverage and unit test cases |
| SB-30069 | Group service - Deployment and Release processes |
| SB-29819 | Group-service Migration to Lern and deployment setup |
Configurations:
groups-service:
File Path:
cassandra-utils/src/main/resources/cassandra.config.properties
Changes:
isMultiDCEnabled=false
File Path:
sb-utils/src/main/resources/cassandra.config.properties
Changes:
isMultiDCEnabled=false
{% endtab %}
{% tab title="Notification Service" %}
| JIRA ID | Description |
|---|---|
| SB-30077 | Sunbird-notification-service - increase code coverage and unit test cases |
| SB-29828 | Sunbird-notification-jobs migration to Lern |
| SB-29827 | Sunbird-notification-service migration to Lern |
| SB-30071 | Sunbird-notification-service - Deployment and Release processes |
Configurations:
Sunbird-notification-service:
File path:
sb-utils/src/main/resources/cassandra.config.properties
Changes:
isMultiDCEnabled=false
{% endtab %} {% endtabs %}
Detailed Information is present in the JIRA list.
Configurations:
- Nodebb upstream branch: v1.18.6
- Jenkins build, deploy and upload related changes for Flink jobs are present in below link:
{% embed url="https://github.com/Sunbird-Lern/data-pipeline/tree/release-5.0.0/kubernetes/pipelines" %}
3. There is a new variable added in devops repo to configure the bb name for kafka topics of flink jobs. This variable should be appended after the env name.
{% embed url="https://github.com/Sunbird-Lern/data-pipeline/blob/release-5.0.0/ansible/inventory/env/group_vars/all.yml#L10" %}
List of Lern Flink jobs:
collection-cert-pre-processor
collection-certificate-generator
activity-aggregate-updater
relation-cache-updater
merge-user-courses
assessment-aggregator
enrolment-reconciliation
notification-job
env variable changes are listed below:
kubernetes/ansible/roles/flink-jobs-deploy/defaults/main.yml
### Merge User Courses Job related Vars
merge_user_courses_consumer_parallelism: 1
merge_user_courses_parallelism: 1
merge_user_courses_course_batch_parallelism: 1
merge_user_courses_course_date_format: "yyyy-MM-dd HH:mm:ss:SSSZ"
### Notification Job related Vars
notification_job_consumer_parallelism: 1
notification_job_parallelism: 1
### assessment-aggregator related vars
assessaggregator_parallelism: 1
assessaggregator_consumer_parallelism: 1
assessaggregator_downstream_parallelism: 1
assessaggregator_scoreaggregator_parallelism: 1
middleware_cassandra_courses_keyspace: sunbird_courses
middleware_cassandra_assessment_aggregator_table: assessment_aggregator
middleware_cassandra_assessment_question_type : question
middleware_cassandra_user_enrolments_table: user_enrolments
middleware_cassandra_user_activity_agg_table: user_activity_agg
content_read_api_host: "http://dev.sunbirded.org"
content_read_api_endpoint: "/api/content/v1/read/"
merge-user-courses:
job_class_name: 'org.sunbird.job.merge.user.courses.task.MergeUserCoursesStreamTask'
replica: 1
jobmanager_memory: 1024m
taskmanager_memory: 1024m
taskslots: 1
cpu_requests: 0.3
assessment-aggregator:
job_class_name: 'org.sunbird.dp.assessment.task.AssessmentAggregatorStreamTask'
replica: 1
jobmanager_memory: 1024m
taskmanager_memory: 1024m
taskmanager_process_memory: 1700m
jobmanager_process_memory: 1600m
taskslots: 1
cpu_requests: 0.3
scale_enabled: false
notification-job:
job_class_name: 'org.sunbird.job.notification.task.NotificationStreamTask'
replica: 1
jobmanager_memory: 1024m
taskmanager_memory: 1024m
taskslots: 1
cpu_requests: 0.3
Stop the existing Samza jobs - (merge-user-courses and notification-job)
Stop the assessment-aggregator job from sunbird-data-pipeline and remove it from the corresponding Jenkins job as well
All these 3 jobs will be running from LERN repo now
4. Jenkins build, deploy and upload related changes for data products are in below link:
{% embed url="https://github.com/Sunbird-Lern/data-products/tree/release-5.0.0/pipelines" %}
LERN data-products list
exhaust / progressexhaustjob
exhaust / responseexhaust
exhaust / userinfoexhaust
job / course consumption
job / course enrolment
job / stateadminreport
job / stateadmingeoreport
job / collectionsummaryjobv2
audit / collection reconcilation
audit / coursebatch status updater
updater / Cassandramigrator
5. Jenkins build, deploy and upload related changes for microservices are in below link:
{% embed url="https://github.com/project-sunbird/sunbird-devops/tree/learn-bb" %}
Devops config changes:
{% code overflow="wrap" %}
File Path:
ansible/inventory/env/group_vars/all.yml
### Release-5.0.0 cloud service provider changes for supporting multiple providers ###
### cloud_service_provider value should be either (azure, aws, gcloud) as per cloud sdk dependency ###
cloud_service_provider: "azure"
## modified consistency levels from quorum to local_quorum for multiple data centers support
File Path:
ansible/roles/stack-sunbird/templates/sunbird_cert-registry-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum
File Path:
ansible/roles/stack-sunbird/templates/sunbird_groups-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum
File Path:
ansible/roles/stack-sunbird/templates/sunbird_learner-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum
sunbird_user_cert_kafka_topic={{env_name}}{{bb}}.lms.user.account.merge
File Path:
ansible/roles/stack-sunbird/templates/sunbird_lms-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum
File Path:
ansible/roles/stack-sunbird/templates/sunbird_notification-service.env
Changes:
sunbird_cassandra_consistency_level=local_quorum
sunbird_notification_kafka_topic={{env_name}}{{bb}}.lms.notification
File Path:
kubernetes/helm_charts/sunbird-RC/registry/schemas/TrainingCertificate.json
Changes:
"credentialTemplate": "https://{{upstream_url}}/schema/credential_template.json"
File Path:
utils/sunbird-RC/schema/credential_template.json
Changes:
"https://{{upstream_url}}/schema/v1_context.json",
"https://{{upstream_url}}/schema/sunbird_context.json"
{% endcode %}
Design Documentation:
{% embed url="https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3077767171/Sunbird-Lern+Code+Flink+Jobs+and+Data+Products" %} Sunbird-Lern Code, Flink Jobs and Data Products {% endembed %}
{% embed url="https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3149922315/Lern+BB+repositories" %} Lern BB repositories {% endembed %}
{% embed url="https://project-sunbird.atlassian.net/wiki/spaces/UM/pages/3156869136/Sunbird-Lern+existing+jenkins+jobs" %} Sunbird-Lern existing jenkins jobs {% endembed %}