This page details out the jobs required to be run as part of the upgrade from Sunbird Ed and CoKreat release 5.0.0 to release 5.1.0. Use the following table to understand the jobs that need to be executed in order to successfully complete the upgrade. Any jenkins job configuration or pre-requisites mentioned under manual configuration section needs to be done first before running any of the mentioned jobs. The order of the jobs should also be run as shown below. They can be run in parallel to speed up the execution.
Create cname for storage account
Create custom domain for storage account and update variables Before starting deployments.
Below are the steps to add custom domain to storage account
For Azure Click Here , AWS Click Here , for GCP Click Here
| Service | Build job | Build tag | Deploy Job | Deploy tag | Comments |
|---|---|---|---|---|---|
| Mobile App | Mobile/job/Sunbird-IONIC-Mobile-App/ |
release-5.1.0.7_RC1 | release-5.1.0.1 | ||
| Portal | Build/Core/Player |
release-5.1.0.6_RC1 | Deploy/Kubernetes/Player | release-5.1.0.2_RC1 | release-5.1.0.6 |
| ML Services | Build/managed-learn/ml-projects-service | release-5.1.0_RC32 | Deploy/managed-learn/ml-projects-service | release-5.1.0_RC7 | release-5.1.0.1 |
| Build/managed-learn/ml-core-service | release-5.1.0_RC11 | Deploy/managed-learn/ml-core-service | release-5.1.0_RC7 | release-5.1.0.1 | |
| Build/managed-learn/ml-reports-service | release-5.1.0_RC10 | Deploy/managed-learn/ml-reports-service | release-5.1.0_RC7 | release-5.1.0.1 | |
| Build/managed-learn/ml-survey-service | release-5.1.0_RC5 | Deploy/managed-learn/ml-survey-service | release-5.1.0_RC7 | release-5.1.0.1 | |
| ML-Analytics | Deploy/managed-learn/ml-analytics-service | release-5.1.0_RC14 | release-5.1.0.1 | ||
| Other | Deploy/Kubernetes/OnboardAPI | 5.1.0_RC5 | |||
| Deploy/Kubernetes/OnboardConsumers | 5.1.0_RC5 | ||||
| Deploy/Kubernetes/nginx-public-ingress | 5.1.0_RC9 | ||||
| Deploy/DataPipeline/InternalKong |
| Service | Build Job | Build Tag | Deploy Job | Deploy Tag | Comments |
|---|---|---|---|---|---|
| Certificate API | CertificateAPI | v0.0.11 | CertificateAPI | release-5.1.0_RC5 | First build the service with values - |
| and then deploy | |||||
| Registry | Registry | v0.0.13 | Registry | release-5.1.0_RC5 | First build the service with values - |
| and then deploy | |||||
| UploadRCSchema | release-5.1.0_RC5 |
| Service/Compon | Build job | Build tag | Deploy Job | Deploy tag | Comments |
|---|---|---|---|---|---|
| Sync tool | Build/KnowledgePlatform/SyncTool | release-5.2.0_RC9 | migratecspdata - **** Add this command to the command list in the Deploy/KnowledgePlatform/Neo4jElasticSearchSyncTool Jenkins job. | ||
| Content service | Build/Core/Content |
release-5.2.0_RC5 | Deploy/Kubernetes/Content | release-5.1.0_RC6 | |
| Taxonomy service | Build/Core/Taxonomy | release-5.2.0_RC5 | Deploy/Kubernetes/Taxonomy | release-5.1.0_RC6 | **** |
| DIAL service | Build/Core/Dial | release-5.2.0_RC1 | Deploy/Kubernetes/Dial | release-5.1.0_RC6 | **** |
| Learning service | Build/KnowledgePlatform/Learning | release-5.2.0_RC9 | Deploy/KnowledgePlatform/Learning | release-5.2.0_RC9 | **** |
| Knowlg middleware | Build/Core/KnowledgeMW | release-5.2.0_RC1 | Deploy/Kubernetes/KnowledgeMW | release-5.1.0_RC6 | Deploy this before deploying the service and flink-jobs. |
| Knowlg platform jobs | Build/KnowledgePlatform/FlinkJobs | release-5.2.0_RC11 | Deploy/KnowledgePlatform/FlinkJobs | release-5.2.0_RC8 | Add the below job list to the job_names_to_deploy list in the Deploy/KnowledgePlatform/FlinkJobs jenkins job csp-migrator, live-node-publisher, live-video-stream-generator, cassandra-data-migration Deploy the assessment-enrichment, content-publish, qrcode-image-generator, search-indexer, csp-migrator, live-node-publisher, live-video-stream-generator, cassandra-data-migration flink jobs. |
| Interactive editor | Build/Plugins/ContentEditor | release-5.2.0_RC1 | Deploy/Plugins/ContentEditor | release-5.1.0_RC5 | |
| Content plugins | Build/Plugins/ContentPlugins | release-5.2.0_RC1 | Deploy/Plugins/ContentPlugins | release-5.0.0_RC5 | |
| File upload editor | Build/Plugins/GenericEditor | release-5.2.0_RC1 | Deploy/Plugins/GenericEditor | release-5.0.0_RC5 | |
| Service/Component | Build job | Build tag | Deploy Job | Deploy tag | Comments |
|---|---|---|---|---|---|
| User & Org service | Build/Core/Learner | release-5.0.1_RC2 | Deploy/Kubernetes/Learner | release-5.1.0_RC6 | release-5.1.0 |
| LMS | Build/Core/Lms | release-5.0.1_RC3 | Deploy/Kubernetes/Lms | release-5.1.0_RC6 | |
| Certificate | Build/Core/Cert | release-5.0.1_RC2 | Deploy/Kubernetes/Cert | release-5.1.0_RC6 | release-5.1.0.1 |
| Lern data products | Build/Lern/LernDataProducts | release-5.0.1_RC4 | Deploy/Lern/LernDataProducts | release-5.0.1_RC4 | release-5.1.0.1 |
| Lern jobs | Build/Lern/LernFlinkJobs | release-5.0.1_RC5 | Deploy/Lern/LernFlinkJobs | release-5.0.1_RC5 | release-5.1.0.1 |
| Service to be Build | Build Tag | Service to Deploy | Deploy Tag | Comments |
|---|---|---|---|---|
| Deploy/Kubernetes/InquiryUploadSchema | schema_repo_branch_or_tag: release-5.2.0_RC2 source_folder: question,questionset | Create jenkins jobs from config | ||
| Build/Core/Assessment | release-5.2.0_RC3 | Deploy/Kubenetes/Assessment | release-5.1.0_RC4 | Update build and deploy job config, job-config |
| Build/job/KnowledgePlatform/InquiryFlinkJob | release-5.2.0_RC3 | Deploy/KnowledgePlatform/InquiryFlinkJob | release-5.2.0_RC3 | create build, deploy and artifactupload jobs, job-config |
| Build/Core/Analytics | release-5.1.0_RC2 | Deploy/Kubernetes/Analytics | release-5.1.0_RC3 |
| Service to be Build | Build Tag | Service to Deploy | Deploy Tag | Comments |
|---|---|---|---|---|
| Build/Core/Analytics | release-5.1.0_RC2 | Deploy/Kubernetes/Analytics | release-5.1.0_RC3 |
| variable name | description | Value |
|---|---|---|
| cloudstorage_replace_absolute_path | Boolean value is required to enable/disable cloud storage agnostic urls in db. Update in core/common.yml and KnowledgePlatform/common.yml | 'true' |
| cloudstorage_base_path | String value is required. The value could be either cloud specific base path or cname configured for current cloud storage provider. Update in core/common.yml and KnowledgePlatform/common.yml | ex: storage url: "https://storagename.blob.core.windows.net" cname url: https://obj.stage.sunbirded.org |
| valid_cloudstorage_base_urls | Array of String value is required. This array will have list of all cloud path which should be accepted by service. Update in core/common.yml and KnowledgePlatform/common.yml | ex: '["https://storagename.blob.core.windows.net","https://obj.stage.sunbirded.org"]' |
| cloudstorage_relative_path_prefix | String value is required. This value will be used as prefix for any metadata which holds cloud storage path. Update in Core/common.yml and KnowledgePlatform/common.yml | CONTENT_STORAGE_BASE_PATH |
| cloudstorage_metadata_list | Array of String value is required. This array will have list of metadata keys which holds cloud storage path. Update in Core/common.yml | ["appIcon","posterImage","artifactUrl","downloadUrl","variants","previewUrl","pdfUrl"] |
| cloud_storage_base_url | storage url, Update in Core/common.yml. | ex: "https://storagename.blob.core.windows.n |
| cloud_storage_cname_url | cname url, Update in Core/common.yml | ex: "https://obj.stage.sunbirded.org" |
| lern variables to be updated in inventory | update Lern/common.yml | https://lern.sunbird.org/use/release-notes/release-v-5.0.1#ansible-changes-need-to-be-updated-in-common.yml-secrets.yml-hosts.yml |
| cloud_service_provider | Update in core, kp, dp common.yml | Supported values are aws, azure, gcloud |
| lp_vault_youtube_api_key | youtube api key. Add in core/common.yml | copy value from KP secrets file |
| cloud_storage_url | storage url | ex: "https://storagename.blob.core.windows.n |
Check if below api's exists or not before adding, if exists replace the existing one if not add it.
- name: createAndUpdateCertificateTemplate
uris: "{{ private_ml_core_prefix }}/api/v1/certificateTemplates/createOrUpdate"
upstream_url: "{{ ml_core_private_url }}/v1/certificateTemplates/createOrUpdate"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: uploadCertificateTemplate
uris: "{{ private_ml_core_prefix }}/api/v1/certificateTemplates/uploadTemplate"
upstream_url: "{{ ml_core_private_url }}/v1/certificateTemplates/uploadTemplate"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: projectCertificateReissue
uris: "{{ private_ml_project_prefix }}/api/v1/userProjects/certificateReIssue"
upstream_url: "{{ ml_project_private_url }}/v1/userProjects/certificateReIssue"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: createCertificateTemplateSvg
uris: "{{ private_ml_core_prefix }}/api/v1/certificateTemplates/createSvg"
upstream_url: "{{ ml_core_private_url }}/v1/certificateTemplates/createSvg"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: createAndUpdateCertificateBaseTemplate
uris: "{{ private_ml_core_prefix }}/api/v1/certificateBaseTemplates/createOrUpdate"
upstream_url: "{{ ml_core_private_url }}/v1/certificateBaseTemplates/createOrUpdate"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
- name: certificateBaseTemplateDetailsList
uris: "{{ private_ml_core_prefix }}/api/v1/certificateBaseTemplates/list"
upstream_url: "{{ ml_core_private_url }}/v1/certificateBaseTemplates/list"
strip_uri: true
plugins:
- name: jwt
- name: cors
- "{{ statsd_pulgin }}"
- name: acl
config.whitelist:
- 'projectAccess'
- name: rate-limiting
config.policy: local
config.hour: "{{ medium_rate_limit_per_hour }}"
config.limit_by: credential
- name: request-size-limiting
config.allowed_payload_size: "{{ medium_request_size_limit }}"
curl -X POST \
{{host}}/api/dataset/v1/add \
-H 'authorization: Bearer {{jwt_token}}' \
-H 'content-type: application/json' \
-d '{
"id": "ekstep.analytics.dataset.add",
"ver": "1.0",
"ts": "2016-12-07T12:40:40+05:30",
"params": {
"msgid": "4f04da60-1e24-4d31-aa7b-1daf91c46341"
},
"request": {"dataset":"druid-dataset","datasetSubId":"ml-survey-status-detail-exhaust","datasetConfig":{"type":"ml-survey-status-detail-exhaust","params":{"programId":"program-1","solutionId":"solution-1"}},"datasetType":"druid","visibility":"private","version":"v1","authorizedRoles":["PROGRAM_MANAGER","PROGRAM_DESIGNER"],"validationJson":{"type":"object","properties":{"tag":{"id":"http://api.ekstep.org/dataexhaust/request/tag","type":"string"},"dataset":{"id":"http://api.ekstep.org/dataexhaust/request/dataset","type":"string"},"datasetSubId":{"id":"http://api.ekstep.org/dataexhaust/request/datasetSubId","type":"string"},"requestedBy":{"id":"http://api.ekstep.org/dataexhaust/request/requestedBy","type":"string"},"encryptionKey":{"id":"http://api.ekstep.org/dataexhaust/request/encryptionKey","type":"string"},"datasetConfig":{"id":"http://api.ekstep.org/dataexhaust/request/datasetConfig","type":"object"}},"required":["tag","dataset","datasetConfig"]},"druidQuery":{"id":"ml-survey-status-detail-exhaust","labels":{"user_id" : "UUID","user_type" : "User type","user_sub_tye" : "User sub type","state_name" : "Declared State","district_name" : "District","block_name" : "Block","school_code" : "School ID","school_name" : "School Name","board_name" : "Declared Board","organisation_name" : "Org Name","program_name" : "Program Name","program_externalId" : "Program ID","survey_name" : "Survey Name","survey_externalId" : "Survey ID","survey_submission_id" : "survey_submission_id","submission_status" : "Status of submission","submission_date" : "Submission date"},"dateRange":{"interval":{"startDate":"1901-01-01","endDate":"2101-01-01"},"granularity":"all","intervalSlider":0},"metrics":[{"metric":"total_content_plays_on_portal","label":"total_content_plays_on_portal","druidQuery":{"intervals":"1901-01-01T00:00+00:00/2101-01-01T00:00:00+00:00","dataSource":"ml-survey-status","columns":["user_id", "user_type", "user_sub_type", "state_name", "district_name", "block_name", "school_code", "school_name", "board_name", "organisation_name", "program_name", "program_externalId", "survey_name", "survey_externalId", "survey_submission_id", "submission_status","submission_date"],"queryType":"scan"}}],"output":[{"zip":false,"label":"","dims":["date"],"fileParameters":["id","dims"],"metrics":["user_id", "user_type", "user_sub_type", "state_name", "district_name", "block_name", "school_code", "school_name", "board_name", "organisation_name", "program_name", "program_externalId", "survey_name", "survey_externalId", "survey_submission_id", "submission_status","submission_date"],"type":"csv"}],"sort":["UUID","Program ID","Survey ID","survey_submission_id"],"queryType":"scan"},"supportedFormats":["csv","zip"],"exhaustType":"OnDemand"}}'
curl -X POST \
{{host}}/api/dataset/v1/add \
-H 'authorization: Bearer {{jwt_token}}' \
-H 'content-type: application/json' \
-d '{
"id": "ekstep.analytics.dataset.add",
"ver": "1.0",
"ts": "2016-12-07T12:40:40+05:30",
"params": {
"msgid": "4f04da60-1e24-4d31-aa7b-1daf91c46341"
},
"request": {
"dataset": "druid-dataset",
"datasetSubId": "ml-project-status-exhaust",
"datasetConfig": {
"type": "ml-project-status-exhaust",
"params": {
"programId": "program-1",
"solutionId": "solution-1"
}
},
"datasetType": "druid",
"visibility": "private",
"version": "v1",
"authorizedRoles": [
"PROGRAM_MANAGER",
"PROGRAM_DESIGNER"
],
"validationJson": {"type":"object","properties":{"tag":{"id":"http://api.ekstep.org/dataexhaust/request/tag","type":"string"},"dataset":{"id":"http://api.ekstep.org/dataexhaust/request/dataset","type":"string"},"datasetSubId":{"id":"http://api.ekstep.org/dataexhaust/request/datasetSubId","type":"string"},"requestedBy":{"id":"http://api.ekstep.org/dataexhaust/request/requestedBy","type":"string"},"encryptionKey":{"id":"http://api.ekstep.org/dataexhaust/request/encryptionKey","type":"string"},"datasetConfig":{"id":"http://api.ekstep.org/dataexhaust/request/datasetConfig","type":"object"}},"required":["tag","dataset","datasetConfig"]},
"druidQuery":{"id":"ml-project-status-exhaust","labels":{"block_name":"Block","board_name":"Declared Board","project_title_editable":"Project Title","project_completed_date":"Project completion date of the user","user_type":"User Type","designation":"User sub type","school_code":"School ID","project_created_date":"Project start date of the user","project_last_sync":"Project last Synced date","project_duration":"Project Duration","status_of_project":"Project Status","project_id":"Project ID","project_description":"Project Objective","program_externalId":"Program ID","organisation_name":"Org Name","createdBy":"UUID","school_name":"School Name","district_name":"District","program_name":"Program Name","certificate_status_customised":"Certificate Status","state_name":"Declared State"},"dateRange":{"interval":{"startDate":"1901-01-01","endDate":"2101-01-01"},"granularity":"all","intervalSlider":0},"metrics":[{"metric":"total_content_plays_on_portal","label":"total_content_plays_on_portal","druidQuery":{"intervals":"1901-01-01T00:00+00:00/2101-01-01T00:00:00+00:00","dataSource":"sl-project","columns":["createdBy","user_type","designation","state_name","district_name","block_name","school_name","school_code","board_name","organisation_name","program_name","program_externalId","project_id","project_title_editable","project_description","project_created_date","project_completed_date","project_duration","project_last_sync","status_of_project","certificate_status_customised"],"queryType":"scan"}}],"output":[{"zip":false,"label":"","dims":["date"],"fileParameters":["id","dims"],"metrics":["createdBy","user_type","designation","state_name","district_name","block_name","school_name","school_code","board_name","organisation_name","program_name","program_externalId","project_id","project_title_editable","project_description","project_created_date","project_completed_date","project_duration","project_last_sync","status_of_project","certificate_status_customised"],"type":"csv"}],"sort":["UUID","Program ID","Project ID"],"queryType":"scan"},
"supportedFormats": ["csv","zip"],
"exhaustType": "OnDemand"
}
}'
Refer to variables and comment section from here link to update below variables.
storage_endpoint variables for public, private, management and artifact is not required in current release, give empty string as value (cloud_public_storage_endpoint, cloud_private_storage_endpoint, etc)
// ###### common.yml ###############
cloud_public_storage_accountname: ""
cloud_public_storage_endpoint: ""
cloud_public_storage_region: ""
cloud_public_storage_project: ""
cloud_private_storage_accountname: ""
cloud_private_storage_endpoint: ""
cloud_private_storage_region: ""
cloud_private_storage_project: ""
cloud_management_storage_accountname: ""
cloud_management_storage_endpoint: ""
cloud_management_storage_region: ""
cloud_management_storage_project: ""
cloud_artifact_storage_accountname: ""
cloud_artifact_storage_endpoint: ""
cloud_artifact_storage_region: ""
cloud_artifact_storage_project: ""
Refer to variables and comment section from here link to update below secrets variables
// ########## secrets.yml ###############
cloud_public_storage_secret: ""
cloud_private_storage_secret: ""
cloud_management_storage_secret: ""
cloud_artifact_storage_secret: ""
Refer to variables and comment section from here link to update below variables
// ############# common.yml ##############
cloud_storage_certqr_bucketname: "certqr"
cloud_storage_chatbot_bucketname: "chatbot"
cloud_storage_dial_bucketname: "dial"
cloud_storage_flink_bucketname: "flink-state-backend"
cloud_storage_playercdn_bucketname: "player"
cloud_storage_public_bucketname: "public"
cloud_storage_publicreports_bucketname: "public-reports"
cloud_storage_privatereports_bucketname: "reports"
cloud_storage_samiksha_bucketname: "samiksha"
cloud_storage_schema_bucketname: "schema"
cloud_storage_sourcing_bucketname: "sourcing"
cloud_storage_offlineinstaller_bucketname: "{{env}}-offlineinstaller"
cloud_storage_content_bucketname: "sunbird-content-{{env}}"
cloud_storage_telemetry_bucketname: "telemetry-data-store"
cloud_storage_termsandcondtions_bucketname: "termsandcondtions"
cloud_storage_user_bucketname: "user"
cloud_storage_desktopappcrashlogs_bucketname: "desktopappcrashlogs"
cloud_storage_label_bucketname: "label"
cloud_storage_certservice_bucketname: "{{env}}-e-credentials"
cloud_storage_uci_bucketname: "uci-staging"
cloud_storage_cassandrabackup_bucketname: "cassandra-backup"
cloud_storage_dpcassandrabackup_bucketname: "dp-cassandra-backup"
cloud_storage_dppostgresbackup_bucketname: "dp-postgresql-backup"
cloud_storage_dpredisbackup_bucketname: "dp-redis-backup"
cloud_storage_esbackup_bucketname: "elasticsearch-snapshots"
cloud_storage_influxdbbackup_bucketname: "influxdb-backup"
cloud_storage_jenkinsbackup_bucketname: "jenkins-backup"
cloud_storage_mongobackup_bucketname: "mongodb-backup"
cloud_storage_neo4jbackup_bucketname: "neo4j-backup"
cloud_storage_redisbackup_bucketname: "redis-backup"
cloud_storage_artifacts_bucketname: "{{ artifacts_container }}"
Variables to remove from inventory
delete below variables from core/common.yml if its defined.
upstream_url
plugin_upstream_url
discussion_upstream_url
Migrations
Please execute the migration instructions mentioned by the dependent building blocks
To create a new form to show or remove the onboarding popups\
// Some codecurl --location --request POST '<https://{{env}}>/api/data/v1/form/create' \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer {{API KEY}}' \
--data-raw '{
"request": {
"type": "newUserOnboarding",
"action": "onboarding",
"subType": "global",
"rootOrgId": <please specify your root org id>,
"component": "portal",
"data":{
"fields": {
"shownewUserOnboarding": "true"
}
}
}
}'
Please follow the instructions mentioned in the below link by CoKreat building block to upgrade CoKreat from 5.0.0 to 5.1.0
{% hint style="info" %} Variables lists and build tag is same for Ed and vdn. Use release-5.1.0-vdn_RC1 for knowlg service deployments in VDN {% endhint %}
{% embed url="https://cokreat.sunbird.org/use/release-notes/cokreat-release-v5.1.0-upcoming-release#jenkins-jobs" %}