userorg-flink-job.md

User Cache Updater

'user-cache-updater-2.0' job is used to generate the user-metadata information which is complied by fetching information from various Cassandra tables and are stored into the Redis cache. This user-metadata information is used by few exhaust reports.

Additional Reading:

https://project-sunbird.atlassian.net/wiki/spaces/AN/pages/1520074753/Design+Denormalise+User+Metadata,

Design guide:

https://lern.sunbird.org/learn/product-and-developer-guide/user-and-org-service/caching-and-denormalising-user-metadata/usercacheupdaterflinkjob

Configuration variables:

Variable	Default Value	Purpose
kafka.input.topic	{{env}}.telemetry.audit	Kafka topic from which messages/events are read to be processed.
kafka.groupId	{{env}}-user-cache-updater-group	Kafka input topic group Id
redis-meta.database.userstore.id	12	Redis index to which user metadata is to be written to for caching
redis-meta.database.key.expiry.seconds	3600	Redis cache expiry in seconds
user-read.api.url	"/learner/private/user/v1/read"	API Endpoint for fetching User profile details
regd.user.producer.pid	learner-service	used to specify service providing user microservice
user.self.signin.types	["google","self"]	used to specify self sign-in modes available in application
user.validated.types	["sso"]	used to specify sign-in modes where user validation is signed from third party system
user.self.signin.key	"Self-Signed-In"
user.valid.key	"Validated"
user.read.url.fields	"locations,organisations"	used to specify the user metadata properties that are to be cached to Redis
user.read.api.error	["CLIENT_ERROR"]

Sample event:

{
  "eid": "AUDIT",
  "ets": 1573121861118,
  "ver": "3.0",
  "mid": "1573121861118.40f9136b-1cc3-458d-a04a-4459606df",
  "actor": {
    "id": "5609876543234567890987654345678",
    "type": "Request"
  },
  "context": {
    "channel": "01285019302823526477",
    "pdata": {
      "id": "dev.sunbird.portal",
      "pid": "learner-service",
      "ver": "2.5.0"
    },
    "env": "User",
    "did": "user-3",
    "cdata": [
      {
        "id": "google",
        "type": "SignupType"
      }
    ],
    "rollup": {
      "l1": "01285019302823526477"
    }
  },
  "object": {
    "id": "user-1",
    "type": "user"
  },
  "edata": {
    "state": "Update",
    "props": [
      "recoveryEmail",
      "recoveryPhone",
      "userId",
      "id",
      "externalIds",
      "updatedDate",
      "updatedBy"
    ]
  },
  "syncts": 1573121861125,
  "@timestamp": "2019-11-07T10:17:41.125Z",
  "flags": {
    "tv_processed": true,
    "dd_processed": true
  },
  "type": "events",
  "ts": "2019-11-07T10:17:41.118+0000"
}

Source code:

{% embed url="https://github.com/Sunbird-Lern/data-pipeline/tree/release-5.3.0/user-org-jobs/user-cache-updater-2.0" %}