All instructions to upgrade this project from one release to the next will be documented in this file. Upgrades must be run sequentially, meaning you should not skip minor/major releases while upgrading (fix releases can be skipped).
This project adheres to Semantic Versioning.
xAPI learning statements validator and converter are built with Pydantic. Ralph 5.x
is compatible with Pydantic 2.x. Please refer to Pydantic migration
guide if you are using Ralph models
feature.
Most of fields in Pydantic models that are optional are set with
None
as default value in Ralph 5.y. If you serialize some Pydantic models from ralph and want to keep the same content in your serialization, please setexclude_none
toTrue
in the serialization methodmodel_dump
.
To conform to xAPI specifications, we need to represent users as xAPI Agents. You must therefore delete and re-create the credentials file using the updated cli, OR you can modify it directly to add the agents
field. The credentials file is located in { RALPH_APP_DIR }/{ RALPH_AUTH_FILE }
(defaults to .ralph/auth.json
). Each user profile must follow the following pattern (see this post for examples of valid agent objects) :
{
"username": "USERNAME_UNCHANGED",
"hash": "PASSWORD_HASH_UNCHANGED",
"scopes": [ LIST_OF_SCOPES_UNCHANGED ],
"agent": { A_VALID_AGENT_OBJECT }
}
Agent can take one of the following forms, as specified by the xAPI specification:
- mbox:
"agent": {
"mbox": "mailto:[email protected]"
}
- mbox_sha1sum:
"agent": {
"mbox_sha1sum": "ebd31e95054c018b10727ccffd2ef2ec3a016ee9",
}
- openid:
"agent": {
"openid": "http://foo.openid.example.org/"
}
- account:
"agent": {
"account": {
"name": "simonsAccountName",
"homePage": "http://www.exampleHomePage.com"
}
For example here is a valid auth.json
file:
[
{
"username": "[email protected]",
"hash": "$2b$12$yBXrzIuRIk6yaft5KUgVFOIPv0PskCCh9PXmF2t7pno.qUZ5LK0D2",
"scopes": ["example_scope"],
"agent": {
"mbox": "mailto:[email protected]"
}
},
{
"username": "[email protected]",
"hash": "$2b$12$yBXrzIuRIk6yaft5KUgVFOIPv0PskCCh9PXmF2t7pno.qUZ5LK0D2",
"scopes": ["second_scope", "third_scope"],
"agent": {
"account": {
"name": "simonsAccountName",
"homePage": "http://www.exampleHomePage.com"
}
}
}
]
If you are using Ralph's CLI, the following changes may affect you:
-
The
ralph fetch
command changed toralph read
-
The
-b ws
backend option changed to-b async_ws
- The corresponding
--ws-uri
option changed to--async-ws-uri
- The corresponding
-
The
-c --chunk-size
option changed to-s --chunk-size
-
The
DEFAULT_BACKEND_CHUNK_SIZE
environment variable configuration is removed in favor of allowing each backend to define their own defaults:Backend Environment variable for default (read) chunk size async_es/es RALPH_BACKENDS__DATA__ES__READ_CHUNK_SIZE=500 async_lrs/lrs RALPH_BACKENDS__DATA__LRS__READ_CHUNK_SIZE=500 async_mongo/mongo RALPH_BACKENDS__DATA__MONGO__READ_CHUNK_SIZE=500 clickhouse RALPH_BACKENDS__DATA__CLICKHOUSE__READ_CHUNK_SIZE=500 fs RALPH_BACKENDS__DATA__FS__READ_CHUNK_SIZE=4096 ldp RALPH_BACKENDS__DATA__LDP__READ_CHUNK_SIZE=4096 s3 RALPH_BACKENDS__DATA__S3__READ_CHUNK_SIZE=4096 swift RALPH_BACKENDS__DATA__SWIFT__READ_CHUNK_SIZE=4096
-
-
The
ralph push
command changed toralph write
-
The
-c --chunk-size
option changed to-s --chunk-size
-
The
DEFAULT_BACKEND_CHUNK_SIZE
environment variable configuration is removed in favor of allowing each backend to define their own defaults:Backend Environment variable for default (write) chunk size async_es/es RALPH_BACKENDS__DATA__ES__WRITE_CHUNK_SIZE=500 async_lrs/lrs RALPH_BACKENDS__DATA__LRS__WRITE_CHUNK_SIZE=500 async_mongo/mongo RALPH_BACKENDS__DATA__MONGO__WRITE_CHUNK_SIZE=500 clickhouse RALPH_BACKENDS__DATA__CLICKHOUSE__WRITE_CHUNK_SIZE=500 fs RALPH_BACKENDS__DATA__FS__WRITE_CHUNK_SIZE=4096 ldp RALPH_BACKENDS__DATA__LDP__WRITE_CHUNK_SIZE=4096 s3 RALPH_BACKENDS__DATA__S3__WRITE_CHUNK_SIZE=4096 swift RALPH_BACKENDS__DATA__SWIFT__WRITE_CHUNK_SIZE=4096
-
-
Environment variables used to configure backend options for CLI usage (read/write/list commands) changed their prefix:
RALPH_BACKENDS__{{DATABASE or HTTP or STORAGE or STREAM}}__{{BACKEND}}__{{OPTION}}
changed toRALPH_BACKENDS__DATA__{{BACKEND}}__{{OPTION}}
-
Environment variables used to configure backend options for LRS usage (runserver command) changed their prefix:
RALPH_BACKENDS__{{DATABASE}}__{{BACKEND}}__{{OPTION}}
changed toRALPH_BACKENDS__LRS__{{BACKEND}}__{{OPTION}}
CLI syntax has been changed from fetch
& push
to read
& write
affecting the command history. You must replace the command history after updating:
- locate your history file path, which is in
{ RALPH_APP_DIR }/history.json
(defaults to.ralph/history.json
) - run the commands below to update history
sed -i 's/"fetch"/"read"/g' { my_history_file_path }
sed -i 's/"push"/"write"/g' { my_history_file_path }
If you use Ralph's backends in your application, the following changes might affect you:
Backends from ralph.backends.database
, ralph.backends.http
, ralph.backends.stream
,
and ralph.backends.storage
packages have moved to a single ralph.backends.data
package.
Ralph v3 (database/http/storage/stream) backends | Ralph v4 data backends |
---|---|
ralph.backends.database.clickhouse.ClickHouseDatabase |
ralph.backends.data.clickhouse.ClickHouseDataBackend |
ralph.backends.database.es.ESDatabase |
ralph.backends.data.es.ESDataBackend |
ralph.backends.database.mongo.MongoDatabase |
ralph.backends.data.mongo.MongoDataBackend |
ralph.backends.http.async_lrs.AsyncLRSHTTP |
ralph.backends.data.async_lrs.AsyncLRSDataBackend |
ralph.backends.http.lrs.LRSHTTP |
ralph.backends.data.lrs.LRSDataBackend |
ralph.backends.storage.fs.FSStorage |
ralph.backends.data.fs.FSDataBackend |
ralph.backends.storage.ldp.LDPStorage |
ralph.backends.data.ldp.LDPDataBackend |
ralph.backends.storage.s3.S3Storage |
ralph.backends.data.s3.S3DataBackend |
ralph.backends.storage.swift.SwiftStorage |
ralph.backends.data.swift.SwiftDataBackend |
ralph.backends.stream.ws.WSStream |
ralph.backends.data.async_ws.AsyncWSDataBackend |
LRS-specific query_statements
and query_statements_by_ids
database backend methods
have moved to a dedicated ralph.backends.lrs.BaseLRSBackend
interface that extends the
data backend interface with these two methods.
The query_statements_by_ids
method return type changed to Iterator[dict]
.
Ralph v3 database backends for lrs usage | Ralph v4 LRS data backends |
---|---|
ralph.backends.database.clickhouse.ClickHouseDatabase |
ralph.backends.lrs.clickhouse.ClickHouseLRSBackend |
ralph.backends.database.es.ESDatabase |
ralph.backends.lrs.es.ESLRSBackend |
ralph.backends.database.mongo.MongoDatabase |
ralph.backends.lrs.mongo.MongoLRSBackend |
Backend interface differences
- Data backends are read-only by default
- Data backends that support write operations inherit from the
ralph.backends.data.base.Writable
interface - Data backends that support list operations inherit from the
ralph.backends.data.base.Listable
interface - Data backends that support LRS operations
(
query_statements
/query_statements_by_ids
) inherit from theralph.backends.lrs.BaseLRSBackend
interface __init__(self, **kwargs)
changed to__init__(self, settings: DataBackendSettings)
where each DataBackend defines it's own Settings object For example theFSDataBackend
usesFSDataBackendSettings
stream
andget
methods changed toread
put
methods changed towrite
Backend usage migration example
Ralph v3 using ESDatabase
:
from ralph.conf import ESClientOptions
from ralph.backends.database.es import ESDatabase, ESQuery
# Instantiate the backend.
backend = ESDatabase(
hosts="localhost",
index="statements"
client_options=ESClientOptions(verify_certs=False)
)
# Read records from backend.
query = ESQuery(query={"query": {"term": {"modulo": 0}}})
es_statements = list(backend.get(query))
# Write records to backend.
backend.put([{"id": 1}])
Ralph v4 using ESDataBackend
:
from ralph.backends.data.es import (
ESClientOptions,
ESDataBackend,
ESDataBackendSettings,
ESQuery,
)
# Instantiate the backend.
settings = ESDataBackendSettings(
HOSTS="localhost",
INDEX="statements",
CLIENT_OPTIONS=ESClientOptions(verify_certs=False)
)
backend = ESDataBackend(settings)
# Read records from backend.
query = ESQuery(query={"term": {"modulo": 0}})
es_statements = list(backend.read(query))
# Write records to backend.
backend.write([{"id": 1}])
If you are using the ClickHouse backend, schema changes have been made to drop the existing JSON column in favor of the String version of the same data. See this issue for details.
Ralph does not manage the ClickHouse schema, so if you have existing data you will need to manually alter it as an admin user. Note: this will rewrite the statements table, which may take a long time if you have many rows. The command to run is:
-- If RALPH_BACKENDS__DATA__CLICKHOUSE__DATABASE is 'xapi'
-- and RALPH_BACKENDS__DATA__CLICKHOUSE__EVENT_TABLE_NAME is 'test'
ALTER TABLE xapi.test DROP COLUMN event, RENAME COLUMN event_str to event;