Skip to content

Commit

Permalink
Merge branch 'release-v61.2.0' into release
Browse files Browse the repository at this point in the history
  • Loading branch information
badboy committed Oct 7, 2024
2 parents 2c7ad32 + 56168b6 commit d15d644
Show file tree
Hide file tree
Showing 20 changed files with 232 additions and 37 deletions.
2 changes: 1 addition & 1 deletion .buildconfig.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
libraryVersion: 61.1.0
libraryVersion: 61.2.0
groupId: org.mozilla.telemetry
projects:
glean:
Expand Down
11 changes: 10 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
# Unreleased changes

[Full changelog](https://github.com/mozilla/glean/compare/v61.1.0...main)
[Full changelog](https://github.com/mozilla/glean/compare/v61.2.0...main)

# v61.2.0 (2024-10-07)

[Full changelog](https://github.com/mozilla/glean/compare/v61.1.0...v61.2.0)

* Kotlin
* Accept a ping schedule map on initialize ([#2967](https://github.com/mozilla/glean/pull/2967))
* Swift
* Accept a ping schedule map on initialize ([#2967](https://github.com/mozilla/glean/pull/2967))

# v61.1.0 (2024-09-24)

Expand Down
4 changes: 2 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions DEPENDENCIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -4948,9 +4948,9 @@ SOFTWARE.

The following text applies to code linked from these dependencies:

* [glean-core 61.1.0]( https://github.com/mozilla/glean )
* [glean-core 61.2.0]( https://github.com/mozilla/glean )
* [glean-build 15.0.1]( https://github.com/mozilla/glean )
* [glean 61.1.0]( https://github.com/mozilla/glean )
* [glean 61.2.0]( https://github.com/mozilla/glean )
* [zeitstempel 0.1.1]( https://github.com/badboy/zeitstempel )

```
Expand Down
1 change: 0 additions & 1 deletion build-scripts/xc-universal-binary.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ if [[ -n "${DEVELOPER_SDK_DIR:-}" ]]; then
# # In this case, we need to add an extra library search path for build scripts and proc-macros,
# # which run on the host instead of the target.
# # (macOS Big Sur does not have linkable libraries in /usr/lib/.)
local libpath
libpath="${DEVELOPER_SDK_DIR}/MacOSX.sdk/usr/lib"
export LIBRARY_PATH="${libpath}:${LIBRARY_PATH:-}"
fi
Expand Down
49 changes: 33 additions & 16 deletions docs/user/user/howto/real-time-events/real-time-events.md
Original file line number Diff line number Diff line change
@@ -1,36 +1,53 @@
# "Real-Time" Events

## Defining "Real-Time" Glean Events
## Defining "real-time" events within the Glean SDK

"Real-Time" in the context of [Glean events](../../../reference/metrics/event.md) directly relates to configuring the Glean SDK to send each event in an ["events" ping](../../pings/events.md) as soon as it is recorded.
This minimizes the time between instrumentation and reporting, but doesn't describe how quickly received data is made available for querying.
For the purposes of the Glean SDK and its capabilities, "real-time" is limited to: minimizing the time between instrumentation and reporting.
It does not imply or describe how quickly received data is made available for querying.

## Configuring Glean For "Real-Time" Events
## Methods to achieve this with Glean

Glean "events" ping submission can either be configured at [initialization](../../../reference/general/initializing.md) or through [Server Knobs](../../../user/server-knobs/other/max-events.md).
### Option 1: Configuring Glean to send all events as soon as they are recorded

Setting the maximum event threshold to a value of `1` will configure the Glean SDK to submit an "events" ping for every event recorded.
Glean ["events" ping](../../pings/events.md) submission can be configured either during [initialization](../../../reference/general/initializing.md) or through [Server Knobs](../../../user/server-knobs/other/max-events.md).

Setting the maximum event threshold to a value of `1` will configure the Glean SDK to submit an "events" ping for each and every [event](../../../reference/metrics/event.md) as they
are recorded. By default, the Glean SDK will batch 500 events per "events" ping.

### Option 2: Using a custom ping and submitting it immediately ("Pings-as-Events")

If it isn't necessary to receive all Glean SDK events that are instrumented in an application in "real-time", it may be preferable to create a
[custom ping](../../pings/custom.md) which contains the relevant information to capture the context around the event and submit it as soon as
the application event occurs.

This has some additional advantages over using just an event in that custom pings are less restrictive than the extras attached to the event
in what data and Glean SDK metric types can be used.

If it is important to see the event that is being represented as a custom ping in context with other application events, then you only need to
define an event metric and use the `send_in_pings` parameter to send it in both the custom ping and the Glean built-in "events" ping. It can
then be seen in sequence and within context of all of the application events, and still be sent in "real-time" as needed.

## Considerations

### What "Real-Time" Glean Events Are _Not_
### What "real-time" Glean events/pings are _not_

Configuring the Glean SDK to submit events as soon as they are recorded does not mean to imply that the event data is available for analysis in
real time. There are networks to traverse, ingestion pipelines, etl, etc. that are all factors to keep in mind when considering how soon the data
is available for analysis purposes. This documentation only purports to cover configuring the Glean SDK to send the data in a real-time fashion and
does not make any assumptions about the analysis of data in real-time.
Configuring the Glean SDK to submit events as soon as they are recorded or using custom pings to submit data immediately does not mean that the
data is available for analysis in real time. There are networks to traverse, ingestion pipelines, etl, etc. that are all factors to keep in
mind when considering how soon the data is available for analysis purposes. This documentation only purports to cover configuring the Glean SDK
to send the data in a real-time fashion and does not make any assumptions about the analysis of data in real-time.

### More Network Requests
### More network requests

For every event recorded, a network request will be generated when the event is submitted for ingestion. By default, the Glean SDK batches up to
500 events per "events" ping, so this has the potential to generate up to 500 times as many network requests than the current default.
For every event recorded or custom ping submitted, a network request will be generated as the ping is submitted for ingestion. By default, the
Glean SDK batches up to 500 events per "events" ping, so this has the potential to generate up to 500 times as many network requests than the
current defaults for the Glean SDK "events" ping.

### More Ingestion Endpoint Traffic
### More ingestion endpoint traffic

As a result of the increased network requests, the ingestion endpoint will need to handle this additional traffic. This increases the load
of all the processing steps that are involved with ingesting event data from an application.

### Storage Space
### Storage space requirements

Typically the raw dataset for Glean events contains 1-500 events in a single row of the database. This row also includes metadata such as
information about the client application and the ping itself. With only a single event per "events" ping, the replication of this metadata
Expand Down
2 changes: 1 addition & 1 deletion glean-core/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "glean-core"
version = "61.1.0"
version = "61.2.0"
authors = ["Jan-Erik Rediger <[email protected]>", "The Glean Team <[email protected]>"]
description = "A modern Telemetry library"
repository = "https://github.com/mozilla/glean"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ open class GleanInternalAPI internal constructor() {
enableEventTimestamps = configuration.enableEventTimestamps,
experimentationId = configuration.experimentationId,
enableInternalPings = configuration.enableInternalPings,
pingSchedule = emptyMap(),
pingSchedule = configuration.pingSchedule,
pingLifetimeThreshold = configuration.pingLifetimeThreshold.toULong(),
pingLifetimeMaxTime = configuration.pingLifetimeMaxTime.toULong(),
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ import mozilla.telemetry.glean.net.PingUploader
* @property delayPingLifetimeIo Whether Glean should delay persistence of data from metrics with ping lifetime.
* @property pingLifetimeThreshold Write count threshold when to auto-flush. `0` disables it.
* @property pingLifetimeMaxTime After what time to auto-flush (in milliseconds). 0 disables it.
* @property pingSchedule A ping schedule map.
* Maps a ping name to a list of pings to schedule along with it.
* Only used if the ping's own ping schedule list is empty.
*/
data class Configuration @JvmOverloads constructor(
val serverEndpoint: String = DEFAULT_TELEMETRY_ENDPOINT,
Expand All @@ -44,6 +47,7 @@ data class Configuration @JvmOverloads constructor(
val delayPingLifetimeIo: Boolean = true,
val pingLifetimeThreshold: Int = 1000,
val pingLifetimeMaxTime: Int = 0,
val pingSchedule: Map<String, List<String>> = emptyMap(),
) {
companion object {
/**
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */

package mozilla.telemetry.glean.scheduler

import androidx.test.ext.junit.runners.AndroidJUnit4
import mozilla.telemetry.glean.Glean
import mozilla.telemetry.glean.delayMetricsPing
import mozilla.telemetry.glean.getContext
import mozilla.telemetry.glean.getMockWebServer
import mozilla.telemetry.glean.private.NoReasonCodes
import mozilla.telemetry.glean.private.PingType
import mozilla.telemetry.glean.resetGlean
import mozilla.telemetry.glean.testing.GleanTestRule
import mozilla.telemetry.glean.triggerWorkManager
import okhttp3.mockwebserver.MockWebServer
import org.junit.After
import org.junit.Assert.assertEquals
import org.junit.Before
import org.junit.Rule
import org.junit.Test
import org.junit.runner.RunWith
import java.util.concurrent.TimeUnit

/**
* Testing behavior of custom pings.
*
* We already rely on the Rust side to test custom pings,
* but this enables us to test the upload mechanism specifically.
*
* Even if this seemingly duplicates some of the testing, this should be kept around.
*/
@RunWith(AndroidJUnit4::class)
class RidealongPingTest {
private val context = getContext()
private lateinit var server: MockWebServer

@get:Rule
val gleanRule = GleanTestRule(context)

@Before
fun setup() {
server = getMockWebServer()
}

@After
fun teardown() {
server.shutdown()
}

@Test
fun `sends a ride-along custom ping on baseline schedule`() {
delayMetricsPing(context)
resetGlean(
context,
Glean.configuration.copy(
serverEndpoint = "http://" + server.hostName + ":" + server.port,
pingSchedule = mapOf("baseline" to listOf("custom-ping")),
),
clearStores = true,
uploadEnabled = true,
)

// Define a new custom ping inline.
PingType<NoReasonCodes>(
name = "custom-ping",
includeClientId = true,
sendIfEmpty = true,
preciseTimestamps = true,
includeInfoSections = true,
enabled = true,
schedulesPings = emptyList(),
reasonCodes = emptyList(),
)

Glean.handleBackgroundEvent()
// Trigger it to upload
triggerWorkManager(context)

var request = server.takeRequest(2L, TimeUnit.SECONDS)!!
var docType = request.path!!.split("/")[3]
assertEquals("baseline", docType)

request = server.takeRequest(2L, TimeUnit.SECONDS)!!
docType = request.path!!.split("/")[3]
assertEquals("custom-ping", docType)
}
}
6 changes: 5 additions & 1 deletion glean-core/ios/Glean.xcodeproj/project.pbxproj
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@
CD0CADA427E216810015A997 /* glean.swift in Sources */ = {isa = PBXBuildFile; fileRef = CDD08C8527E21104007C8400 /* glean.swift */; };
CD0F7CC026F0F27900EDA6A4 /* UrlMetric.swift in Sources */ = {isa = PBXBuildFile; fileRef = CD0F7CBF26F0F27900EDA6A4 /* UrlMetric.swift */; };
CD0F7CC226F0F28900EDA6A4 /* UrlMetricTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = CD0F7CC126F0F28900EDA6A4 /* UrlMetricTests.swift */; };
CD3682F32CAC110300B02F04 /* RidealongPingTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = CD3682F22CAC10FE00B02F04 /* RidealongPingTests.swift */; };
CD38786D271DCCC700C097D8 /* libglean_ffi.a in Frameworks */ = {isa = PBXBuildFile; fileRef = CD38786C271DCCC700C097D8 /* libglean_ffi.a */; };
CD70CF932850D69500FC2014 /* Gzip in Frameworks */ = {isa = PBXBuildFile; productRef = CD70CF922850D69500FC2014 /* Gzip */; };
CD70CF982850D77200FC2014 /* OHHTTPStubs in Frameworks */ = {isa = PBXBuildFile; productRef = CD70CF972850D77200FC2014 /* OHHTTPStubs */; };
Expand Down Expand Up @@ -144,6 +145,7 @@
CD07A4DD2BC808AE007A0F1C /* ObjectMetric.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = ObjectMetric.swift; sourceTree = "<group>"; };
CD0F7CBF26F0F27900EDA6A4 /* UrlMetric.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = UrlMetric.swift; sourceTree = "<group>"; };
CD0F7CC126F0F28900EDA6A4 /* UrlMetricTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = UrlMetricTests.swift; sourceTree = "<group>"; };
CD3682F22CAC10FE00B02F04 /* RidealongPingTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RidealongPingTests.swift; sourceTree = "<group>"; };
CD387868271D9CD100C097D8 /* glean.udl */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; name = glean.udl; path = ../../src/glean.udl; sourceTree = "<group>"; };
CD38786C271DCCC700C097D8 /* libglean_ffi.a */ = {isa = PBXFileReference; lastKnownFileType = archive.ar; name = libglean_ffi.a; path = ../../target/libglean_ffi.a; sourceTree = "<group>"; };
CD81DCF9282A8F9A00347965 /* RateMetric.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RateMetric.swift; sourceTree = "<group>"; };
Expand Down Expand Up @@ -301,7 +303,6 @@
isa = PBXGroup;
children = (
C27E756429D4B56500C6AADD /* Utils */,
60691AEA28DD0BF200BDF31A /* BaselinePingTests.swift */,
1F39E7B1239F0741009B13B3 /* Debug */,
1FB8F8392326EBA500618E47 /* Config */,
BF43A8CB232A613100545310 /* Metrics */,
Expand Down Expand Up @@ -372,6 +373,8 @@
BF80AA5923992FFB00A8B172 /* Net */ = {
isa = PBXGroup;
children = (
60691AEA28DD0BF200BDF31A /* BaselinePingTests.swift */,
CD3682F22CAC10FE00B02F04 /* RidealongPingTests.swift */,
BF80AA5E2399305200A8B172 /* DeletionRequestPingTests.swift */,
BF80AA5A2399301200A8B172 /* HttpPingUploaderTests.swift */,
);
Expand Down Expand Up @@ -637,6 +640,7 @@
CD0F7CC226F0F28900EDA6A4 /* UrlMetricTests.swift in Sources */,
BFCBD6AB246D55CC0032096D /* TestUtils.swift in Sources */,
AC06529E26E034BF00D92D5E /* QuantityMetricTypeTest.swift in Sources */,
CD3682F32CAC110300B02F04 /* RidealongPingTests.swift in Sources */,
1F58921223C923C4007D2D80 /* MetricsPingSchedulerTests.swift in Sources */,
CD9DA7852BC809BE00E18F31 /* ObjectMetricTests.swift in Sources */,
1FD4527723395EEB00F4C7E8 /* UuidMetricTests.swift in Sources */,
Expand Down
8 changes: 7 additions & 1 deletion glean-core/ios/Glean/Config/Configuration.swift
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ public struct Configuration {
let enableInternalPings: Bool
let pingLifetimeThreshold: Int
let pingLifetimeMaxTime: Int
let pingSchedule: [String: [String]]

struct Constants {
static let defaultTelemetryEndpoint = "https://incoming.telemetry.mozilla.org"
Expand All @@ -36,6 +37,9 @@ public struct Configuration {
/// * enableInternalPings Whether to enable internal pings.
/// * pingLifetimeThreshold Write count threshold when to auto-flush. `0` disables it.
/// * pingLifetimeMaxTime After what time to auto-flush (in milliseconds). 0 disables it.
/// * pingSchedule A ping schedule map.
/// Maps a ping name to a list of pings to schedule along with it.
/// Only used if the ping's own ping schedule list is empty.
public init(
maxEvents: Int32? = nil,
channel: String? = nil,
Expand All @@ -46,7 +50,8 @@ public struct Configuration {
experimentationId: String? = nil,
enableInternalPings: Bool = true,
pingLifetimeThreshold: Int = 0,
pingLifetimeMaxTime: Int = 0
pingLifetimeMaxTime: Int = 0,
pingSchedule: [String: [String]] = [:]
) {
self.serverEndpoint = serverEndpoint ?? Constants.defaultTelemetryEndpoint
self.maxEvents = maxEvents
Expand All @@ -58,5 +63,6 @@ public struct Configuration {
self.enableInternalPings = enableInternalPings
self.pingLifetimeThreshold = pingLifetimeThreshold
self.pingLifetimeMaxTime = pingLifetimeMaxTime
self.pingSchedule = pingSchedule
}
}
2 changes: 1 addition & 1 deletion glean-core/ios/Glean/Glean.swift
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ public class Glean {
enableEventTimestamps: configuration.enableEventTimestamps,
experimentationId: configuration.experimentationId,
enableInternalPings: configuration.enableInternalPings,
pingSchedule: [:],
pingSchedule: configuration.pingSchedule,
pingLifetimeThreshold: UInt64(configuration.pingLifetimeThreshold),
pingLifetimeMaxTime: UInt64(configuration.pingLifetimeMaxTime)
)
Expand Down
2 changes: 1 addition & 1 deletion glean-core/ios/Glean/Metrics/ObjectMetric.swift
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ extension Array: ObjectSerialize where Element: Codable {
public func intoSerializedObject() -> String {
let jsonEncoder = JSONEncoder()
let jsonData = try! jsonEncoder.encode(self)
let json = String(decoding: jsonData, as: UTF8.self)
let json = String(data: jsonData, encoding: String.Encoding.utf8)!
return json
}
}
Expand Down
Loading

0 comments on commit d15d644

Please sign in to comment.