Crash Reporting: Collect Crashes and Metrics

[mackoj]

Add support for metric kit as signals.

[winsmith]

Awesome idea! Added it to the roadmap and will investigate further.

[Sherlouk]

This is an amazing idea! I wonder how this would work from a data perspective though?

Currently we send signals which essentially increments a counter by 1 for each time that signal is received. We can add text properties though, but unsure how we can query those effectively.

There are quite a few different data types with MetricKit.

An easy one to start with might be the cumulative app exit counts. There are about 20 of these (describing the different ways the app can be exited) each with a count between 0 and .

We could create a payload which looks a bit like:

[
 { type: 'cumulativeAppWatchdogExitCount' },
 { type: 'cumulativeAppWatchdogExitCount' },
 { type: 'cumulativeAppWatchdogExitCount' },
 { type: 'cumulativeMemoryResourceLimitExitCount' },
]

Essentially creating a massive payload where we duplicate certain types based on the integer provided by Apple. So in the example above, we had 3 exits due to watchdog and 1 due to memory limit.

This feels like with larger numbers though it would dramatically increase the payload size. Perhaps we need a new value count: Int which is optional and defaults to 1. If this is set, then it increases the count by X where X is the number provided?

This perhaps gets more complicated when you look at different data types though such as kB, ms/sec, etc.

  "networkTransferMetrics" : {
    "cumulativeCellularDownload" : "80,000 kB",
    "cumulativeWifiDownload" : "60,000 kB",
    "cumulativeCellularUpload" : "70,000 kB",
    "cumulativeWifiUpload" : "50,000 kB"
  },

metric-kit-sample.json.zip

I've attached a sample JSON in case folks want to take a look. I'd love to hear from @winsmith on how we could approach this though as it would be an amazing capability to have such information about stability and performance that Apple doesn't currently surface to us.

[winsmith]

Thanks! I think what precedes this is a move to a different signal format that supports not only individual dimensions (i.e. the payload which is string: string) but also metrics, (i.e. a value that maps from string to Int or Double or something). This is on our roadmap but there's so much else to do it'll be a little while before I can hash this out.

[Sherlouk]

Agreed, I think a move to support metrics with numeric values would be a huge value add. The Telemetry Viewer app could then have a field for "unit" from a dropdown which would then allow for some automatic formatting.

So if the client always sends values in milliseconds, the viewer could then format it as seconds/minutes/etc based on how big the number is.

If there's anything I can do to support that piece of work Daniel then please let me know.

[Sherlouk]

Curious @winsmith if there's been any progression on this? 👀 I know it was recently brought up on Twitter, and I totally forgot that this thread existed.

MetricKit provides a lot of valuable data, and is already aggregated by the platform in a privacy friendly way. Would be nice to take full advantage of that.

[winsmith]

How would that support look like? Should the SDK get these metrics automatically? Then you might run into compatibility problems depending on which platform you include the SDK. Or is it more to do with the fact that the API needs support for numerical data? Because that's coming :)

[mackoj]

Having it automated would be best, with an option to manually add custom metrickit data.

[Sherlouk]

I suppose there are a few options.

One would be for the backend to just support the MetricKit JSON payload verbatim... this would likely be easiest, but from a technical perspective is likely far from where you want to be.

The other option is to create an optional target (or use compiler flags to include only on iOS) a MetricKit manager. The client app would then just need to add this to their app using something like MXMetricManager.shared.add(TelemetryDeckMetricKitExporter()). This is doable as soon as TD supports all the same data types as MetricKit exports - which includes numeric data via histograms etc.

[winsmith]

Hmm okay that makes sense. I'm wondering if it's possible to make this as cross-platform as possible, so that e.g. Android users could also get similar data.

[Sherlouk]

MetricKit itself is an iOS specific technology and format. That doesn't mean that most of the data couldn't be ascertained on Android (it can) but it defeats the point of this being a quick win for privacy-friendly metrics without a complex client side integration on iOS apps.

[winsmith]

Thanks a lot for your input! I'm bumping this up on the roadmap.

[Sherlouk]

@winsmith let me know if you'd like to dive deeper into the solution or goals! happy to help how I can.

[winsmith]

FYI for everyone still attached to this ticket: this is moving forward now

[Manuel-Kehl]

I'm playing around with adding very lightweight MetricKit integration to my own app's use of TelemetryDeck and just quickly wanted to share one thing I learned that might be relevant (and you're likely already aware but I wanted to share anyway - just in case 😉):

Since the metrics are collected over 24h periods and aggregated, the app version may not necessarily match that of the running app. This could lead to slightly skewed stats, e.g. when setting up insights in TelemetryDeck to see the count of abnormal background app exists per build version...

MXMetricPayload does have a latestApplicationVersion property as well as a includesMultipleApplicationVersions property which could be used to maybe change the app version metadata of the relevant signals accordingly? Not sure if that could have any other side effects or if there are other organizational/semantic reasons one would not want to do that? For my own implementation, I simply decided to add an extra metadata pair for the "metrics app version", so that I can query that for those specific scenarios (which works good enough).

But for an official, first-party integration of MetricKit into the TelemetryDeck SwiftSDK it might be worth considering, maybe? 🤔 Not sure, tbh... 😊