Simplify stats polling logic

[bduffany]

This PR cleans up some stats logic in preparation for tracking firecracker stats using cgroups, instead of tracking stats using the guest vmexec server. The current logic is somewhat tangled up which would make the integration with firecracker a bit more messy than it needs to be.

• Remove the logic to record prometheus metrics for running containers, which manually totals the results of c.Stats(). This logic is complex and its value is pretty questionable.
  • On Linux, a better way to do this would be to poll the stats from the action parent cgroup once executor.child_cgroups_enabled is rolled out.
  • On macOS we could probably just live with the node metrics instead of trying to separate the executor metrics and task metrics. The accuracy of the task metrics on macOS is especially questionable, since the poll rate is lower, and it requires walking the process tree, which can miss short-lived processes spawned during an execution.
• Simplify the stats polling logic, which was tangled up and somewhat circular.
  • The previous logic would have each container call the TrackStats utility function, which would start a loop that calls back to the container's Stats() function, which was expected to call usageStats.Update() then return usageStats.TaskStats().
  • Now, TrackStats is a method of the UsageStats struct, and manages these careful state updates internally, without a circular call back to the container.Stats() function, and without expecting the caller to call usageStats.Reset() and usageStats.TaskStats()

[vanja-p]

is the TODO on line 126 completed now?

[bduffany]

Removed this TODO for now - if we wanted to remove this "baseline" tracking logic, but still be able to properly track usage for remote persistent workers, then we'd have to migrate all of the container processes to a new cgroup for each task executed, which seems a bit too complicated. We can maybe revisit later.

[vanja-p]

the mention of podman on the line above might be out of date.

[bduffany]

That part is still accurate, since podman doesn't (yet) have the logic where we create the cgroup then run the container in that cgroup that we created. Instead, podman manages the cgroup lifecycle itself.

I removed this note for now though, since it's not super important. The bigger TODO is to make sure that the container implementations don't delete the cgroup from under us (this is a problem with both ociruntime and podman). That way, we could remove this polling, and reliably read the cgroup stats once at the end of execution.