Investigate TPU benchmark suite

[coolalexzb]

This issue aims to collect the investigation of TPU benchmark suite.

Google’s Cloud TPU v2/v3 is the most commonly used TPU. From Using Cloud TPU Tools and Cloud TPU Monitoring with Stackdriver, users can use command line to capture profile for static TPU system metrics or use stackdriver to monitor dynamic TPU usage. One simple approach is creating log-based metrics programmatically using the Stackdriver REST API, crawling log-based metrics and plotting the graph we need.

[matthewfeickert]

Can we use stackdriver with the TPUs that are freely available on Colab, or do we need to use the paid TPU service?

@lukasheinrich Do we have any Google Cloud credits yet that we could use for TPU tests?

[matthewfeickert]

This is a low enough priority item that this should get shelved until all other aspects of the project are done.