Prometheus metrics collection

[lwangbm]

/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]
Add Prometheus metrics to record experiment, trial, job information

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]