Log metrics outside sagemaker studio

[andrew-ge-wu]

Is your feature request related to a problem? Please describe.
Today we are using sagemaker notebook as development environment, and we like the sagemaker experiment concepts and try to use sagemaker experiments API to track our work.
We found most of the features we need works just fine without sagemaker studio except metrics.

Describe the solution you'd like
A consistent API experience with or without sagemaker studio.
When tracking metrics, it saves to trail component.

Describe alternatives you've considered
Configurable backend alternative to sagemaker studio, when it is not yet in GA.

Additional context

[danabens]

Hi Andrew, thanks for your issue. We are working on this and will update this issue on progress, however no ETA yet.

[edgBR]

Hi community,

We had the same problem here and that´s why we were not using experiments in our current project. We actually had to create a "model registry" table in our DWH to deal with this issue. We were using boto3 to retrieve all of the parameters from our consequent training jobs.

BR
/E

[andrew-ge-wu]

Hi
We still need this feature, because our retraining is happening outside of sagemaker context. We will have to make a choice of continuing to invest in sagemaker experiment functionalities or using other opensource softwares.

Best

/AW

[wangweixun]

Hi Andrew -- unfortunately we still don't have an ETA on this. Can I ask how you track "retraining is happening outside of sagemaker"? Do you create a custom trial component and add metrics as TrialComponentArtifact? Note that we do not support calling CreateTrialComponent outside of an SageMaker env by default today.