Sagemaker inference components are unsuable in pulumi

[isaac-smothers]

Describe what happened

Currently, you cannot use sagemaker endpoint inference components via pulumi. The TL;DR is that you need to connect your inference components to an endpoint deployed without a model attached to its production variant, and this isn't possible as modelName is a required property.

Long winded version -
Pulumi's aws-native provider enables you to create inference components, which is great. You create an inference component and you attach it to an existing endpoint. If you try to attach it to an endpoint that already has a running model, you get Invalid request provided: Inference Components are not supported in this Endpoint. Please make sure this endpoint can deploy inference components.

I did some research and found that in order to attach an inference component to an endpoint, the endpoint must have been configured without any models. This is done by not specifying a modelName when creating a productionVariant inside of a endpointConfiguration. You can do this via boto (although not the UI), as modelName is not a required property (see here). However pulumi's AWS package does specify it as required. This blocks you from being able to attach an inference component to a pulumi-created endpoint.

I'm going to see if I can resolve this myself and submit a PR, but as I've never contributed to the pulumi project before I'm not sure how successful I'll be.

Sample program

import * as pulumi from "@pulumi/pulumi"
import * as aws from "@pulumi/aws"
import * as aws2 from "@pulumi/aws-native"

const roleToAssumeARN = "arn:aws:iam::1234567890:role/deployer"

const provider = new aws.Provider("privileged", {
    assumeRole: {
        roleArn: roleToAssumeARN,
        sessionName: "PulumiSession",
        externalId: "PulumiApplication",
    },
    region: aws.config.requireRegion(),
})

const awsNativeProvider = new aws2.Provider("provider", {
    assumeRole: {
        roleArn: roleToAssumeARN,
        sessionName: "PulumiSession",
        externalId: "PulumiApplication",
    },
    region: aws.config.requireRegion(),
})

const name = "sagemaker-test"

const endpointConfig = new aws.sagemaker.EndpointConfiguration(
    `${name}-endpoint-config`,
    {
        namePrefix: `${name}-endpoint-config`,
        productionVariants: [
            {
                // modelName: "REMOVE-ME",
                initialInstanceCount: 1,
                instanceType: "ml.g5.12xlarge"
            },
        ],
    },
    { provider },
)

const endpoint = new aws.sagemaker.Endpoint(
    `${name}-endpoint`,
    {
        name: `${name}-endpoint`,
        endpointConfigName: endpointConfig.name,
    }, { provider },
)

new aws2.sagemaker.InferenceComponent(
    `${name}-inference-component`,
    {
        endpointName: endpoint.name,
        endpointArn: endpoint.arn,
        variantName:
            endpointConfig.productionVariants[0].variantName,
        inferenceComponentName: `${name}-base-component`,
        specification: {
            modelName: "model-sold-separately",
        },
    },
    { provider: awsNativeProvider },
)

Log output

Error when giving it a valid model name:

error: creating resource: creating resource (await): operation CREATE failed with "InvalidRequest": Invalid request provided: Inference Components are not supported in this Endpoint. Please make sure this endpoint can deploy inference components.

Affected Resource(s)

sagemaker endpoints w/ inference components

Output of pulumi about

CLI
Version 3.115.2
Go Version go1.22.2
Go Compiler gc

Plugins
KIND NAME VERSION
resource archive 0.0.5
resource aws 6.33.1
resource aws-native 1.17.0
resource command 0.10.0
resource datadog 4.28.1
language nodejs unknown
resource random 4.16.1

Host
OS darwin
Version 12.5
Arch arm64

This project is written in nodejs: executable='/opt/homebrew/bin/node' version='v22.2.0'

Backend
Token type personal

Dependencies:
NAME VERSION
eslint-config-prettier 9.1.0
eslint-plugin-promise 6.1.1
ts-node 10.9.2
typescript-json-schema 0.65.1
typescript 4.9.5
@pulumi/aws-native 1.17.0
@typescript-eslint/eslint-plugin 6.21.0
crypto-js 4.2.0
jszip 3.10.1
lint-staged 14.0.1
@types/mocha 10.0.6
@pulumi/command 0.10.0
@types/ajv 1.0.0
eslint-config-standard-with-typescript 39.1.1
eslint 8.56.0
proxyquire 2.1.3
sinon 15.2.0
@aws-sdk/client-sts 3.577.0
@pulumi/archive 0.0.5
@pulumi/aws 6.33.1
chai 4.4.1
eslint-plugin-import 2.29.1
@aws-sdk/credential-providers 3.577.0
@aws-sdk/client-sagemaker 3.578.0
@pulumi/pulumi 3.115.2
@pulumi/random 4.16.1
ajv 8.12.0
husky 8.0.3
prettier 3.2.5
@aws-sdk/client-elastic-load-balancing-v2 3.509.0
@types/node 16.18.79
@types/sinon 10.0.20
@typescript-eslint/parser 6.21.0
@types/chai 4.3.11
eslint-plugin-n 16.6.2
js-yaml 4.1.0
mocha 10.3.0
@aws-sdk/client-route-53 3.509.0
@aws-sdk/types 3.502.0
@pulumi/datadog 4.28.1
@types/crypto-js 4.2.2
@types/js-yaml 4.0.9
aws-sdk-client-mock 3.0.1
aws-sdk 2.1623.0
yaml 2.3.4
@aws-sdk/client-resource-groups-tagging-api 3.509.0

Pulumi locates its logs in /var/folders/bw/69jmnfd15ds9yh1r1l9s74b00000gn/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

Thanks for the analysis @isaac-smothers this sounds pretty unfortunate.

The reason this property is required in Pulumi is because it is specified as a required property in terraform-provider-aws:

https://github.com/hashicorp/terraform-provider-aws/blob/c38d7c284d0684653d53452eb1f9dd3e65b905fd/internal/service/sagemaker/endpoint_configuration.go#L334

It looks like it is coming from 2022:

hashicorp/terraform-provider-aws@f789e1a#diff-da5946c33131d2431de063014445ec281a6381f36d7a229902a6fa4aa22bef12

If model_name should not be a required property, the easiest way for the Pulumi provider to land this change is by opening an issue upstream and seeing if TF provider can make this property optional and still work as expected and pass tests. That'd be the ideal outcome. In this case the Pulumi provider will land the functionality in a regular update.

[isaac-smothers]

Looks like inference components were introduced in late 2023 so the timeline makes sense. I'll take a look at updating the TF provider. @t0yv0 Say I get a fix merged upstream, how long would I be looking at waiting for pulumi to propagate that change?

Alternatively, do you know if anyone is working on implementing endpoints / endpoint configuration within the aws-native provider? Could be an opportunity to get this stuff off terraform entirely.

[t0yv0]

Next day typically after an upstream release.

[t0yv0]

within the aws-native provider? Could be an opportunity to get this stuff off terraform entirely.

That's a great question - I have not looked specifically for sagemaker endpoints but my team maintains both this provider and https://github.com/pulumi/pulumi-aws-native - typically if the resource is not available in pulumi-aws-native, the root cause is that AWS CloudControl API is not able to support it yet. The place to look is the backlog at https://github.com/aws-cloudformation/cloudformation-coverage-roadmap - and perhaps upvote or add a request there.

[isaac-smothers]

just for documentation purposes hashicorp/terraform-provider-aws#40644

[isaac-smothers]

the root cause is that AWS CloudControl API is not able to support it yet. The place to look is the backlog at https://github.com/aws-cloudformation/cloudformation-coverage-roadmap - and perhaps upvote or add a request there.

Great to know. Looks like it isn't supported by their API yet :( Looks like I'll be going down the route of getting this fixed upstream, thanks.