diff --git a/themes/default/content/docs/pulumi-cloud/esc/_index.md b/themes/default/content/docs/esc/_index.md similarity index 82% rename from themes/default/content/docs/pulumi-cloud/esc/_index.md rename to themes/default/content/docs/esc/_index.md index bedd52408a2..274ba5c431a 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/_index.md +++ b/themes/default/content/docs/esc/_index.md @@ -3,22 +3,27 @@ title: Pulumi ESC title_tag: Pulumi ESC (Environments, Secrets, and Configuration) h1: Pulumi ESC (Environments, Secrets, and Configuration) meta_desc: Pulumi ESC allows you to compose and manage hierarchical collections of configuration and secrets and consume them in various ways. +meta_image: /images/docs/meta-images/docs-meta.png menu: - pulumicloud: + pulumiesc: + name: Overview identifier: esc - weight: 4 + weight: 5 aliases: - /docs/esc/ + - /docs/pulumi-cloud/esc/ --- Pulumi ESC (Environments, Secrets, and Configuration) allows teams to tackle secrets and configuration complexity for modern cloud environments, alleviating maintenance burden and reducing costly mistakes, and creating a "secure by default" posture. Pulumi ESC is a new category of configuration as code product, motivated by our experience working with hundreds of Pulumi IaC customers to address their needs in managing secrets and configuration at scale within their Pulumi infrastructure and across other cloud applications and infrastructure projects. Pulumi ESC enables teams to aggregate secrets and configuration from many sources into a composable collection called an [environment](/docs/concepts/environments/). Teams can then consume those configuration and secrets from a variety of different infrastructure and application services. Pulumi ESC works hand-in-hand with Pulumi IaC to simplify configuration management, as well as a standalone CLI and API for other use cases apart from Pulumi IaC. -Pulumi ESC is offered as a fully managed cloud service in Pulumi Cloud (and Pulumi Cloud Self-hosted in the near future). The pulumi/esc project is open source, and contains the evaluation engine for environments, the esc CLI, and in the future, the extensible plugins for source and target integrations. +Pulumi ESC is offered as a fully managed cloud service in [Pulumi Cloud](/docs/pulumi-cloud/) (and Pulumi Cloud Self-hosted in the near future). The [pulumi/esc project](https://github.com/pulumi/esc) is open source, and contains the evaluation engine for environments, the esc CLI, and in the future, the extensible plugins for source and target integrations. ![Pulumi ESC ecosystem](img/pulumi_esc.png) +The following details corresponds to the numbered sections in the above diagram: + 1. Pulumi ESC enables you to define environments, which contain collections of secrets and configuration. Each environment can be composed from multiple environments. 2. Pulumi ESC supports a variety of configuration and secrets sources, and it has an extensible plugin model that allows third-party sources. @@ -29,7 +34,7 @@ Pulumi ESC is offered as a fully managed cloud service in Pulumi Cloud (and Pulu ## Dynamic Secrets Providers -Support for dynamic configuration providers allow Pulumi ESC to integrate with secrets stored in any other provider. Organizations often use AWS OIDC, AWS Secrets Manager, Vault, Azure OIDC, Azure KeyVault, GCP OIDC, and GCP Secrets Manager plus many more sources of truth for their secrets and configuration. Pulumi ESC supports them all, providing a single interface to your configuration and secrets, no matter where their source of truth is. Pulumi ESC works with these tools to provide improved management of secrets and configuration. +Support for dynamic configuration providers allow Pulumi ESC to [integrate with secrets stored in any other provider](/docs/esc/providers/). Organizations often use AWS OIDC, AWS Secrets Manager, Vault, Azure OIDC, Azure KeyVault, GCP OIDC, and GCP Secrets Manager plus many more sources of truth for their secrets and configuration. Pulumi ESC supports them all, providing a single interface to your configuration and secrets, no matter where their source of truth is. Pulumi ESC works with these tools to provide improved management of secrets and configuration. Teams can setup OIDC in their cloud providers to allow Environments to retrieve dynamic short-lived credentials. They can also pull secrets from other secrets managers and vaults. @@ -45,7 +50,7 @@ Environments are defined as YAML documents which can describe how to project and ## Authentication and RBAC -Pulumi ESC brokers access to secrets and configuration that live in other systems, and so authentication and granular RBAC are critical to ensure robust access controls across your organization. Pulumi ESC leverages the same Pulumi Cloud identity, RBAC, Teams, SAML/SCIM and scoped access tokens that are used for Pulumi IaC today, extending these all to managing access to environments as well as Stacks. +Pulumi ESC brokers access to secrets and configuration that live in other systems, and so authentication and granular RBAC are critical to ensure robust access controls across your organization. Pulumi ESC leverages the same Pulumi Cloud identity, RBAC, Teams, SAML/SCIM and scoped access tokens that are used for Pulumi IaC today, extending these all to managing access to environments as well as [Stacks](/docs/concepts/stack/). Teams can create and control access to their environments. They can control who can update and preview environments, as well as who can open environments and retrieve their secrets. Audit logs let teams know who has changed or accessed configuration. diff --git a/themes/default/content/docs/pulumi-cloud/esc/environments.md b/themes/default/content/docs/esc/environments.md similarity index 99% rename from themes/default/content/docs/pulumi-cloud/esc/environments.md rename to themes/default/content/docs/esc/environments.md index 92b41235740..d78caf667bb 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/environments.md +++ b/themes/default/content/docs/esc/environments.md @@ -4,8 +4,7 @@ title_tag: Pulumi ESC Environments h1: Pulumi ESC Environments meta_desc: Pulumi ESC allows you to compose and manage hierarchical collections of configuration and secrets and consume them in various ways. menu: - pulumicloud: - parent: esc + pulumiesc: identifier: environments weight: 2 search: @@ -14,6 +13,8 @@ search: - environments - secrets - configuration +aliases: + - /docs/pulumi-cloud/esc/environments/ --- Pulumi ESC (Environments, Secrets, and Configuration) lets you define collections of configuration settings and secrets called _environments_ and use them in any application or service. Environments are YAML documents composed of static key-value pairs, programmatic expressions, dynamically retrieved values from supported providers including all major clouds through OpenID Connect (OIDC), and other Pulumi ESC environments. diff --git a/themes/default/content/docs/pulumi-cloud/esc/faq.md b/themes/default/content/docs/esc/faq.md similarity index 98% rename from themes/default/content/docs/pulumi-cloud/esc/faq.md rename to themes/default/content/docs/esc/faq.md index 87d0500b87b..db2cb9b8e44 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/faq.md +++ b/themes/default/content/docs/esc/faq.md @@ -5,8 +5,7 @@ title: FAQ h1: Pulumi ESC FAQs meta_image: /images/docs/meta-images/docs-meta.png menu: - pulumicloud: - parent: esc + pulumiesc: weight: 6 identifier: faq --- diff --git a/themes/default/content/docs/esc/get-started/_index.md b/themes/default/content/docs/esc/get-started/_index.md new file mode 100644 index 00000000000..40e6b1fa68f --- /dev/null +++ b/themes/default/content/docs/esc/get-started/_index.md @@ -0,0 +1,20 @@ +--- +title: Get started +title_tag: Get Started with Pulumi ESC (Environments, Secrets, and Configuration) +h1: Get Started with Pulumi ESC (Environments, Secrets, and Configuration) +meta_desc: Learn how to manage secrets and hierarchical configuration with Pulumi. +menu: + pulumiesc: + identifier: esc-get-started + weight: 1 +aliases: + - /docs/pulumi-cloud/esc/get-started/ +--- + +In a typical application or infrastructure development workflow, there's often a need to maintain multiple environments such as development, staging, and production. Each of these environments might have its own set of configuration values: API endpoints, database connection strings, third-party secrets, and more. + +Hardcoding these values or keeping them inside source code is a security risk and makes managing configurations complex. [Pulumi ESC (Environments, Secrets and Configuration)](/docs/esc/) offers a centralized store to manage configuration data, plain-text data, and secrets. + +In this tutorial, we’ll demonstrate how to use Pulumi ESC as well as the power of this service in managing configuration and secrets. + +{{< get-started-stepper >}} diff --git a/themes/default/content/docs/esc/get-started/begin.md b/themes/default/content/docs/esc/get-started/begin.md new file mode 100644 index 00000000000..c6e3259f3ca --- /dev/null +++ b/themes/default/content/docs/esc/get-started/begin.md @@ -0,0 +1,93 @@ +--- +title_tag: Before You Begin | Pulumi ESC +title: Before you begin +h1: "Pulumi ESC: Before you begin" +meta_desc: This page provides an overview on how to get started with Pulumi ESC. +weight: 2 +menu: + pulumiesc: + parent: esc-get-started + identifier: esc-get-started-begin +--- + +Before you get started using Pulumi ESC, let's run through a few quick steps to ensure your environment is set up correctly. + +### Create a Pulumi account + +Pulumi ESC is a service of Pulumi Cloud, meaning you will need to create a Pulumi account to be able to use it. To do so, navigate to the [Pulumi Cloud console](https://app.pulumi.com) and create a new account. Once created, you can [optionally create an access token](/docs/pulumi-cloud/access-management/access-tokens/). Doing so will provide you an alternative way to sign into the Pulumi Cloud via the CLI. The token can also be used to automate your usage of the Pulumi Cloud using the REST API. + +### Install the Pulumi ESC CLI + +{{< notes type="info" >}} +Pulumi ESC can be used with or without Pulumi IaC. This means that if you already have the [Pulumi IaC CLI](/docs/cli/) installed, you do not need to install the Pulumi ESC CLI, and you may substitute `pulumi env` anywhere you see the `esc env` command in the rest of this tutorial. +{{< /notes >}} + +Use the below option to install the Pulumi ESC CLI based on your operating system. + +{{< chooser os "macos,windows,linux" >}} + +{{% choosable os macos %}} + +```bash +$ brew update && brew install pulumi/tap/esc +``` + +{{% /choosable %}} + +{{% choosable os linux %}} + +```bash +$ curl -fsSL https://get.pulumi.com/esc/install.sh | sh +``` + +{{% /choosable %}} + +{{% choosable os windows %}} + +
+
+

Windows Binary Download

+

+amd64 +

+
+
+ +{{% /choosable %}} + +{{% /chooser %}} + +You can explore more installation options by visiting the [ESC installation docs](/docs/install/esc/). + +### Login to the ESC CLI + +Run the following command to log into the CLI: + +```bash +esc login +``` + +You will be prompted to log in to the Pulumi Cloud using either the browser or by optionally providing an access token. + +```bash +$ esc login +Manage your Pulumi ESC environments by logging in. +Run `esc --help` for alternative login options. +Enter your access token from https://app.pulumi.com/account/tokens + or hit to log in using your browser : +Logged in to https://api.pulumi.com/ as your-pulumi-org (https://app.pulumi.com/your-pulumi-org) +``` + +### [Optional] Configure OpenID Connect (OIDC) + +Pulumi supports [OpenID Connect (OIDC) integration](/docs/pulumi-cloud/oidc/) across various services including Pulumi ESC. OIDC enables secure interactions between Pulumi and cloud providers by leveraging signed, short-lived tokens issued by the Pulumi Cloud. Use one of the following guides below to configure OIDC between Pulumi ESC and your chosen cloud provider: + +- [OIDC Configuration for AWS](/docs/pulumi-cloud/oidc/aws/) +- [OIDC Configuration for Azure](/docs/pulumi-cloud/oidc/azure/) +- [OIDC Configuration for Google Cloud](/docs/pulumi-cloud/oidc/gcp/) + +This is an optional step that is not required to get started with Pulumi ESC. There are some steps in this series that will require OIDC configuration to complete, but that will be indicated on the relevant pages. + +In the next section, you will start your journey with Pulumi ESC by creating a new environment. + +{{< get-started-stepper >}} diff --git a/themes/default/content/docs/esc/get-started/create-environment.md b/themes/default/content/docs/esc/get-started/create-environment.md new file mode 100644 index 00000000000..d8e989338b3 --- /dev/null +++ b/themes/default/content/docs/esc/get-started/create-environment.md @@ -0,0 +1,59 @@ +--- +title_tag: Create a New Environment | Pulumi ESC +title: Create environment +h1: "Pulumi ESC: Create A New Environment" +meta_desc: This page provides an overview on how to create a new Pulumi ESC environment. +weight: 3 +menu: + pulumiesc: + parent: esc-get-started + identifier: esc-get-started-create-environment + +--- + +## Overview + +In Pulumi ESC, an environment is a collection of configuration intended to capture the configuration values needed to work with a particular environment. + +An environment can be created one of two ways: + +- via the Pulumi Cloud console +- via the CLI + +This tutorial will walk you through how to create a new environment. + +## Create an environment + +### Create via the console + +To create an environment via the console, navigate to [Pulumi Cloud](https://app.pulumi.com) and select the **Environments** link in the left-hand menu. + +You will be directed to the Environments landing page. To create a new environment, click the **Create Environment** button. Enter a name for your environment (e.g., `my-dev-environment` for a development environment) and then click **Create Environment**. You will then be directed to the environment definition page. + +{{< video title="Creating a new environment in the Pulumi ESC console" src="/docs/esc/get-started/esc-create-new-env.mp4" autoplay="true" loop="true" >}} + +### Create via the CLI + +To create an environment via the CLI, use the `esc env init` command as shown below, where `` is optional and defaults to your Pulumi Cloud username. + +```bash +esc env init [/] +``` + +Note that environment names must be unique within an organization and may only contain alphanumeric characters, hyphens, underscores, and periods. + +```bash +$ esc env init my-dev-environment +Environment created. +``` + +You can validate that your environment was created by running the `esc env ls` command which will list all of the environments that you have access to. + +```bash +$ esc env ls +myorg/test +``` + +In the next section, you will learn how to store configuration values and secrets in your environment. + +{{< get-started-stepper >}} diff --git a/themes/default/content/docs/esc/get-started/esc-add-env-values.mp4 b/themes/default/content/docs/esc/get-started/esc-add-env-values.mp4 new file mode 100644 index 00000000000..40d28788208 Binary files /dev/null and b/themes/default/content/docs/esc/get-started/esc-add-env-values.mp4 differ diff --git a/themes/default/content/docs/esc/get-started/esc-base-url.png b/themes/default/content/docs/esc/get-started/esc-base-url.png new file mode 100644 index 00000000000..916da64e9a1 Binary files /dev/null and b/themes/default/content/docs/esc/get-started/esc-base-url.png differ diff --git a/themes/default/content/docs/esc/get-started/esc-create-global-config.mp4 b/themes/default/content/docs/esc/get-started/esc-create-global-config.mp4 new file mode 100644 index 00000000000..397ec4802d5 Binary files /dev/null and b/themes/default/content/docs/esc/get-started/esc-create-global-config.mp4 differ diff --git a/themes/default/content/docs/esc/get-started/esc-create-new-env.mp4 b/themes/default/content/docs/esc/get-started/esc-create-new-env.mp4 new file mode 100644 index 00000000000..e4af30a1659 Binary files /dev/null and b/themes/default/content/docs/esc/get-started/esc-create-new-env.mp4 differ diff --git a/themes/default/content/docs/esc/get-started/esc-env-show-secrets.mp4 b/themes/default/content/docs/esc/get-started/esc-env-show-secrets.mp4 new file mode 100644 index 00000000000..e2a3c324dbe Binary files /dev/null and b/themes/default/content/docs/esc/get-started/esc-env-show-secrets.mp4 differ diff --git a/themes/default/content/docs/esc/get-started/esc-import-global-config.mp4 b/themes/default/content/docs/esc/get-started/esc-import-global-config.mp4 new file mode 100644 index 00000000000..c4bc0e5d216 Binary files /dev/null and b/themes/default/content/docs/esc/get-started/esc-import-global-config.mp4 differ diff --git a/themes/default/content/docs/esc/get-started/esc-open-env-view-values.mp4 b/themes/default/content/docs/esc/get-started/esc-open-env-view-values.mp4 new file mode 100644 index 00000000000..eb78e77b8ad Binary files /dev/null and b/themes/default/content/docs/esc/get-started/esc-open-env-view-values.mp4 differ diff --git a/themes/default/content/docs/esc/get-started/esc-open-env.mp4 b/themes/default/content/docs/esc/get-started/esc-open-env.mp4 new file mode 100644 index 00000000000..dbd70382d1c Binary files /dev/null and b/themes/default/content/docs/esc/get-started/esc-open-env.mp4 differ diff --git a/themes/default/content/docs/esc/get-started/esc-run-command.md b/themes/default/content/docs/esc/get-started/esc-run-command.md new file mode 100644 index 00000000000..6c15d76ba82 --- /dev/null +++ b/themes/default/content/docs/esc/get-started/esc-run-command.md @@ -0,0 +1,190 @@ +--- +title_tag: Run Commands Without Local Secrets | Pulumi ESC +title: Run commands without local secrets +h1: "Pulumi ESC: Run Commands Without Local Secrets" +meta_desc: This page provides an overview on how to run commands without using local secrets using the "esc run" command. +weight: 6 +menu: + pulumiesc: + parent: esc-get-started + identifier: esc-get-started-esc-run-command +--- + +## Overview + +The Pulumi ESC CLI also has a [`run` command](/docs/esc-cli/commands/esc_run/) that enables you to run other commands using environment variables without having to locally set the environment variables first. For example, let's say that you have a CI/CD pipeline that will automatically push blog post updates to a content-management system (CMS). Instead of storing the values of the CMS endpoint and the corresponding environment's API key locally, you can configure your pipeline to retrieve these values from your environment file before running the command to update the post. + +## Expose environment variables + +Values defined in your environment file are not exposed as environment variables by default. You can expose them by adding your key-value pairs under a second-level key labeled `environmentVariables`: + +```yaml +values: + environmentVariables: # Configuration will be exported to the provided environment variables. + myEnvVarKey: myEnvVarValue +``` + +## Run commands with static secrets + +Following the format above, add the following configuration to your environment file: + +```yaml +values: + environmentVariables: + ENDPOINT_URL: "https://wordsapiv1.p.rapidapi.com/" + API_KEY: + fn::secret: "my-api-key-1234567" +``` + +Then run the following command to echo the value of `API_KEY`, which should be empty: + +```bash +# The output should not return anything +$ echo $ENDPOINT_URL $API_KEY + +``` + + Now run the command using `esc run` as shown below, making sure to replace and with the names of your own Pulumi organization and environment respectively + +```bash +esc run / -- bash -c "echo \$ENDPOINT_URL \$API_KEY" +``` + +{{< notes type="info" >}} +Commands run using `esc run` are not run in a subshell. This means that any commands that reference an environment variable like in the example shown above are not expanded by default. The `bash -c` portion of the command is what invokes the command inside a shell with environment variable expansion. See the [`esc run` documentation for the ESC CLI](/docs/esc-cli/commands/esc_run/) for more information. +{{< /notes >}} + +Because you have stored the value of `API_KEY` [as a secret](/docs/esc/get-started/store-and-retrieve-secrets/), your output will resemble the following: + +```bash +$ esc run pulumi/my-dev-environment -- bash -c "echo \$ENDPOINT_URL \$API_KEY" +https://wordsapiv1.p.rapidapi.com/ [secret] +``` + +The CLI intentionally redacts the secret value when printing to the terminal. If you want to disable the redaction, add the `--interactive` or `-i` flag to the command as shown below: + +```bash +$ esc run pulumi/my-dev-environment -i -- bash -c "echo \$ENDPOINT_URL \$API_KEY" +https://wordsapiv1.p.rapidapi.com/ my-api-key-1234567 +``` + +Alternatively, if you need to be able to run multiple commands without always having to specify the above command string each time, you can run the following command to open up a subshell that will have access to your values in your environment file: + +```bash +$ esc run pulumi/my-dev-environment -i -- bash +``` + +From there, try running the `echo` command individually for each example variable: + +```bash +$ echo $API_KEY +my-api-key-1234567 + +$ echo $ENDPOINT_URL +https://wordsapiv1.p.rapidapi.com/ +``` + +You can close this subshell by running the `exit` command. + +## Run commands with dynamic credentials + +For supported cloud providers, the `esc run` command also enables you to run commands like `aws s3 ls` without having to manually configure provider credentials in your local environments. In this section, you will learn how to use Pulumi ESC with dynamically generated cloud credentials so that every provider command you run uses short-term, scoped credentials issued via OpenID Connect (OIDC). + +If you have not done so already, make sure you have [configured OIDC connectivity](/docs/esc/get-started/begin/#configure-openid-connect-oidc) between Pulumi and one of the below supported providers. + +{{< notes type="info" >}} +This functionality is currently not available for the Azure cloud provider. +{{< /notes >}} + +{{% chooser cloud "aws,gcp" / %}} + +{{% choosable cloud aws %}} + +First check that your local environment does not have any AWS credentials configured by running the `aws configure list` command as shown below: + +```bash +$ aws configure list + Name Value Type Location + ---- ----- ---- -------- + profile None None +access_key None None +secret_key None None + region None None +``` + +Then run the `aws s3 ls` command as normal. You should see the following response indicating that you do not have local credentials configured to run this command: + +```bash +$ aws s3 ls + +Unable to locate credentials. You can configure credentials by running "aws configure". +``` + +Now run the command using `esc run` as shown below, making sure to replace and with the names of your own Pulumi organization and environment respectively: + +```bash +esc run / -- aws s3 ls +``` + +You should be presented with a list of S3 buckets in the account associated with your credentials. + +```bash +# example command and output +esc run pulumi/my-dev-environment -- aws s3 ls + +2023-12-10 02:52:46 my-bucket-4a67543 +2023-11-16 21:37:40 my-bucket-4b1e6cb +2023-10-27 21:04:59 my-bucket-50da4ad +2023-11-02 18:57:36 my-bucket-51385eb +``` + +{{% /choosable %}} + +{{% choosable cloud gcp %}} + +First check that your local environment does not have any Google Cloud credentials configured by running the following command: + +```bash +$ gcloud auth revoke +``` + +Then run the `gcloud iam service-accounts list` command as normal. You should see the following response indicating that you do not have local credentials configured to run this command: + +```bash +$ gcloud iam service-accounts list +ERROR: (gcloud.iam.service-accounts.list) You do not currently have an active account selected. +Please run: + + $ gcloud auth login + +to obtain new credentials. + +If you have already logged in with a different account, run: + + $ gcloud config set account ACCOUNT + +to select an already authenticated account to use. +``` + +Now run the command using `esc run` as shown below, making sure to replace and with the names of your own Pulumi organization and environment respectively: + +```bash +esc run / -- gcloud iam service-accounts list +``` + +You should be presented with a list of Service Accounts in the account associated with your credentials. + +```bash +# example command and output +$ esc run pulumi/my-dev-environment -- gcloud iam service-accounts list + +DISPLAY NAME EMAIL DISABLED +service-account-1 service-account-1@my-project.iam.gserviceaccount.com False +service-account-2 service-account-2@my-project.iam.gserviceaccount.com False +``` + +{{% /choosable %}} + +In the next section, you will learn how to retrieve secret values from external sources. + +{{< get-started-stepper >}} diff --git a/themes/default/content/docs/esc/get-started/import-environments.md b/themes/default/content/docs/esc/get-started/import-environments.md new file mode 100644 index 00000000000..ea15c6e98b4 --- /dev/null +++ b/themes/default/content/docs/esc/get-started/import-environments.md @@ -0,0 +1,79 @@ +--- +title_tag: Import Environments | Pulumi ESC +title: Import environments +h1: "Pulumi ESC: Import Environments" +meta_desc: This page provides an overview on how to import environments in Pulumi ESC. +weight: 5 +menu: + pulumiesc: + parent: esc-get-started + identifier: esc-get-started-import-environments +--- + +## Overview + +There may be scenarios where the value you need to retrieve is stored in a different environment file. For example, imagine you are building a system that integrates with a third-party service such as: + +- a payment provider +- weather data provider +- a content-management system + +In the development environment, you might be integrating with the sandbox or development endpoint of the third-party service, while in the testing environment, you might be integrating with the production endpoint. Both endpoints may share the same base URL. + +![Image showing same base URL and different endpoints](/docs/esc/get-started/esc-base-url.png) + +Since this base URL would be the same across environments, it would be more efficient to define it once in one place rather than multiple times across multiple environment files. + +With Pulumi ESC, you can import other environments into your environment file and make use of the imported configuration values and secrets. Similar to `values`, `imports` is a top-level key you will need to define in the environment file, meaning the syntax to create an import looks like the following: + +```yaml +imports: + - environment-name-1 + - environment-name-2 +values: + ... + ... +``` + +## Import an environment + +To demonstrate how this works, you will need to create a second environment. For the purposes of this guide, we will name this new environment `app-global-config`. In this environment, replace the placeholder content with the following configuration: + +```yaml +values: + ENDPOINT_URL: "https://wordsapiv1.p.rapidapi.com/" +``` + +{{< video title="Creating a global config environment in the Pulumi Console" src="/docs/esc/get-started/esc-create-global-config.mp4" autoplay="true" loop="true" >}} + +Then, open your first environment (e.g. `my-dev-environment`) via the Pulumi console or the ESC CLI and add the following configuration to the top of your file: + +```yaml +imports: + - app-global-config +``` + +Your updated environment file should look similar to the following: + +```yaml +# Example contents of my-dev-environment +imports: + - app-global-config +values: + myEnvironment: "development" + myPassword: + fn::secret: + ciphertext: ZXNjeAA.... +``` + +{{< video title="Importing the global config environment in the Pulumi Console" src="/docs/esc/get-started/esc-import-global-config.mp4" autoplay="true" loop="true" >}} + +You should now see `"ENDPOINT_URL": "https://wordsapiv1.p.rapidapi.com/"` in the environment preview pane. This indicates that this value was successfully imported from the `app-global-config` environment, and you can now retrieve this value in the same way as the other values that are manually defined in this environment. + +{{% notes type="info" %}} +You can test this out by retrieving the imported value via the console or the CLI. Refer to the [Store and Retrieve Secrets guide](/docs/esc/get-started/store-and-retrieve-secrets/#retrieve-environment-values) for the steps on how to do this. +{{% /notes %}} + +In the next section, you will learn how to run local commands without manually configuring local secrets. + +{{< get-started-stepper >}} diff --git a/themes/default/content/docs/esc/get-started/store-and-retrieve-secrets.md b/themes/default/content/docs/esc/get-started/store-and-retrieve-secrets.md new file mode 100644 index 00000000000..9b0c4532768 --- /dev/null +++ b/themes/default/content/docs/esc/get-started/store-and-retrieve-secrets.md @@ -0,0 +1,163 @@ +--- +title_tag: Store and Retrieve Secrets | Pulumi ESC +title: Store and retrieve secrets +h1: "Pulumi ESC: Store and Retrieve Secrets" +meta_desc: This page provides an overview on how to store and retrieve secrets in Pulumi ESC. +weight: 4 +menu: + pulumiesc: + parent: esc-get-started + identifier: esc-get-started-store-retrieve-secrets + +--- + +In an environment file, values are defined as a series of key-value pairs in YAML format. All variables will be defined under a top-level key named `values`. These values can be strings, numbers, or arrays, and they can be manually provided, dynamically generated from external sources, or referenced from other values in the file. They can also be stored in plain-text or as secrets. + +```yaml +values: + myKey1: "myValue1" + myNestedKey: + myKey2: "myValue2" + myNumber: 1 + myPassword: + fn::secret: + ciphertext: ZXN.... +``` + +You can store and retrieve values in an environment via the Pulumi Cloud console or via the CLI. + +## Store Environment Values + +### Store via the console + +To store values in your environment, first click on the name of the environment to open its definition editor. You will be presented with a split pane view. The left side is where you will write the definition of your environment configuration, and the right side will show a preview of your configuration in JSON format. + +{{< video title="Open environment in Pulumi ESC console" src="/docs/esc/get-started/esc-open-env.mp4" autoplay="true" loop="true" >}} + +Next, delete the placeholder text in the environment file and add the following simple configuration definition in its place: + +```yaml +values: + myEnvironment: "development" + myPassword: + fn::secret: "demo-password-123" +``` + +As shown above, you can specify that a value should be stored as a secret by using the `fn::secret` function. Once you have added the configuration, click the **Save** button located at the bottom of the editor. + +{{< video title="Adding values to the environment in the Pulumi ESC console" src="/docs/esc/get-started/esc-add-env-values.mp4" autoplay="true" loop="true" >}} + +The **Environment preview** pane on the right hand side will then update to show your added configuration in JSON format. You will notice that the value of "myPassword" has been hidden from view in both the defintion and preview panes. + +### Store via the CLI + +To store values or update an existing value via the CLI, use the `esc env set` command as shown below, where `` is optional and defaults to your Pulumi Cloud username: + +```bash +esc env set [/] +``` + +To demonstrate how this works, add the following simple configuration definition to your environment using the following command, making sure to replace the value of `my-dev-environment` with the name of your own environment: + +```bash +esc env set my-dev-environment myEnvironment development +esc env set my-dev-environment myPassword demo-password-123 --secret +``` + +As shown above, you can specify that a value should be stored as a secret by using the `--secret` flag. + +Alternatively, you can directly [edit your environment file with a code editor](/docs/pulumi-cloud/esc/environments/#with-the-pulumi-esc-cli) using the following command, making sure to replace `` with the name of your own environment (e.g. `my-dev-environment`): + +```bash +esc env edit +``` + +Using this method enables you to add your configuration values in the same way that you would [via the console](/docs/esc/get-started/store-and-retrieve-secrets/#store-via-the-console). + +## Retrieve Environment Values + +### Retrieve via the console + +To retrieve values in the console, scroll to the bottom of your environment page and click the **Open** button. This will return any statically defined plain-text values and definitions. + +{{< video title="Clicking the open button in the Pulumi ESC console" src="/docs/esc/get-started/esc-open-env-view-values.mp4" autoplay="true" loop="true" >}} + +As shown above, it does not return the value of secrets defined, nor does it resolve values that are dynamically generated from a provider. To view these values, you will need to click the **Show secrets** slider. + +{{< video title="Clicking the show secrets slider the Pulumi ESC console" src="/docs/esc/get-started/esc-env-show-secrets.mp4" autoplay="true" loop="true" >}} + +### Retrieve via the CLI + +The CLI has a built-in `get` command that enables you to retrieve a single value from your environment. The format of the full command looks like the following: + +```bash +esc env get [/] +``` + +To retrieve the value of the `myEnvironment` variable you created earlier, the command to do so would look like the following, making sure to replace the value of `my-dev-environment` with the name of your own environment: + +```bash +esc env get my-dev-environment myEnvironment +``` + +Running this command should return the following response: + +```bash +$ esc env get my-dev-environment myEnvironment + + Value + + "development" + + Definition + + development + + Defined at + + • my-dev-environment:2:8 +``` + +It is also possible to retrieve all values in an environment. To do so, run the `esc env get` command without specifying a value as shown below: + +```bash +esc env get my-dev-environment +``` + +Running this command should return the following response: + +```bash +$ esc env get my-dev-environment + + Value + + { + "myEnvironment": "development", + "myPassword": "[secret]" + } + + Definition + + values: + myEnvironment: "development" + myPassword: + fn::secret: + ciphertext: ZXNjeAA.... + +``` + +The `esc env get` command only returns statically defined plain-text values and definitions. This means that it does not return the value of any defined secrets, nor does it resolve values that are dynamically generated from a provider. To view these values, you must run the `esc env open` command as shown below. This will open the environment and resolve any secrets or dynamically retrieved values: + +```bash +$ esc env open my-dev-environment + +{ + "myEnvironment": "development", + "myPassword": "demo-password-123" +} + +``` + +In the next section, you will learn how to import configuration values from other environments. + +{{< get-started-stepper >}} diff --git a/themes/default/content/docs/pulumi-cloud/esc/img/pulumi_esc.png b/themes/default/content/docs/esc/img/pulumi_esc.png similarity index 100% rename from themes/default/content/docs/pulumi-cloud/esc/img/pulumi_esc.png rename to themes/default/content/docs/esc/img/pulumi_esc.png diff --git a/themes/default/content/docs/pulumi-cloud/esc/providers/_index.md b/themes/default/content/docs/esc/providers/_index.md similarity index 97% rename from themes/default/content/docs/pulumi-cloud/esc/providers/_index.md rename to themes/default/content/docs/esc/providers/_index.md index 3515616412f..68c46bd8cda 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/providers/_index.md +++ b/themes/default/content/docs/esc/providers/_index.md @@ -5,10 +5,11 @@ title: Providers h1: Pulumi ESC Providers meta_image: /images/docs/meta-images/docs-meta.png menu: - pulumicloud: - parent: esc + pulumiesc: weight: 5 identifier: esc-providers +aliases: +- /docs/pulumi-cloud/esc/providers/ --- Pulumi ESC providers enable you to dynamically import secrets and configuration from the provider into your environment. diff --git a/themes/default/content/docs/pulumi-cloud/esc/providers/aws-login.md b/themes/default/content/docs/esc/providers/aws-login.md similarity index 98% rename from themes/default/content/docs/pulumi-cloud/esc/providers/aws-login.md rename to themes/default/content/docs/esc/providers/aws-login.md index 9af7ebd1db0..ddbb450a129 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/providers/aws-login.md +++ b/themes/default/content/docs/esc/providers/aws-login.md @@ -5,10 +5,12 @@ title: aws-login h1: aws-login meta_image: /images/docs/meta-images/docs-meta.png menu: - pulumicloud: + pulumiesc: identifier: aws-login parent: esc-providers weight: 1 +aliases: +- /docs/pulumi-cloud/esc/providers/aws-login/ --- The `aws-login` provider enables you to log in to your AWS account using OpenID Connect or by providing static credentials. The provider will return a set of credentials that can be used to access AWS resources or fetch secrets using the `aws-secrets` provider. diff --git a/themes/default/content/docs/pulumi-cloud/esc/providers/aws-secrets.md b/themes/default/content/docs/esc/providers/aws-secrets.md similarity index 98% rename from themes/default/content/docs/pulumi-cloud/esc/providers/aws-secrets.md rename to themes/default/content/docs/esc/providers/aws-secrets.md index e4826c83814..ee85f2862b2 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/providers/aws-secrets.md +++ b/themes/default/content/docs/esc/providers/aws-secrets.md @@ -5,10 +5,12 @@ title: aws-secrets h1: aws-secrets meta_image: /images/docs/meta-images/docs-meta.png menu: - pulumicloud: + pulumiesc: identifier: aws-secrets parent: esc-providers weight: 2 +aliases: +- /docs/pulumi-cloud/esc/providers/aws-secrets/ --- The `aws-secrets` provider enables you to dynamically import Secrets from AWS Secrets Manager into your Environment. The provider will return a map of names to Secrets. diff --git a/themes/default/content/docs/pulumi-cloud/esc/providers/azure-login.md b/themes/default/content/docs/esc/providers/azure-login.md similarity index 97% rename from themes/default/content/docs/pulumi-cloud/esc/providers/azure-login.md rename to themes/default/content/docs/esc/providers/azure-login.md index ce80a5359e4..1b561475431 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/providers/azure-login.md +++ b/themes/default/content/docs/esc/providers/azure-login.md @@ -5,10 +5,12 @@ title: azure-login h1: azure-login meta_image: /images/docs/meta-images/docs-meta.png menu: - pulumicloud: + pulumiesc: identifier: azure-login parent: esc-providers weight: 3 +aliases: +- /docs/pulumi-cloud/esc/providers/azure-login/ --- The `azure-login` provider enables you to log in to Azure using OpenID Connect or by providing static credentials. The provider will return a set of credentials that can be used to access Azure resources or fetch secrets using the `azure-secrets` provider. diff --git a/themes/default/content/docs/pulumi-cloud/esc/providers/azure-secrets.md b/themes/default/content/docs/esc/providers/azure-secrets.md similarity index 98% rename from themes/default/content/docs/pulumi-cloud/esc/providers/azure-secrets.md rename to themes/default/content/docs/esc/providers/azure-secrets.md index ad211cfee70..353d59c3713 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/providers/azure-secrets.md +++ b/themes/default/content/docs/esc/providers/azure-secrets.md @@ -5,10 +5,12 @@ title: azure-secrets h1: azure-secrets meta_image: /images/docs/meta-images/docs-meta.png menu: - pulumicloud: + pulumiesc: identifier: azure-secrets parent: esc-providers weight: 4 +aliases: +- /docs/pulumi-cloud/esc/providers/azure-secrets/ --- The `azure-secrets` provider enables you to dynamically import Secrets and Configuration from Azure Key Vault into your Environment. The provider will return a map of names to Secrets. diff --git a/themes/default/content/docs/pulumi-cloud/esc/providers/gcp-login.md b/themes/default/content/docs/esc/providers/gcp-login.md similarity index 98% rename from themes/default/content/docs/pulumi-cloud/esc/providers/gcp-login.md rename to themes/default/content/docs/esc/providers/gcp-login.md index b6f9d268c79..47c3735b10a 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/providers/gcp-login.md +++ b/themes/default/content/docs/esc/providers/gcp-login.md @@ -5,10 +5,12 @@ title: gcp-login h1: gcp-login meta_image: /images/docs/meta-images/docs-meta.png menu: - pulumicloud: + pulumiesc: identifier: gcp-login parent: esc-providers weight: 5 +aliases: +- /docs/pulumi-cloud/esc/providers/gcp-login/ --- The `gcp-login` provider enables you to log in to Google Cloud using OpenID Connect or by providing static credentials. The provider will return a set of credentials that can be used to access Google Cloud resources or fetch secrets using the `gcp-secrets` provider. diff --git a/themes/default/content/docs/pulumi-cloud/esc/providers/gcp-secrets.md b/themes/default/content/docs/esc/providers/gcp-secrets.md similarity index 98% rename from themes/default/content/docs/pulumi-cloud/esc/providers/gcp-secrets.md rename to themes/default/content/docs/esc/providers/gcp-secrets.md index 4f03fbfb5ce..fab4ada9905 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/providers/gcp-secrets.md +++ b/themes/default/content/docs/esc/providers/gcp-secrets.md @@ -5,10 +5,12 @@ title: gcp-secrets h1: gcp-secrets meta_image: /images/docs/meta-images/docs-meta.png menu: - pulumicloud: + pulumiesc: identifier: gcp-secrets parent: esc-providers weight: 6 +aliases: +- /docs/pulumi-cloud/esc/providers/gcp-secrets/ --- The `gcp-secrets` provider enables you to dynamically import Secrets from Google Cloud Secrets Manager into your Environment. The provider will return a map of names to Secrets. diff --git a/themes/default/content/docs/pulumi-cloud/esc/providers/pulumi-stacks.md b/themes/default/content/docs/esc/providers/pulumi-stacks.md similarity index 94% rename from themes/default/content/docs/pulumi-cloud/esc/providers/pulumi-stacks.md rename to themes/default/content/docs/esc/providers/pulumi-stacks.md index a298cfc2616..c6e93a9bc78 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/providers/pulumi-stacks.md +++ b/themes/default/content/docs/esc/providers/pulumi-stacks.md @@ -5,10 +5,12 @@ title: pulumi-stacks h1: pulumi-stacks meta_image: /images/docs/meta-images/docs-meta.png menu: - pulumicloud: + pulumiesc: identifier: pulumi-stacks parent: esc-providers weight: 7 +aliases: +- /docs/pulumi-cloud/esc/providers/pulumi-stacks/ --- The `pulumi-stacks` provider enables you to import Stack outputs from Pulumi into your Environment. diff --git a/themes/default/content/docs/pulumi-cloud/esc/providers/vault-login.md b/themes/default/content/docs/esc/providers/vault-login.md similarity index 97% rename from themes/default/content/docs/pulumi-cloud/esc/providers/vault-login.md rename to themes/default/content/docs/esc/providers/vault-login.md index 8111e4f3205..511870f7604 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/providers/vault-login.md +++ b/themes/default/content/docs/esc/providers/vault-login.md @@ -5,10 +5,12 @@ title: vault-login h1: vault-login meta_image: /images/docs/meta-images/docs-meta.png menu: - pulumicloud: + pulumiesc: identifier: vault-login parent: esc-providers weight: 8 +aliases: +- /docs/pulumi-cloud/esc/providers/vault-login/ --- The `vault-login` provider enables you to log in to HashiCorp Vault using OpenID Connect or by providing static credentials. The provider will return a set of credentials that can be used to fetch secrets using the `vault-secrets` provider. diff --git a/themes/default/content/docs/pulumi-cloud/esc/providers/vault-secrets.md b/themes/default/content/docs/esc/providers/vault-secrets.md similarity index 98% rename from themes/default/content/docs/pulumi-cloud/esc/providers/vault-secrets.md rename to themes/default/content/docs/esc/providers/vault-secrets.md index 8046b428614..97fb3e59d5e 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/providers/vault-secrets.md +++ b/themes/default/content/docs/esc/providers/vault-secrets.md @@ -5,10 +5,12 @@ title: vault-secrets h1: vault-secrets meta_image: /images/docs/meta-images/docs-meta.png menu: - pulumicloud: + pulumiesc: identifier: vault-secrets parent: esc-providers weight: 9 +aliases: +- /docs/pulumi-cloud/esc/providers/vault-secrets/ --- The `vault-secrets` provider enables you to dynamically import Secrets from HashiCorp Vault into your Environment. The provider will return a map of names to Secrets. diff --git a/themes/default/content/docs/pulumi-cloud/esc/reference.md b/themes/default/content/docs/esc/reference.md similarity index 99% rename from themes/default/content/docs/pulumi-cloud/esc/reference.md rename to themes/default/content/docs/esc/reference.md index 87c80847359..c1aea3f8259 100644 --- a/themes/default/content/docs/pulumi-cloud/esc/reference.md +++ b/themes/default/content/docs/esc/reference.md @@ -4,9 +4,8 @@ title_tag: Syntax Reference h1: Pulumi ESC Syntax Reference meta_desc: Pulumi ESC allows you to compose and manage hierarchical collections of configuration and secrets and consume them in various ways. menu: - pulumicloud: + pulumiesc: identifier: reference - parent: esc weight: 4 --- diff --git a/themes/default/content/docs/pulumi-cloud/esc/get-started/_index.md b/themes/default/content/docs/pulumi-cloud/esc/get-started/_index.md deleted file mode 100644 index 1b733310c99..00000000000 --- a/themes/default/content/docs/pulumi-cloud/esc/get-started/_index.md +++ /dev/null @@ -1,618 +0,0 @@ ---- -title: Get started -title_tag: Get Started with Pulumi ESC (Environments, Secrets, and Configuration) -h1: Get Started with Pulumi ESC (Environments, Secrets, and Configuration) -meta_desc: Learn how to manage secrets and hierarchical configuration with Pulumi. -meta_image: /images/docs/meta-images/docs-meta.png -menu: - pulumicloud: - identifier: get-started - parent: esc - weight: 1 ---- - -In a typical development workflow, there's often a need to maintain multiple environments such as development, staging, and production. Each of these environments might have its own set of configuration values: API endpoints, database connection strings, third-party secrets, and more. - -Hardcoding these values or keeping them inside source code is a security risk and makes managing configurations complex. Pulumi ESC (Environments, Secrets and Configuration) offers a centralized store to manage configuration data, plain-text data, and secrets. - -In this tutorial, we’ll demonstrate the power of Pulumi ESC in managing configuration across multiple environments. - -## Prerequisites - -You will need the following tools to complete this tutorial: - -- A [Pulumi account](https://app.pulumi.com) - - [Optional] Create an [access token](/docs/pulumi-cloud/access-management/access-tokens/) -- The [Pulumi ESC CLI](/docs/install/esc/) -{{< notes type="info" >}} -Pulumi ESC is a service of Pulumi Cloud that can be used with or without Pulumi IaC. This means that if you already have the Pulumi IaC CLI installed, you do not need to install the Pulumi ESC CLI, and you may substitute `pulumi env` anywhere you see the `esc env` command in this guide. -{{< /notes >}} -- An [Amazon Web Services](https://aws.amazon.com/) account -- The [AWS CLI](https://aws.amazon.com/cli/) -- An [OIDC provider created for Pulumi](/docs/pulumi-cloud/oidc/aws/) in AWS - - Note that when defining the `Subject Identifier`, the format for environments is `pulumi:environments:org::env:` -- Python 3.7 or higher installed - -Let's get started! - -## Managing API Endpoints and API Keys - -Imagine you're building a system that integrates with a third-party service such as: - -- a payment provider -- weather data provider -- a content-management system - -In the development environment, you might be integrating with the sandbox or development endpoint of the third-party service, while in the testing environment, you might be integrating with the production endpoint. Each of these third-party services could potentially have different API endpoints and API keys, and having this separation of concerns enables developers to be able to interact with their environments without affecting real users, data, or API limits. - -In the next steps, you will deploy an example application that will simulate a third-party service with both a dev and test endpoint. - -### Deploy the Application - -Start by cloning the [sample application code](https://github.com/pulumi/tutorials/tree/pulumi-esc-get-started) locally to your machine. - -```bash -git clone -b pulumi-esc-get-started --single-branch https://github.com/pulumi/tutorials.git -``` - -Navigate to the root of the repo and deploy the application resources using the following commands: - -```bash -cd tutorials -chmod a+x cfn-deploy.sh -./cfn-deploy.sh -``` - -You will see an output similar to the following: - -```bash -... -... - -Waiting for changeset to be created.. -Waiting for stack create/update to complete -Successfully created/updated stack - pulumi-esc-tutorial-stack -Please see the below for your application output values: - -[ - [ - { - "OutputKey": "StackName", - "OutputValue": "pulumi-esc-tutorial-stack-10026875", - "Description": "The name of your CloudFormation stack" - }, - { - "OutputKey": "ApplicationEndpointUrl", - "OutputValue": "https://nixsrrftwh.execute-api.eu-central-1.amazonaws.com", - "Description": "The base endpoint URL for the sample application" - }, - { - "OutputKey": "DevEndpointUrl", - "OutputValue": "https://nixsrrftwh.execute-api.eu-central-1.amazonaws.com/dev", - "Description": "The URL path for the DEV service endpoint" - }, - { - "OutputKey": "TestEndpointUrl", - "OutputValue": "https://nixsrrftwh.execute-api.eu-central-1.amazonaws.com/test", - "Description": "The URL path for the TEST service endpoint" - }, - { - "OutputKey": "DevApiKeySecretName", - "OutputValue": "escDevApiKey", - "Description": "The Secrets Manager secret name for the DEV API key" - }, - { - "OutputKey": "TestApiKeySecretName", - "OutputValue": "escTestApiKey", - "Description": "The Secrets Manager secret name for the Test API key" - } - ] -] - -``` - -The API keys for this service have been randomly generated and stored as secrets in AWS Secrets Manager. - -The endpoint URLs as well as the names of the Secrets Manager secrets have been outputted for easy reference. You can also see these values in the `Outputs` tab of your CloudFormation stack in the AWS Console. - -Before moving on to the next part of the tutorial, you will want to go into the Secrets Manager console and retrieve those secret values. - -![Retrieving secret values from the AWS Console](./esc-retrieve-aws-secret.gif) - -#### Verify Application Endpoints - -Now that the sample application is deployed, you can test that the `dev` and `test` endpoints are working with a simple curl command. Copy and paste the following command into your terminal, replacing the placeholder text with the values relevant to your own application endpoint: - -```bash -curl -i --location '' \ ---header 'ApiKey: ' \ ---header 'Content-Type: application/json' \ ---data '{}' -``` - -You should see output similar to the following: - -```bash -$ curl -i --location 'https://nixsrrftwh.execute-api.eu-central-1.amazonaws.com/test' \ -> --header 'ApiKey: vayzbHHZ4BjohgrKX12E' \ -> --header 'Content-Type: application/json' \ -> --data '{}' -HTTP/2 200 -date: Thu, 05 Oct 2023 10:59:08 GMT -content-type: text/plain; charset=utf-8 -content-length: 47 -apigw-requestid: MUyHcj0UFiAEMtA= - -"You have reached the simulated TEST endpoint." -``` - -Now run the same curl command, but this time enter an incorrect value for the `ApiKey` header: - -```bash -$ curl -i --location 'https://nixsrrftwh.execute-api.eu-central-1.amazonaws.com/test' \ -> --header 'ApiKey: incorrect-api-key' \ -> --header 'Content-Type: application/json' \ -> --data '{}' -HTTP/2 403 -date: Thu, 05 Oct 2023 11:00:02 GMT -content-type: application/json -content-length: 23 -apigw-requestid: MUyP8gBqFiAEMlQ= - -{"message":"Forbidden"} -``` - -This is a simple way to quickly test that your endpoints are working for a single application, but when it comes to using and managing endpoints and API keys for real-world workloads at scale, this practice is not sustainable or secure. - -Many applications require access to configuration values or secrets, which often means those values need to be directly embedded or read from local environment variables or files. If you have multiple applications that use the same configuration details, and each application is referencing its own local environment files or variables, this can lead to a very error-prone process when it comes to updating or removing configuration values. - -With Pulumi ESC, you can centralize these values as a single source of truth because you will only need to make updates in one place instead of multiple places. Further, you can grant your applications permissions to directly fetch these values at runtime, ensuring that: - -- applications always have the latest values without manual intervention -- sensitive values are not unintentionally exposed - -In the next section, you will learn how to centralize and access your application configuration information by creating a Pulumi ESC environment. - -### Create an Environment - -In Pulumi ESC, an environment is a collection of configuration intended to capture the configuration needed to work with a particular environment. - -In an environment file, values are defined as a series of key-value pairs in YAML format. All variables will be defined under a top-level key named `values`. These values can be strings, numbers, or arrays, and they can be manually provided, dynamically generated from external sources, or referenced from other values in the file. - -```yaml -values: - myKey1: "myValue1" - myNestedKey: - myKey2: "myValue2" -``` - -In this section, we will be creating three separate environments: - -- `app-env-global` for our globally shared configuration -- `app-env-dev` for our development configuration -- `app-env-test` for our test configuration - -To do so, navigate to [Pulumi Cloud](https://app.pulumi.com) and select the `Environments` link in the left-hand menu. - -You will be directed to the Environments landing page. To create a new environment, click the `Create environment` button, and then enter a name for your environment (e.g., `app-env-global` for the shared configuration environment). - -![Creating a new environment in the Pulumi Cloud console](./esc-create-environment.gif) - -Repeat the same steps to create the environments for `app-env-dev` and `app-env-test`. Then, click on the name of the `app-env-global` environment to open its definition editor. - -In the editor, you will be presented with a split pane view. The left side is where you will write the definition of your environment configuration, and the right side will show a preview of your configuration in JSON format. - -The configuration that you will be adding to the `app-env-global` file is the base URL of your application endpoint. Because this URL will be the same across environments, you only need to define it in one place, and you can reference it from other environments accordingly. You will learn more about how to do this using `imports` later in this tutorial. - -Using the value of the `ApplicationEndpointUrl` from the CloudFormation output in the previous step, add the following details to the `app-env-global` environment file, making sure to replace the placeholder text with the actual value of your application endpoint URL: - -```yaml -values: - ENDPOINT_URL: "" -``` - -Scroll to the bottom of the page and click `Save`. - -The environment preview will update to show your added configuration similar to the following: - -![Adding configuration values to an environment](./esc-add-global-env-config.gif) - -Repeat the same steps to populate your development and test environment files with the following details: - -```yaml -values: - ENVIRONMENT: "dev" - API_KEY: "YOUR_DEV_API_KEY_HERE" -``` - -{{< notes type="info" >}} -Your `app-env-test` environment file should have a value of "test" for the `ENVIRONMENT` parameter, and the value of `API_KEY` should be the auto-generated value of the `escTestApiKey` secret from Secrets Manager. -{{< /notes >}} - -### Retrieve Environment Values - -Now that you have populated your environment files, you can verify that your values have been successfully stored by retrieving them through the Pulumi ESC CLI. Start by running the following command to log into the CLI: - -```bash -esc login -``` - -You will be prompted to log in to the Pulumi Cloud using either the browser or by optionally providing an access token. - -```bash -$ esc login -Manage your Pulumi ESC environments by logging in. -Run `esc --help` for alternative login options. -Enter your access token from https://app.pulumi.com/account/tokens - or hit to log in using your browser : -Logged in to https://api.pulumi.com/ as your-pulumi-org (https://app.pulumi.com/your-pulumi-org) -``` - -The CLI has a built-in `get` command that enables you to retrieve a single value from your environment. The format of the full command looks like the following: - -```bash -esc env get / -``` - -Let's assume that your Pulumi organization is named `acme` and the environment that you want to retrieve values from is named `app-env-dev`. If you want to retrieve the value of the `API_KEY` variable, the command to do so would look like the following: - -```bash -esc env get acme/app-env-dev API_KEY -``` - -Running this command should return the value of the API key that you added to your environment file. The output will look similar to the following: - -```bash -$ esc env get acme/app-env-dev API_KEY -"M28zraZb2b42Fu0MD1CA" -``` - -### Importing Environments - -There may be scenarios where the value you need to retrieve is stored in a different environment file. For example, since your base application endpoint URL will be the same across environments, it would be more efficient to define it once in one place rather than multiple times across multiple environment files. - -With Pulumi ESC, you can import other environments into your environment file and make use of the imported configuration values and secrets. Similar to `values`, `imports` is a top-level key you will need to define in the environment file, meaning the syntax to create an import looks like the following: - -```yaml -imports: - - environment-name-1 - - environment-name-2 -values: - ... - ... -``` - -In both your `app-env-dev` and `app-env-test` environment files, add the following code to import the `app-env-global` environment: - -```yaml -imports: - - app-env-global -``` - -When you save the file, the environment preview will update to show that `ENDPOINT_URL` is now included in the list of accessible variables in this environment. - -```yaml -{ - "API_KEY": "M28zraZb2b42Fu0MD1CA", - "ENDPOINT_URL": "https://nixsrrftwh.execute-api.eu-central-1.amazonaws.com", - "ENVIRONMENT": "dev" -} -``` - -You can validate that the import was successful by running the `esc env get` command against the `ENDPOINT_URL` variable as shown below: - -```bash -$ esc env get acme/app-env-dev ENDPOINT_URL -"https://nixsrrftwh.execute-api.eu-central-1.amazonaws.com" -``` - -### Dynamically Generate Environment Values - -For the purposes of this tutorial, you've manually added the values of your API keys to your `dev` and `test` environment files. However, this is not the recommended best practice when it comes to the storing, management, and retrieval of sensitive values. - -With Pulumi ESC, you can dynamically retrieve those values directly from a provider. In this next section, you will learn how to configure your environment file to retrieve secret values from AWS Secrets Manager. - -{{< notes >}} -This step assumes that you already have [Pulumi configured as an OIDC provider in AWS](/docs/pulumi-cloud/esc/providers/#setting-up-oidc). -{{< /notes >}} - -In both your `app-env-dev` and `app-env-test` environment files, update your code to the following: - -```yaml -# Example for the app-env-dev environment file -imports: - - app-env-global -values: - aws: - login: - fn::open::aws-login: - oidc: - roleArn: - sessionName: pulumi-environments-session - duration: 1h - secrets: - fn::open::aws-secrets: - region: - login: ${aws.login} - get: - api-key: - secretId: escDevApiKey - ENVIRONMENT: "dev" - API_KEY: ${aws.secrets.api-key} -``` - -Make sure to do the following: - -- replace the placeholder text for the `roleArn` parameter with the ARN of your OIDC role -- provide a value for `duration` no greater than the [maximum session duration](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) of your OIDC role -- provide the name of the AWS region that you deployed your sample application into -- provide the relevant values for `secretId` and `ENVIRONMENT` in the corresponding environment file - -As shown with the `${aws.login}` and `${aws.secrets.api-key}` references above, values within Pulumi ESC environments, as well as its imports, can be referenced through the use of interpolations. Below is an example of what this would look like in the console: - -![Adding AWS provider integration into an environment](./esc-aws-provider-integration.png) - -You can verify your provider connection and the retrieval of your secret from AWS Secrets Manager by running the `esc open acme/app-env-dev API_KEY` command. - -{{< notes >}} -To see examples for how to accomplish this with other providers, take a look at the [syntax reference documentation for Pulumi ESC](/docs/pulumi-cloud/esc/reference). -{{< /notes >}} - -### Exposing Values as Environment Variables - -The Pulumi ESC CLI also has a `run` command that enables you to run other commands using environment variables (for example the `aws s3 ls` command) without having to locally set the environment variables first. This is great for when you have other components or applications that may need to interact with your endpoints as part of a larger application workflow. - -For example, if you have a CI/CD pipeline that will automatically push blog post updates to a content-management system, you can enable it to retrieve the endpoint and API key specific to its environment and run the command to update the post. - -However, values in your environment file are not exposed as environment variables by default. You can expose them by adding your key-value pairs under a second-level key labeled `environmentVariables`: - -```yaml -values: - environmentVariables: # Configuration will be exported to the provided environment variables. - myEnvVarKey: myEnvVarValue -``` - -Following the above format, add the `environmentVariables` key to your `app-env-dev` and `app-env-test` environment files. Then, move your `ENDPOINT_URL`, `ENVIRONMENT` and `API_KEY` variables so that they are nested underneath it before saving your file. - -```yaml -# Example for the app-env-dev environment file -imports: - - app-env-global -values: - aws: - login: - fn::open::aws-login: - oidc: - roleArn: - sessionName: pulumi-environments-session - duration: 1h - secrets: - fn::open::aws-secrets: - region: - login: ${aws.login} - get: - api-key: - secretId: escDevApiKey - environmentVariables: # Add this variable - ENDPOINT_URL: ${ENDPOINT_URL} # explicitly referenced to be made accessible as an environment variable - ENVIRONMENT: "dev" - API_KEY: ${aws.secrets.api-key} -``` - -Now take a look at the contents of the `validate-endpoints.sh` bash script. - -```bash -#! /bin/sh - -esc run acme/app-env-dev \ --- bash -c "python3 test-endpoint.py \$ENDPOINT_URL \$ENVIRONMENT \$API_KEY" -``` - -This script executes a Python file that is meant to simulate an application that interacts with your endpoint. - -As seen above, the `ENDPOINT_URL`, `ENVIRONMENT`, and `API_KEY` values that you defined in your environment file are referenced directly in the command as environment variables, without needing to explicitly set them locally and without needing to use the `esc env get` command to retrieve their values inline. - -Replace the value of `acme/app-env-dev` with the name of your Pulumi organization and desired environment and save the file. - -Then run the following commands to execute the script: - -```bash -chmod a+x validate-endpoint.sh -./validate-endpoint.sh -``` - -When providing the name of your `app-env-dev` environment file, you should see a response similar to the following: - -``` bash -$ ./validate-endpoints.sh -"You have reached the simulated DEV endpoint." -``` - -If you run the command with `app-env-test`, it will state `You have reached the simulated TEST endpoint.` instead. - -## Integrating with Pulumi IaC - -{{< notes >}} -This section of the tutorial is optional and requires the installation of the [Pulumi IaC CLI](/docs/cli/) and one of [Pulumi's supported language runtimes](/docs/languages-sdks/) as a prerequisite. -{{< /notes >}} - -Pulumi ESC works independently of [Pulumi Infrastructure as Code (IaC)](/docs/get-started/), providing developers the flexibility to centrally manage their environment configuration regardless of how or where those environments are created. - -Pulumi ESC also integrates seamlessly with Pulumi IaC, and this next section will demonstrate how to leverage Pulumi ESC in your own Pulumi programs. This works everywhere, including Pulumi Deployments and GitHub Actions, without any additional work or changes. - -### Create and Configure a New Project - -To start, [create a new Pulumi AWS project](/docs/clouds/aws/get-started/create-project/) and [ensure it is configured to use your AWS account](/registry/packages/aws/installation-configuration/). - -During the creation of your project, you will typically be prompted for some configuration values for the stack. - -```bash -$ pulumi new aws-python -This command will walk you through creating a new Pulumi project. - -Enter a value or leave blank to accept the (default), and press . -Press ^C at any time to quit. - -project name (pulumi-python): -project description (A minimal AWS Python Pulumi program): -Created project 'pulumi-python' - -Please enter your desired stack name. -To create a stack in an organization, use the format / (e.g. `acmecorp/dev`). -stack name (dev): zephyr/esc-dev -Created stack 'zephyr/esc-dev' - -aws:region: The AWS region to deploy into (us-east-1): us-east-2 -Saved config - -Installing dependencies... - -Finished installing dependencies - -Your new project is ready to go! - -To perform an initial deployment, run `pulumi up` -``` - -For AWS projects, you are prompted for the AWS region. - -This value is stored in a file called `Pulumi..yaml`. - -```yaml -config: - aws:region: us-east-2 -``` - -Defining the configuration this way may be fine when dealing with a singular project, but it can become very challenging to maintain securely and consistently across multiple projects. Centralizing these configuration values provides more scalability without increasing administrative overhead along the way. - -### Centralize the Configuration - -To centralize this configuration for your Pulumi program, you will need to add a second-level key named `pulumiConfig` in your environment file that will expose the values nested underneath it to Pulumi IaC. - -```yaml -values: - pulumiConfig: - aws:region: us-east-2 -``` - -Then return to your `Pulumi..yaml` file and update it to import your environment as shown below: - -```yaml -environment: - - app-env-dev -``` - -Make sure to update the value in the `environment` section with the name of your own environment before saving the file. - -Then run the `pulumi up` command. - -```bash -$ pulumi up -y -Previewing update (zephyr/esc-dev) - -View in Browser (Ctrl+O): https://app.pulumi.com/zephyr/pulumi-python/esc-dev/previews/8bfb476b-7ef8-4843-ac0f-4f663577243f - - Type Name Plan Info - + pulumi:pulumi:Stack pulumi-python-esc-dev create - + └─ aws:s3:Bucket my-bucket create - -Outputs: - bucket_name: output - -Resources: - + 2 to create - -Updating (zephyr/esc-dev) - -View in Browser (Ctrl+O): https://app.pulumi.com/zephyr/pulumi-python/esc-dev/updates/5 - - Type Name Status Info - + pulumi:pulumi:Stack pulumi-python-esc-dev created (6s) - + └─ aws:s3:Bucket my-bucket created (2s) - -Outputs: - bucket_name: "my-bucket-6302f8e" - -Resources: - + 2 created - -Duration: 7s -``` - -If you view your resource in the AWS console, you will see that your program resources will get created in the region you specified in your environment file. - -![Showing AWS S3 bucket properties](./esc-show-s3-region.png) - -### Access Configuration from Code - -You can also reference the `pulumiConfig` environment values directly from within your Pulumi program in the same way that you would access them from the project's config file. - -```yaml -# Example environment File -values: - pulumiConfig: - myKey: "Test value" -``` - -```python -# Example Pulumi Python Program -import pulumi - -config = pulumi.Config() -myValue = config.get("myKey") - -pulumi.export('Value', myValue) -``` - -```bash -# Example Program Output -$ pulumi up -y -Previewing update (zephyr/esc-dev) - -View in Browser (Ctrl+O): https://app.pulumi.com/zephyr/pulumi-python/esc-dev/previews/d82a61b3-5405-4f6d-9ac7-336257f5a25f - - Type Name Plan Info - + pulumi:pulumi:Stack pulumi-python-esc-dev create - -Outputs: - Value: "Test value" - -Resources: - + 1 to create - -Updating (zephyr/esc-dev) - -View in Browser (Ctrl+O): https://app.pulumi.com/zephyr/pulumi-python/esc-dev/updates/13 - - Type Name Status Info - + pulumi:pulumi:Stack pulumi-python-esc-dev created (0.51s) - -Outputs: - Value: "Test value" - -Resources: - + 1 created - -Duration: 2s -``` - -See the Pulumi documentation on [Accessing Configuration from Code](https://www.pulumi.com/docs/concepts/config/#code) for more details. - -## Clean Up - -{{< cleanup >}} - -Then run the following command from the root of the sample application folder to delete the AWS resources, replacing the placeholder text with the name of your CloudFormation stack as provided in your stack outputs: - -```bash -aws cloudformation delete-stack --stack-name -``` - -## Next Steps - -In this tutorial, you created and retrieved values from a Pulumi environment. You also exposed and programmatically consumed values as environment variables and as Pulumi configuration. - -To learn more about managing configuration and secrets in Pulumi, take a look at the following resources: - -- Learn more about [environments](/docs/concepts/environments/), [secrets](/docs/concepts/secrets/), and [configuration](/docs/concepts/config/) in the Pulumi documentation. -- Learn more about defining Pulumi ESC environment files in the [Pulumi ESC YAML Syntax Reference](/docs/pulumi-cloud/esc/reference). diff --git a/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-add-global-env-config.gif b/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-add-global-env-config.gif deleted file mode 100644 index f8187bbf1c7..00000000000 Binary files a/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-add-global-env-config.gif and /dev/null differ diff --git a/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-aws-provider-integration.png b/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-aws-provider-integration.png deleted file mode 100644 index 420b488f948..00000000000 Binary files a/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-aws-provider-integration.png and /dev/null differ diff --git a/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-create-environment.gif b/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-create-environment.gif deleted file mode 100644 index 71234e7f259..00000000000 Binary files a/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-create-environment.gif and /dev/null differ diff --git a/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-retrieve-aws-secret.gif b/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-retrieve-aws-secret.gif deleted file mode 100644 index f9e4b102646..00000000000 Binary files a/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-retrieve-aws-secret.gif and /dev/null differ diff --git a/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-show-s3-region.png b/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-show-s3-region.png deleted file mode 100644 index de5e1763391..00000000000 Binary files a/themes/default/content/docs/pulumi-cloud/esc/get-started/esc-show-s3-region.png and /dev/null differ diff --git a/themes/default/layouts/partials/docs/main-nav.html b/themes/default/layouts/partials/docs/main-nav.html index 6df5701ace6..94f163f32b5 100644 --- a/themes/default/layouts/partials/docs/main-nav.html +++ b/themes/default/layouts/partials/docs/main-nav.html @@ -2,8 +2,8 @@ {{ if hasPrefix .RelPermalink "/docs/" }} {{/* Render a top-level menu for our major docs sections */}} {{ $toc_page := . }} - {{ $toc_sections := slice "Download & install" "Get started" "Clouds" "Concepts" "Pulumi Cloud" "Using Pulumi" "Languages & SDKs" "Pulumi CLI" "Pulumi ESC CLI" "Support" }} - {{ $toc_menus := dict "Download & install" .Site.Menus.install "Get started" .Site.Menus.getstarted "Clouds" .Site.Menus.clouds "Concepts" .Site.Menus.concepts "Pulumi Cloud" .Site.Menus.pulumicloud "Using Pulumi" .Site.Menus.usingpulumi "Languages & SDKs" .Site.Menus.languages "Pulumi CLI" .Site.Menus.cli "Pulumi ESC CLI" .Site.Menus.esc_cli "Support" .Site.Menus.support }} + {{ $toc_sections := slice "Download & install" "Get started" "Clouds" "Concepts" "Pulumi Cloud" "Pulumi ESC" "Using Pulumi" "Languages & SDKs" "Pulumi CLI" "Pulumi ESC CLI" "Support" }} + {{ $toc_menus := dict "Download & install" .Site.Menus.install "Get started" .Site.Menus.getstarted "Clouds" .Site.Menus.clouds "Concepts" .Site.Menus.concepts "Pulumi Cloud" .Site.Menus.pulumicloud "Pulumi ESC" .Site.Menus.pulumiesc "Using Pulumi" .Site.Menus.usingpulumi "Languages & SDKs" .Site.Menus.languages "Pulumi CLI" .Site.Menus.cli "Pulumi ESC CLI" .Site.Menus.esc_cli "Support" .Site.Menus.support }} {{ range $index, $element := $toc_sections }} {{ $sidenav_selected := "" }} diff --git a/themes/default/layouts/shortcodes/esc-get-started-note.html b/themes/default/layouts/shortcodes/esc-get-started-note.html index c5f12ad5df3..c3b18d26412 100644 --- a/themes/default/layouts/shortcodes/esc-get-started-note.html +++ b/themes/default/layouts/shortcodes/esc-get-started-note.html @@ -6,7 +6,7 @@

For a streamlined Pulumi ESC walkthrough, including language runtime installation and cloud configuration, see the - Get Started guide. + Get Started guide.