UPDATE_ROLLBACK_FAILED after pushing with insufficient permissions

[kolodi]

Before opening, please confirm:

• I have searched for duplicate or closed issues and discussions.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

Next.js

Amplify APIs

Authentication

Amplify Version

v6

Amplify Categories

auth

Backend

Amplify CLI

Environment information

  System:
    OS: Linux 5.15 Ubuntu 20.04.4 LTS (Focal Fossa)
    CPU: (12) x64 Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz
    Memory: 5.22 GB / 7.65 GB
    Container: Yes
    Shell: 5.8 - /usr/bin/zsh
  Binaries:
    Node: 18.20.2 - ~/.nvm/versions/node/v18.20.2/bin/node
    Yarn: 1.22.22 - ~/.nvm/versions/node/v18.20.2/bin/yarn
    npm: 10.5.0 - ~/.nvm/versions/node/v18.20.2/bin/npm
  npmPackages:
    @ampproject/toolbox-optimizer:  undefined ()
    @aws-amplify/ui-react: ^6.1.8 => 6.1.8 
    @aws-amplify/ui-react-internal:  undefined ()
    @babel/core:  undefined ()
    @babel/runtime:  7.22.5 
    @edge-runtime/cookies:  4.1.1 
    @edge-runtime/ponyfill:  2.4.2 
    @edge-runtime/primitives:  4.1.0 
    @hapi/accept:  undefined ()
    @mswjs/interceptors:  undefined ()
    @napi-rs/triples:  undefined ()
    @next/font:  undefined ()
    @opentelemetry/api:  undefined ()
    @react-three/drei: ^9.88.4 => 9.105.4 
    @react-three/fiber: ^8.15.4 => 8.16.2 
    @react-three/postprocessing: ^2.15.1 => 2.16.2 
    @tanstack/query-codemods:  4.24.3 
    @tanstack/react-query: ^5.28.6 => 5.29.2 
    @testing-library/jest-dom: ^6.4.2 => 6.4.2 
    @testing-library/react: ^15.0.2 => 15.0.2 
    @types/debounce: ^1.2.1 => 1.2.4 
    @types/deep-equal: ^1.0.1 => 1.0.4 
    @types/mime-types: ^2.1.1 => 2.1.4 
    @types/node: ^18.16.18 => 18.19.31 
    @types/react: ^18.2.13 => 18.2.79 
    @types/react-copy-to-clipboard: ^5.0.4 => 5.0.7 
    @types/react-currency-format: ^1.0.0 => 1.0.4 
    @types/react-dom: ^18.2.6 => 18.2.25 
    @types/react-modal: ^3.16.0 => 3.16.3 
    @types/react-resizable: ^3.0.4 => 3.0.7 
    @types/three: ^0.152.1 => 0.152.1 (0.163.0)
    @vercel/nft:  undefined ()
    @vercel/og:  0.6.2 
    acorn:  undefined ()
    amphtml-validator:  undefined ()
    anser:  undefined ()
    arg:  undefined ()
    assert:  undefined ()
    async-retry:  undefined ()
    async-sema:  undefined ()
    autoprefixer: ^10.4.14 => 10.4.19 
    aws-amplify: ^6.0.28 => 6.0.28 
    aws-amplify/adapter-core:  undefined ()
    aws-amplify/analytics:  undefined ()
    aws-amplify/analytics/kinesis:  undefined ()
    aws-amplify/analytics/kinesis-firehose:  undefined ()
    aws-amplify/analytics/personalize:  undefined ()
    aws-amplify/analytics/pinpoint:  undefined ()
    aws-amplify/api:  undefined ()
    aws-amplify/api/server:  undefined ()
    aws-amplify/auth:  undefined ()
    aws-amplify/auth/cognito:  undefined ()
    aws-amplify/auth/cognito/server:  undefined ()
    aws-amplify/auth/enable-oauth-listener:  undefined ()
    aws-amplify/auth/server:  undefined ()
    aws-amplify/data:  undefined ()
    aws-amplify/data/server:  undefined ()
    aws-amplify/datastore:  undefined ()
    aws-amplify/in-app-messaging:  undefined ()
    aws-amplify/in-app-messaging/pinpoint:  undefined ()
    aws-amplify/push-notifications:  undefined ()
    aws-amplify/push-notifications/pinpoint:  undefined ()
    aws-amplify/storage:  undefined ()
    aws-amplify/storage/s3:  undefined ()
    aws-amplify/storage/s3/server:  undefined ()
    aws-amplify/storage/server:  undefined ()
    aws-amplify/utils:  undefined ()
    aws-sdk: ^2 => 2.1601.0 
    babel-packages:  undefined ()
    browserify-zlib:  undefined ()
    browserslist:  undefined ()
    buffer:  undefined ()
    bytes:  undefined ()
    ci-info:  undefined ()
    cli-select:  undefined ()
    client-only:  0.0.1 
    clsx: ^2.0.0 => 2.1.0 (1.2.1)
    commander:  undefined ()
    comment-json:  undefined ()
    compression:  undefined ()
    conf:  undefined ()
    constants-browserify:  undefined ()
    content-disposition:  undefined ()
    content-type:  undefined ()
    cookie:  undefined ()
    cross-spawn:  undefined ()
    crypto-browserify:  undefined ()
    css.escape:  undefined ()
    data-uri-to-buffer:  undefined ()
    debounce: ^1.2.1 => 1.2.1 
    debug:  undefined ()
    deep-equal: ^2.2.1 => 2.2.3 
    devalue:  undefined ()
    domain-browser:  undefined ()
    dotenv: ^16.3.1 => 16.4.5 
    edge-runtime:  undefined ()
    eslint: ^8.43.0 => 8.57.0 
    eslint-config-next: ^13.4.7 => 13.5.6 
    events:  undefined ()
    fflate: ^0.8.2 => 0.8.2 (0.6.10)
    find-cache-dir:  undefined ()
    find-up:  undefined ()
    fresh:  undefined ()
    get-orientation:  undefined ()
    glob: ^10.3.0 => undefined (7.2.3, 7.1.7, 10.3.12, )
    graphql: ^16.8.1 => 16.8.1 (15.8.0)
    gzip-size:  undefined ()
    http-proxy:  undefined ()
    http-proxy-agent:  undefined ()
    https-browserify:  undefined ()
    https-proxy-agent:  undefined ()
    icss-utils:  undefined ()
    ignore-loader:  undefined ()
    image-size:  undefined ()
    is-animated:  undefined ()
    is-docker:  undefined ()
    is-wsl:  undefined ()
    jest: ^29.7.0 => 29.7.0 
    jest-environment-jsdom: ^29.7.0 => 29.7.0 
    jest-worker:  undefined ()
    json5:  undefined ()
    jsonwebtoken:  undefined ()
    leva: ^0.9.35 => 0.9.35 
    loader-runner:  undefined ()
    loader-utils:  undefined ()
    local-ssl-proxy: ^2.0.5 => 2.0.5 
    lodash.curry:  undefined ()
    lru-cache:  undefined ()
    mime-types: ^2.1.35 => 2.1.35 
    mini-css-extract-plugin:  undefined ()
    nanoid:  undefined ()
    native-url:  undefined ()
    neo-async:  undefined ()
    next: ^14 => 14.2.2 
    node-fetch:  undefined ()
    node-html-parser:  undefined ()
    ora:  undefined ()
    os-browserify:  undefined ()
    p-limit:  undefined ()
    path-browserify:  undefined ()
    picomatch:  undefined ()
    platform:  undefined ()
    postcss: ^8.4.24 => 8.4.38 (8.4.31)
    postcss-flexbugs-fixes:  undefined ()
    postcss-modules-extract-imports:  undefined ()
    postcss-modules-local-by-default:  undefined ()
    postcss-modules-scope:  undefined ()
    postcss-modules-values:  undefined ()
    postcss-preset-env:  undefined ()
    postcss-safe-parser:  undefined ()
    postcss-scss:  undefined ()
    postcss-value-parser:  undefined ()
    process:  undefined ()
    punycode:  undefined ()
    qrcode.react: ^3.1.0 => 3.1.0 
    querystring-es3:  undefined ()
    raw-body:  undefined ()
    react: ^18 => 18.2.0 
    react-arborist: ^3.2.0 => 3.4.0 
    react-builtin:  undefined ()
    react-copy-to-clipboard: ^5.1.0 => 5.1.0 
    react-device-detect: ^2.2.3 => 2.2.3 
    react-dom: ^18 => 18.2.0 
    react-dom-builtin:  undefined ()
    react-dom-experimental-builtin:  undefined ()
    react-experimental-builtin:  undefined ()
    react-hotkeys-hook: ^4.4.1 => 4.5.0 
    react-icons: ^5 => 5.1.0 
    react-is:  18.2.0 
    react-modal: ^3.16.1 => 3.16.1 
    react-refresh:  0.12.0 
    react-resizable: ^3.0.5 => 3.0.5 
    react-server-dom-turbopack-builtin:  undefined ()
    react-server-dom-turbopack-experimental-builtin:  undefined ()
    react-server-dom-webpack-builtin:  undefined ()
    react-server-dom-webpack-experimental-builtin:  undefined ()
    regenerator-runtime:  0.13.4 
    sass-loader:  undefined ()
    scheduler-builtin:  undefined ()
    scheduler-experimental-builtin:  undefined ()
    schema-utils:  undefined ()
    semver:  undefined ()
    send:  undefined ()
    server-only:  0.0.1 
    setimmediate:  undefined ()
    shell-quote:  undefined ()
    source-map:  undefined ()
    source-map08:  undefined ()
    stacktrace-parser:  undefined ()
    stream-browserify:  undefined ()
    stream-http:  undefined ()
    string-hash:  undefined ()
    string_decoder:  undefined ()
    strip-ansi:  undefined ()
    superstruct:  undefined ()
    tailwindcss: ^3 => 3.4.3 
    tar:  undefined ()
    terser:  undefined ()
    text-table:  undefined ()
    three: ^0.157.0 => 0.157.0 
    three-stdlib: ^2.29.6 => 2.29.6 
    timers-browserify:  undefined ()
    ts-node: ^10.9.2 => 10.9.2 
    tty-browserify:  undefined ()
    typescript: ^5 => 5.4.5 
    ua-parser-js:  undefined ()
    unistore:  undefined ()
    use-resize-observer: ^9.1.0 => 9.1.0 
    usehooks-ts: ^2.9.1 => 2.16.0 
    util:  undefined ()
    vm-browserify:  undefined ()
    watchpack:  undefined ()
    web-vitals:  undefined ()
    webpack:  undefined ()
    webpack-sources:  undefined ()
    ws:  undefined ()
    zod:  undefined ()
    zustand: ^4 => 4.5.2 (3.7.2)
  npmGlobalPackages:
    @aws-amplify/cli: 12.11.1
    corepack: 0.25.2
    npm: 10.5.0

Describe the bug

I tied to promote changes from dev env to staging env. One of the changes was relative to the auth UserPools. In my staging environment there was an old credentials that had limited access to the Cognito service so the push failed due to insufficient permissions. Now the CloudFormation stack is in UPDATE_ROLLBACK_FAILED state. I tried to continue rollback with optional skips for all possible skip combinations and the all failed.
So now I am stuck and cannot push or do anything. I certainly do not want to delete the whole stack as there are DB and users data I don't want to lose.

Here is the detailed message from CloudFormation about the "root cause":
Currently in UPDATE_ROLLBACK_FAILED with reason: The following resource(s) failed to update: [UserPool].

Expected behavior

push can recover from this state automatically or at least rollback can be done in CloudFormation.

Reproduction steps

1. Make changed to auth
2. Push with AWS credentials having insufficient permissions for the operation on Cognito UserPools

Code Snippet

// Put your code below this line.

Log output

// Put your logs below this line

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

[chrisbonifacio]

Hi @kolodi thanks for raising this issue, I am going to transfer it to the Amplify CLI repo for better support

[kolodi]

UPDATE: Issue has been solved for me.
I have finally managed to rollback in CloudFormation. Among many resources to skip in advanced options there was 1 particular to select. You can try them one by one and see events log to see if it was a correct one to skip or not. After that the stack has been recovered and I can continue with amplify push. It failed once more then because I had added many new GSI to may models. So i went commenting all new indexes and adding them back one at the time and pushing many times.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.