AWS Secrets Manager [Secrets Rotation] Error : A previous rotation isn’t complete. That rotation will be reattempted.

Introduction

This article talks about an error encountered for AWS Secrets Manager – A previous rotation isn’t complete. That rotation will be reattempted.

Where is the error caused?

This error can be caused either –

• On AWS Console – If you are trying to rotate the secret on AWS manually, you might see the error prompting on console. See below –

• On AWS Cloud Formation – You can also face this if you are deploying or updating the secret using cloud formation template. See below –

Root-cause of the error

If the AWSPENDING label is present but not attached to the same version as AWSCURRENT, then any later invocation of rotation assumes that a previous rotation request is still in progress and returns an error.

https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html#rotate-secrets_how

Before discussing about the root-cause, let’s quickly understand how rotation works.

• AWS Secrets Manager uses a lambda function to rotate a secret. Every secret has a secret rotation lambda associated to it and when a secret rotation need to be rotated the lambda is called.
• This lambda performs the following steps –
  • Creates a new version of secret and labels this version as AWSPENDING
  • Update the credentials on the database or service for this new version of secret.
  • Tests this new version secret credentials.
  • In the same API call, it will change the this new version labelled as AWSPENDING to AWSCURRENT. It will also assign the the previous correct version before this newly created version from AWSCURRENT to AWSPREVIOUS
• Once all of this is done, secret rotation is successfully done.

Root-cause

While performing these steps sometimes the secret is stuck in the AWSPENDING state. This happens because –

• AWSPENDING status is assigned to an empty secret version.
• If the AWSPENDING label is present but not attached to the same version as AWSCURRENT, then any later invocation of rotation assumes that a previous rotation request is still in progress and returns an error.

Fix of the issue

Simple fix is to delete the AWSPENDING version, but the problem is you will not be able to see this AWSPENDING version on the AWS Console (UI). So you will have to do this using aws-cli (command line interface). If you have not setup aws-cli, I would recommend you to do follow this – here

To setup aws-cli easily if you already have the admin access to the aws account, run the ada-command on terminal –

~ % ada credentials update --account=<YOUR_AWS_ACCOUNT> --provider=conduit --role=<ADMIN_ACCESS_ROLE>

Once you have the aws-cli, simply run the following command on the terminal to list the secrets –

~ % aws secretsmanager list-secrets

Now you should be able to see the version with AWSPENDING status –

{
    "SecretList": [
        {
            "ARN": "<SECRET_ARN>",
            "Name": "<SECRET_NAME>",
            "Description": "<SECRET_DESCRIPTIOn>",
            "KmsKeyId": "<KMS_KEY_ID>",
            "RotationEnabled": true,
            "RotationLambdaARN": "<ROTATION_LAMBDA_ARN>",
            "RotationRules": {
                "AutomaticallyAfterDays": 30
            },
            "LastRotatedDate": "2020-02-09T21:12:54.131000+05:30",
            "LastChangedDate": "2020-04-09T21:12:54.140000+05:30",
            "LastAccessedDate": "2021-01-17T05:30:00+05:30",
            "NextRotationDate": "2021-01-31T05:29:59+05:30",
            "Tags": [// TAGS],
            "SecretVersionsToStages": {
                "72********************************1": [
                    "AWSPREVIOUS"
                ],
                "a5********************************5": [
                    "AWSCURRENT"
                ],
                "e6********************************0": [
                    "AWSPENDING"
                ]
....

Now, you can run the command to remove the AWSPENDING version –

~ % aws secretsmanager update-secret-version-stage --version-stage AWSPENDING --remove-from-version-id <YOUR_VERSION_ID> --secret-id <YOUR_SECRET_NAME>

If this command runs successfully, you will get the output –

{
    "ARN": "<SECRET_ARN>",
    "Name": "<SECRET_NAME>"
}

You can verify if the AWSPENDING version is removed by running the command ‘aws secretsmanager list-secrets’ again.

Thanks for reading. Hope this helps !

References :

AWS Documentation: https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html#rotate-secrets_how