fizz.today

GovCloud root credential delivery requires a KMS policy you don’t have

We signed up for GovCloud and waited three weeks for root credentials that never arrived.

How RCR works

AWS will happily create you a GovCloud account you can’t use yet. The account exists, but you have no root credentials. To get them, you open a support case, re-affirm in writing that you’re a US citizen (the click-through agreement wasn’t enough), create an asymmetric KMS key in the linked commercial account, and ask AWS to run a Root Credential Rotation (RCR). RCR encrypts the root credentials with your KMS key and delivers the blob. You decrypt it locally.

Five support cases across three accounts

This is not a common flow. Most AWS support agents will never encounter an RCR request, and they’re rarely working the case directly — they escalate internally, and every round-trip costs a business day. We had an upgraded support plan on each of three accounts and it still took three weeks. The first case was on the wrong account. The second referenced the wrong key. The third got a well-meaning response that the credentials were “already delivered” — they weren’t. Each tiny variation in our setup kicked us back to square one.

The fifth case landed on Marco, who actually pulled up the RCR process logs and found the answer: AWS’s credential delivery service runs from account 536883072436. That account needs kms:Encrypt on our key. Our key policy only granted access to our own account. One statement fixed it:

{
  "Sid": "Allow AWS GovCloud RCR process to encrypt",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::536883072436:root"
  },
  "Action": "kms:Encrypt",
  "Resource": "*"
}

Two minutes to apply. Root credentials arrived the next day.

Beware of the Leopard

AWS does document this. The cross-account KMS requirement is in the GovCloud credential rotation docs — in the way that the plans for the demolition of Arthur Dent’s house were on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying “Beware of the Leopard.” It’s there. You won’t find it by following the getting-started guide. You’ll find it after your fifth support case, when an engineer named Marco actually reads the RCR process logs.

The RCR process fails silently — no error in the console, no email, no CloudTrail event on your side. It just never delivers. We spent days checking Secrets Manager in our account, assuming AWS would place the root credentials there as a permanent secret. They don’t — the credentials land in a one-time-access self-burning secret in AWS’s credential delivery account, not yours. But none of that matters if RCR can’t encrypt them in the first place. Maybe that’s documented too — ask the leopard, he knows.

If you’re setting up GovCloud and your root credentials haven’t arrived after 48 hours, check the KMS key policy. Account 536883072436 needs kms:Encrypt. That’s the entire problem.

#aws #govcloud #kms #iam