You will finish able to tell a one-byte-past-the-end read from real state corruption using nothing but the offset printed in the error, with a reproducer that fires the bug on a laptop with no AWS account. The offset is the diagnosis, and on an s3 backend it usually exonerates the file your runbook is about to overwrite.

Short version: Terraform's S3 backend calls HeadObject, allocates a buffer the exact size of the reported ContentLength, downloads into it, and discards the byte count the download returns. If another writer replaces the object with a smaller one between the HEAD and the GET, the tail of that buffer stays zero-filled, the JSON decoder stops at the first 0x00, and you get Unsupported state file format followed by syntax error at byte offset <N> where N is one past the end of a perfectly good state document. This was filed as hashicorp/terraform#39440 on 23 October 2026 against v1.17.0 and v1.14.7, with a standalone reproducer, and it is still open.

Every page currently ranking for this error answers with BOM characters, truncated writes, hand edits and "restore terraform.tfstate.backup". On a remote S3 backend that advice destroys a real apply to fix a read.

Prerequisites

  • Terraform 1.6.0 through 1.17.0 on the s3 backend. Print it with terraform version; the error banner also prints it, for example Terraform v1.17.0 on darwin_arm64. If you are mid-upgrade to 1.17, note the separate behaviour change in Terraform 1.17's -minimal-refresh.
  • AWS CLI v2, jq, xxd, and Python 3 (stdlib only). The reproducer needs no AWS account. The forensic half needs s3:GetObject, s3:ListBucketVersions and s3:GetObjectVersion on the state bucket.
  • The exact byte offset from your error, unrounded and unparaphrased.
  • Ideally, S3 server access logging already enabled on the state bucket and delivering to a second bucket you can query with Athena.

Step-by-step

1. Read the offset, then read the object size

# from the error: "syntax error at byte offset 10732171."
OFFSET=10732171
aws s3api head-object --bucket my-tf-state --key env/prod/terraform.tfstate \
  --query 'ContentLength' --output text

If that prints 10732170, which is OFFSET - 1, stop looking for corruption. The object in S3 is one byte shorter than the position where your decoder choked, which is only possible if the bytes you parsed did not all come from this object.

Here is the window that produces that arithmetic:

HeadObject returns ContentLength 10732228Backend allocates a byte slice of 10732228Another writer replaces the object with 10732170 bytesDownloader writes 10732170 bytes and returns that countBackend discards the count and keeps the full sliceIs dynamodb_table setMD5 mismatch, retry for 10 seconds, clean read58 trailing NUL bytes reach the JSON decodersyntax error at byte offset 10732171yesno

The allocation and the managed downloader both live in internal/backend/remote-state/s3/client.go; the decoder that reports the offset is in internal/states/statefile/read.go.

2. Prove the stored object is valid JSON

aws s3api get-object --bucket my-tf-state --key env/prod/terraform.tfstate /tmp/state.json
jq -e 'has("version") and has("serial")' /tmp/state.json
wc -c /tmp/state.json

jq -e exits 0 and wc -c matches the ContentLength from step 1. You now have two artifacts saying the file is intact while terraform plan calls it unparseable, which puts this in the same family as binary/octet-stream on a 1.16.3 provider zip.

3. Name the shrink event from object versioning

The S3 backend docs say "It is highly recommended that you enable Bucket Versioning on the S3 bucket to allow for state recovery." Here that setting is the evidence trail rather than the recovery path.

aws s3api list-object-versions --bucket my-tf-state \
  --prefix env/prod/terraform.tfstate \
  --query 'Versions[0:4].[LastModified,Size,VersionId]' --output table

Look for adjacent versions where the newer one is smaller. In the filed CI case the pair was 10732228 bytes then 10732170 bytes, a 58-byte shrink, and the error landed at 10732171. State shrinks on terraform destroy, terraform state rm, a reduced count, or a provider dropping attributes from its schema.

4. Reproduce it with no AWS account

Point the backend at a local HTTP server that answers the HEAD with the large object's length and the GET with a smaller body.

terraform {
  backend "s3" {
    bucket                      = "repro"
    key                         = "terraform.tfstate"
    region                      = "us-east-1"
    access_key                  = "test"
    secret_key                  = "test"
    use_path_style              = true
    skip_credentials_validation = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    endpoints = { s3 = "http://127.0.0.1:9000" }
  }
}

Those are the current argument names: endpoints.s3 rather than the deprecated top-level endpoint, and use_path_style rather than force_path_style. Run a control pass where HEAD and GET agree, confirm Terraform reads the state and exits 0, then flip the server to answer the GET with a body 58 bytes shorter.

5. Confirm the padding is NUL, not garbage

printf '%s' "$(cat /tmp/served-body.json)" | xxd | tail -2
python3 -I -c 'import sys;d=open(sys.argv[1],"rb").read();print(len(d), d.count(b"\x00"))' /tmp/served-body.json

The body your fake server sent is OFFSET - 1 bytes long and contains zero NULs. The NULs exist only inside Terraform's over-sized buffer, which is why no byte of the object in S3 ever needs repairing.

6. Pull the two access-log records

CloudTrail does not record object size. The S3 server access log format does, in the Operation, Key, HTTP status, Bytes Sent and Object Size fields.

SELECT requestdatetime, operation, httpstatus, bytessent, objectsize, useragent
FROM s3_access_logs
WHERE key LIKE '%env/prod/terraform.tfstate'
  AND operation IN ('REST.HEAD.OBJECT','REST.GET.OBJECT')
  AND requestdatetime BETWEEN <t-5s> AND <t+5s>
ORDER BY requestdatetime;

The signature is a REST.HEAD.OBJECT with Object Size = N followed inside the same second by a REST.GET.OBJECT with Object Size = M, M smaller than N, and your error offset equal to M + 1. On a bucket without access logging there is no retroactive way to confirm any of this: the lock table does not record it and the Terraform error carries no request ID.

7. Restore the digest check your migration removed

terraform {
  backend "s3" {
    bucket         = "my-tf-state"
    key            = "env/prod/terraform.tfstate"
    region         = "us-east-1"
    use_lockfile   = true
    dynamodb_table = "tf-locks"   # keep the digest while this bug is open
  }
}

Get() compares the downloaded payload's MD5 against the digest stored in DynamoDB and retries for consistencyRetryTimeout, 10 seconds, polling every 2. The retry re-downloads clean bytes. But getMD5() returns nil, nil when c.ddbTable == "", so with use_lockfile = true and no table there is no digest, no comparison and no retry, and the padded buffer goes to the parser untouched. The same backend docs mark dynamodb_table Deprecated and say DynamoDB-based locking "will be removed in a future minor version", so the migration everyone is being told to complete deletes a short-read check that nothing replaces. Setting both is the documented migration path and Terraform acquires both locks.

Second mitigation: treat this specific error as retryable in CI rather than restoring anything. Third, and this one I have not verified: a terraform_remote_state data source and terraform state pull read the object, and whether either takes the lock is not something I have confirmed against internal/states/remote. Test it in your own pipeline before you build a policy on it.

Verify it works

Run the control and race cases back to back and compare exit codes:

terraform init -reconfigure && terraform plan ; echo "exit=$?"

Control exits 0 with a state length equal to the served body. The race exits 1 with Unsupported state file format and syntax error at byte offset <served_len + 1>.. Then add dynamodb_table and run the same race: the failure changes to state data in S3 does not have the expected content and self-heals inside the 10-second retry window instead of reaching the decoder. Finish with jq -e 'has("version")' /tmp/state.json passing on both runs, against the file nobody needed to restore.

For the version floor, the issue attributes the pre-allocated buffer to the aws-sdk-go-v2 migration in v1.6.0, replacing a plain GetObject read with io.Copy into a growable bytes.Buffer. I have not diffed the two tags myself, so check it before quoting it:

git clone --filter=blob:none https://github.com/hashicorp/terraform /tmp/tf-src
git -C /tmp/tf-src diff v1.5.7 v1.6.0 -- internal/backend/remote-state/s3/client.go

Common pitfalls

  • Restoring terraform.tfstate.backup or an older S3 version. This is the destructive move: it rolls the serial backwards and discards a real apply to fix a read.
  • terraform state push after a hand edit. Same error family, genuinely different cause, and the offset separates them. Offset 1 means the file does not start as JSON, usually a BOM or an HTML error page saved over it. Offset equal to object size plus one means this bug.
  • Calling the rerun a fix. Reruns succeed because the window is narrow, which is how this stayed invisible across ten minor releases.
  • Assuming the lock protects reads. The lock serialises writers.
  • Pinning only the AWS provider. This read path is in Terraform's own backend package, so a provider upgrade changes nothing.

Wrap-up

Watch #39440 for either fix the reporter proposed: slicing the buffer down to the count Download returns, or pinning the HEAD and GET to one object with VersionId or an ETag IfMatch. Delete the dynamodb_table line once a release carries one of them, and not before.

One note from 2026-10-22, on an EOL Node 20 box: I checked a patch claim with a behavioural probe instead of node -v, and the decoder's own errno=-9 settled what the version string could not, the way the Node Brotli RCE version string did. The state file's bytes are ground truth here, and the error text names the wrong victim.