How to remove multiple S3 buckets at once?

Question

I have a dozen of buckets that I would like to remove in AWS S3, all having a similar name containing bucket-to-remove and with some objects in it.

Using the UI is quite slow, is there a solution to remove all these buckets quickly using the CLI?

Answer 1

You could try this sample line to delete all at once. Remember that this is highly destructive, so I hope you know what you are doing:

for bucket in $(aws s3 ls | awk '{print $3}' | grep my-bucket-pattern); do  aws s3 rb "s3://${bucket}" --force ; done

You are done with that. May take a while depending on the amount of buckets and their content.

Answer 2

I did this

aws s3api list-buckets \
   --query 'Buckets[?starts_with(Name, `bucket-pattern `) == `true`].[Name]' \
   --output text | xargs -I {} aws s3 rb s3://{} --force

Then update the bucket pattern as needed.

Be careful though, this is a pretty dangerous operation.

Answer 3

The absolute easiest way to bulk delete S3 buckets is to not write any code at all. I use Cyberduck to browse my S3 account and delete buckets and their contents quite easily.

Answer 4

Using boto3 you cannot delete buckets that have objects in it thus you first need to remove the objects before deleting the bucket. The easiest solution is a simple Python script such as:

import boto3
import botocore
import json

s3_client = boto3.client(
    "s3",
    aws_access_key_id="<your key id>",
    aws_secret_access_key="<your secret access key>"
)

response = s3_client.list_buckets()
for bucket in response["Buckets"]:
    # Only removes the buckets with the name you want.
    if "bucket-to-remove" in bucket["Name"]:
        s3_objects = s3_client.list_objects_v2(Bucket=bucket["Name"])
        # Deletes the objects in the bucket before deleting the bucket.
        if "Contents" in s3_objects:
            for s3_obj in s3_objects["Contents"]:
                rm_obj = s3_client.delete_object(
                    Bucket=bucket["Name"], Key=s3_obj["Key"])
                print(rm_obj)
        rm_bucket = s3_client.delete_bucket(Bucket=bucket["Name"])
        print(rm_bucket)

Answer 5

Here is a windows solution.

First test the filter before you delete

aws s3 ls ^| findstr "<search here>"

and then execute

for /f "tokens=3" %a in ('<copy the correct command between the quotes>') do aws s3 rb s3://%a --force

Answer 6

According to the S3 docs you can remove a bucket using the CLI command aws s3 rb only if the bucket does not have versioning enabled. If that's the case you can write a simple bash script to get the bucket names and delete them one by one, like:

#!/bin/bash

# get buckets list => returns the timestamp + bucket name separated by lines
S3LS="$(aws s3 ls | grep 'bucket-name-pattern')" 

# split the lines into an array. @see https://stackoverflow.com/a/13196466/6569593
oldIFS="$IFS"
IFS='
'
IFS=${IFS:0:1}
lines=( $S3LS )
IFS="$oldIFS"

for line in "${lines[@]}"
    do
        BUCKET_NAME=${line:20:${#line}} # remove timestamp
        aws s3 rb "s3://${BUCKET_NAME}" --force
done

Be careful to don't remove important buckets! I recommend to output each bucket name before actually remove them. Also be aware that the aws s3 rb command takes a while to run, because it recursively deletes all the objects inside the bucket.

Answer 7

For deleting all s3 buckets in you account use below technique, It's work very well using local

Step 1 :- export your profile using below command Or you can export access_key and secrete_access_key locally as well

 export AWS_PROFILE=<Your-Profile-Name>  

Step 2:- Use below python code, Run it on local and see your all s3 buckets will delete.

import boto3

client = boto3.client('s3', Region='us-east-2')
response = client.list_buckets()
for bucket in response['Buckets']:
    s3 = boto3.resource('s3')
    s3_bucket = s3.Bucket(bucket['Name'])
    bucket_versioning = s3.BucketVersioning(bucket['Name'])
    if bucket_versioning.status == 'Enabled':
        s3_bucket.object_versions.delete()
    else:
        s3_bucket.objects.all().delete()
    response = client.delete_bucket(Bucket=bucket['Name'])

If you see error like boto3 not found please go to link and install it Install boto3 using pip

Answer 8

I have used lambda for deleting buckets with the specified prefix.

It will delete all the objects regardless versioning is enabled or not.

Note that: You should give appropriate S3 access to your lambda.

import boto3

s3_client = boto3.client('s3')
s3 = boto3.resource('s3')
def lambda_handler(event, context):
    bucket_prefix = "your prefix"
    response = s3_client.list_buckets()
    for bucket in response["Buckets"]:
        # Only removes the buckets with the name you want.
        if bucket_prefix in bucket["Name"]:
            s3_bucket = s3.Bucket(bucket['Name'])
            bucket_versioning = s3.BucketVersioning(bucket['Name'])
            if bucket_versioning.status == 'Enabled':
                s3_bucket.object_versions.delete()
            else:
                s3_bucket.objects.all().delete()
            response = s3_client.delete_bucket(Bucket=bucket['Name'])
    return { 
        'message' : f"delete buckets with prefix {bucket_prefix} was successfull"
    }

Answer 9

The best option that I find is to use the Cyberduck. You can select all the buckets from the GUI and delete them. I provide a screenshot for how to do it.

Answer 10

if you're using PowerShell, this will work:

Foreach($x in (aws s3api list-buckets --query 
'Buckets[?starts_with(Name, `name-pattern`) == 
`true`].[Name]'  --output text) 
{aws s3 rb s3://$x --force}