Paginating S3 objects using boto3

When using Boto you can only List 1000 objects per request. So to obtain all the objects in the bucket. You can use s3’s paginator. To use paginator you should first have a client instance

client = boto3.Session.client( service_name = "s3", region_name=<region-name>
                                                aws_access_key_id=<access-id>, aws_secret_access_key=<secret-key>
                                              )

This initiates a client object which can be used for Boto3 Operations

Using client object we can start a list_object instance

paginator = client.get_paginator( "list_objects" )
    page_iterator = paginator.paginate( Bucket = bucket_name, Prefix = prefix )

This will return a paginator Object which we can iterate with for loop and use for Further Operations. For Instance, to create a List of Bucket Object Keys we can do it as

bucket_object_list = []
for page in page_iterator:
    if "Contents" in page:
        for key in page[ "Contents" ]:
            keyString = key[ "Key" ]
            bucket_object_list.append(keyString)

The article was originally published at MicroPyramid blog