Skip to content

Commit

Permalink
Fix S3Session for pagination
Browse files Browse the repository at this point in the history
According with AWS SDK doc the nextMarker field is only returned when defining the delimiter query param:


>    NextMarker
    When the response is truncated (the IsTruncated element value in the response is true), you can use the key name in this field as the marker parameter in the subsequent request to get the next set of objects. Amazon S3 lists objects in alphabetical order.
    Note
    This element is returned only if you have the delimiter request parameter specified. If the response does not include the NextMarker element and it is truncated, you can use the value of the last Key element in the response as the marker parameter in the subsequent request to get the next set of object keys.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjects.html#API_ListObjects_ResponseElements

After upgrading my project to newest version of `spring-integration-aws` I've started facing this loop issue for buckets with more than `1000` items, where `isTruncated` is true and `nextMarker` is `null`.

* Update the `S3Session` code to use the `key` from last item as `nextMarker` in next the request when response `isTruncated`, avoiding infinite loop.
  • Loading branch information
rogeriolino authored Nov 14, 2023
1 parent 266407e commit 33f1061
Showing 1 changed file with 11 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
* @author Jim Krygowski
* @author Anwar Chirakkattil
* @author Xavier François
* @author Rogerio Lino
*/
public class S3Session implements Session<S3Object> {

Expand Down Expand Up @@ -82,8 +83,11 @@ public S3Object[] list(String path) {
List<S3Object> objectSummaries = new ArrayList<>();
do {
objectListing = this.amazonS3.listObjects(listObjectsRequest.build());
objectSummaries.addAll(objectListing.contents());
listObjectsRequest.marker(objectListing.nextMarker());
List<S3Object> contents = objectListing.contents();
objectSummaries.addAll(contents);
if (objectListing.isTruncated()) {
listObjectsRequest.marker(contents.get(contents.size() - 1).key());
}
}
while (objectListing.isTruncated());

Expand All @@ -108,10 +112,13 @@ public String[] listNames(String path) {
List<String> names = new ArrayList<>();
do {
objectListing = this.amazonS3.listObjects(listObjectsRequest.build());
for (S3Object objectSummary : objectListing.contents()) {
List<S3Object> contents = objectListing.contents();
for (S3Object objectSummary : contents) {
names.add(objectSummary.key());
}
listObjectsRequest.marker(objectListing.nextMarker());
if (objectListing.isTruncated()) {
listObjectsRequest.marker(contents.get(contents.size() - 1).key());
}
}
while (objectListing.isTruncated());

Expand Down

0 comments on commit 33f1061

Please sign in to comment.