Weird behavior for StreamingBody iter_lines

[devfox-se]

Describe the bug

StreamingBody.iter_lines() returns content from wrong offest on the second iteration.
Use the following file for this example: sample.txt

Expected Behavior

out1: Lorem ipsum dolor sit amet, consectetur adipiscing ... erat condimentum, lobortis libero eget, efficitur
out2: Cras commodo dui ante, ac maximus ipsum aliquam ut ... smod lectus. Praesent ut mi metus. Nulla facilisi.
out3: Duis sed est tristique, auctor odio vel, cursus nu ... r. Sed malesuada ipsum vel purus malesuada cursus.

Current Behavior

out1: Lorem ipsum dolor sit amet, consectetur adipiscing ... erat condimentum, lobortis libero eget, efficitur
out2: , nec tincidunt mi purus luctus turpis. Nulla non ... smod lectus. Praesent ut mi metus. Nulla facilisi.
out3: purus arcu quis erat. Nunc ligula sem, tincidunt i ... r. Sed malesuada ipsum vel purus malesuada cursus.

• Correct value is returned for out1
• out2 does not start right after where out1 ends
• same as out3

Reproduction Steps

Assume this file is in custom-bucket bucket : sample.txt

client = boto3.client("s3")
b = 'custom-bocket'
k = 'sample.txt'
body = client.get_object(Bucket=b, Key=k)["Body"]
print(next(body.iter_lines()))
print(next(body.iter_lines()))
print(next(body.iter_lines()))

observe output for each iter_lines() function

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.23.2

Environment details (OS name and version, etc.)

Ubuntu 20.04

[tim-finnigan]

Thanks @devfox-se for reaching out. Could you please provide your debug logs by adding boto3.set_stream_logger('') to your script?

[devfox-se]

Hi @tim-finnigan, thanks for your response!
Sure, here's the output

>>> import boto3
>>> client = boto3.client("s3")
>>> boto3.set_stream_logger('')
>>> b = 'custom-bucket'
>>> k = 'sample.txt'
>>> body = client.get_object(Bucket=b, Key=k)["Body"]
2022-05-19 21:25:30,997 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7fcd98ebb9d0>
2022-05-19 21:25:30,997 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7fcd98ebb940>
2022-05-19 21:25:30,997 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7fcd9868cb80>>
2022-05-19 21:25:30,997 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7fcd9868cc40>>
2022-05-19 21:25:30,997 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function generate_idempotent_uuid at 0x7fcd98ebb790>
2022-05-19 21:25:30,998 botocore.hooks [DEBUG] Event before-call.s3.GetObject: calling handler <function add_expect_header at 0x7fcd98ebbca0>
2022-05-19 21:25:30,998 botocore.hooks [DEBUG] Event before-call.s3.GetObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7fcd9868cb80>>
2022-05-19 21:25:30,998 botocore.hooks [DEBUG] Event before-call.s3.GetObject: calling handler <function add_recursion_detection_header at 0x7fcd98ebb430>
2022-05-19 21:25:30,998 botocore.hooks [DEBUG] Event before-call.s3.GetObject: calling handler <function inject_api_version_header_if_needed at 0x7fcd98ec0040>
2022-05-19 21:25:30,998 botocore.endpoint [DEBUG] Making request for OperationModel(name=GetObject) with params: {'url_path': '/custom-bucket/sample.txt', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.23.2 Python/3.8.10 Linux/5.13.0-41-generic Botocore/1.26.2'}, 'body': b'', 'url': 'https://s3.us-west-2.amazonaws.com/custom-bucket/sample.txt', 'context': {'client_region': 'us-west-2', 'client_config': <botocore.config.Config object at 0x7fcd987d1a60>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'custom-bucket'}}}
2022-05-19 21:25:30,999 botocore.hooks [DEBUG] Event request-created.s3.GetObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fcd987d1a90>>
2022-05-19 21:25:30,999 botocore.hooks [DEBUG] Event choose-signer.s3.GetObject: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x7fcd9868ccd0>>
2022-05-19 21:25:30,999 botocore.hooks [DEBUG] Event choose-signer.s3.GetObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7fcd98e014f0>>
2022-05-19 21:25:30,999 botocore.hooks [DEBUG] Event choose-signer.s3.GetObject: calling handler <function set_operation_specific_signer at 0x7fcd98ebb670>
2022-05-19 21:25:30,999 botocore.hooks [DEBUG] Event before-sign.s3.GetObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7fcd9868ccd0>>
2022-05-19 21:25:30,999 botocore.utils [DEBUG] Defaulting to S3 virtual host style addressing with path style addressing fallback.
2022-05-19 21:25:30,999 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.us-west-2.amazonaws.com/custom-bucket/sample.txt
2022-05-19 21:25:30,999 botocore.utils [DEBUG] URI updated to: https://custom-bucket.s3.us-west-2.amazonaws.com/sample.txt
2022-05-19 21:25:31,000 botocore.auth [DEBUG] Calculating signature using v4 auth.
2022-05-19 21:25:31,000 botocore.auth [DEBUG] CanonicalRequest:
GET
/sample.txt

host:custom-bucket.s3.us-west-2.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20220519T172530Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2022-05-19 21:25:31,000 botocore.auth [DEBUG] StringToSign:
AWS4-HMAC-SHA256
20220519T172530Z
20220519/us-west-2/s3/aws4_request
e2e89da2b752053a39250f26e80ab2bcd11ff64e79c1cec4feef314f153093d3
2022-05-19 21:25:31,000 botocore.auth [DEBUG] Signature:
98a76c3cfec0018dd16b1a0b8fadd60b3a7742a7048621244b64e9e629dbc8b0
2022-05-19 21:25:31,000 botocore.hooks [DEBUG] Event request-created.s3.GetObject: calling handler <function add_retry_headers at 0x7fcd98ec0700>
2022-05-19 21:25:31,000 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=True, method=GET, url=https://custom-bucket.s3.us-west-2.amazonaws.com/sample.txt, headers={'User-Agent': b'Boto3/1.23.2 Python/3.8.10 Linux/5.13.0-41-generic Botocore/1.26.2', 'X-Amz-Date': b'20220519T172530Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIA2HCABY7BMSBYMVKJ/20220519/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=98a76c3cfec0018dd16b1a0b8fadd60b3a7742a7048621244b64e9e629dbc8b0', 'amz-sdk-invocation-id': b'99aa3cfc-e378-49a7-baba-56838627ad1e', 'amz-sdk-request': b'attempt=1'}>
2022-05-19 21:25:31,001 botocore.httpsession [DEBUG] Certificate path: /home/devfox/.venvs/env/lib/python3.8/site-packages/certifi/cacert.pem
2022-05-19 21:25:31,001 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): custom-bucket.s3.us-west-2.amazonaws.com:443
2022-05-19 21:25:31,949 urllib3.connectionpool [DEBUG] https://custom-bucket.s3.us-west-2.amazonaws.com:443 "GET /sample.txt HTTP/1.1" 200 7517
2022-05-19 21:25:31,949 botocore.parsers [DEBUG] Response headers: {'x-amz-id-2': 'dZonyLSXH4deSllFFoqxdehJBxhsC4k0P30WsYSAKyN8KDVeMGt9VW4wCMIvE8p4d6PphgZV/Ww=', 'x-amz-request-id': 'G7DD6A5JJA7DFKPK', 'Date': 'Thu, 19 May 2022 17:25:32 GMT', 'Last-Modified': 'Thu, 19 May 2022 17:23:51 GMT', 'ETag': '"f07ece6c24708364e74db201d32bb14b"', 'Accept-Ranges': 'bytes', 'Content-Type': 'text/plain', 'Server': 'AmazonS3', 'Content-Length': '7517'}
2022-05-19 21:25:31,949 botocore.parsers [DEBUG] Response body:
<botocore.response.StreamingBody object at 0x7fcd9a48b6d0>
2022-05-19 21:25:31,950 botocore.hooks [DEBUG] Event needs-retry.s3.GetObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fcd9868cb20>
2022-05-19 21:25:31,950 botocore.retryhandler [DEBUG] No retry needed.
2022-05-19 21:25:31,950 botocore.hooks [DEBUG] Event needs-retry.s3.GetObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fcd9868cb80>>
>>> print(next(body.iter_lines()))
b'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur et vulputate enim. Fusce ultrices leo risus, vitae tempus lacus consequat id. Quisque convallis sem sit amet diam molestie, aliquam varius ipsum imperdiet. Nunc vitae consequat nunc. Vivamus tincidunt posuere ipsum vel egestas. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quisque nec ante vitae nisi mattis ultricies. Etiam rhoncus fermentum lacinia. Fusce consectetur condimentum tortor non sagittis. Etiam a massa nec leo accumsan dictum non quis massa. Aliquam interdum, nulla nec euismod vulputate, tellus massa porta lectus, sit amet commodo odio diam quis neque. Mauris aliquam eu nisi ac faucibus. Quisque efficitur vehicula tellus, congue fringilla ante venenatis et. Nam imperdiet dictum diam in varius. Ut tincidunt nisi vel tellus maximus, vitae porttitor massa fringilla. Etiam posuere aliquet tellus, non finibus magna varius accumsan. Quisque molestie faucibus congue. Donec elementum nulla vel accumsan sagittis. Morbi porta mauris nibh, et pretium nisi tincidunt ac. Donec tellus dolor, finibus id mi ac, volutpat dignissim velit. Sed id nulla condimentum, lobortis libero eget, efficitur justo. Sed quis sapien consequat, lobortis mauris ut, ultrices urna. Sed volutpat porta nibh ac viverra. Nullam nibh turpis, scelerisque eget magna id, suscipit molestie ex. Etiam suscipit nibh sit amet lacus pellentesque vehicula vel at lectus. In mattis porta sagittis. Aliquam id sapien vitae magna pulvinar consequat. Aliquam eu felis at sem vehicula fermentum ac id diam. Pellentesque tortor ligula, tempor eget ipsum nec, dictum dapibus justo. Vestibulum at rutrum nisi. Proin ac auctor massa. Aenean volutpat lectus a ligula hendrerit egestas. Aliquam ac justo purus. Duis ut enim dui. Nam sit amet felis posuere, vulputate ipsum non, hendrerit urna. In mattis quis felis sed posuere. Sed vestibulum justo non augue iaculis, consequat placerat nulla commodo. Integer malesuada vulputate tellus quis egestas. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Donec quis mauris fringilla, mollis enim vel, viverra velit. Cras non porttitor libero. Ut tellus erat, tempus in risus id, varius porttitor erat condimentum, lobortis libero eget, efficitur'
>>> print(next(body.iter_lines()))
b', nec tincidunt mi purus luctus turpis. Nulla non erat ut nunc vulputate egestas id vel nisi. Mauris vel aliquam leo. Suspendisse luctus pharetra nulla, nec aliquet turpis scelerisque ac. Etiam congue tempor aliquam. Suspendisse fermentum maximus metus, a blandit massa imperdiet non. Vivamus lobortis varius massa, at egestas eros finibus nec. Morbi odio lacus, vehicula faucibus sollicitudin id, posuere vel odio. Vivamus viverra malesuada mattis. Maecenas ornare ligula eu mauris blandit, et tempor eros fermentum. Sed ornare mattis nibh ac pellentesque.Nam bibendum ex a justo rhoncus vehicula non nec eros. Phasellus pulvinar mollis metus. Proin facilisis eros vitae massa posuere faucibus. Sed elit diam, mattis et tortor nec, condimentum interdum turpis. Pellentesque finibus nibh sit amet sapien viverra facilisis. Nulla eget nisi mi. Suspendisse in elementum sem.Curabitur tempor sollicitudin leo, ut scelerisque ipsum. Suspendisse eleifend sem urna, quis ullamcorper libero euismod vel. Fusce eget metus ultrices, sodales nibh posuere, congue neque. Praesent erat erat, rutrum non dui at, ornare condimentum libero. Etiam varius ante ex, at venenatis justo malesuada quis. Duis blandit molestie justo, vel tincidunt sapien condimentum eu. Proin congue lacus eu ante lacinia ultricies. Aenean ultricies tristique mi, vitae rutrum orci hendrerit non. Aliquam elementum tristique quam sed congue. In vestibulum sapien eu elit imperdiet semper. Integer egestas lacinia turpis. Nam nec euismod lectus. Praesent ut mi metus. Nulla facilisi.'
>>> 

[tim-finnigan]

Hi @devfox-se thanks for sharing the logs. It looks like what you want to do is just call iter_lines() once and then iterate on it. For example:

iterator = body.iter_lines()
print(next(iterator))
print(next(iterator))

Is that what you’re trying to do? Here is documentation for reference: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/response.html

[devfox-se]

Thanks @tim-finnigan it seems to work like this! I clearly misunderstood how to use it.

[tim-finnigan]

Glad it is working for you now, thanks for letting us know! I'm going to convert this to a Discussion per our guidance process and will mark this as answered.