ResourceWarning: unclosed file with request.POST['file'].file #3599

hyperknot · 2020-06-29T01:21:36Z

Describe the bug
When I POST to the following endpoint snippet, after a few tries I get a ResourceWarning: unclosed file.

@view_config(route_name='file_upload', renderer='json')
def map_upload(request):
    request.POST['file'].file

    with open('/some/path/on/local/disk/some.json') as infile:
        infile.read()
    
    return {}

Output with
PYTHONTRACEMALLOC=25

Output 1:

..../venv/lib/python3.7/site-packages/pyramid/urldispatch.py:211: ResourceWarning: unclosed file <_io.FileIO name=24 mode='rb+' closefd=True>
  for k, v in m.groupdict().items():
Object allocated at (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", lineno 890
    self._bootstrap_inner()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", lineno 926
    self.run()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", lineno 870
    self._target(*self._args, **self._kwargs)
  File "..../venv/lib/python3.7/site-packages/waitress/task.py", lineno 84
    task.service()
  File "..../venv/lib/python3.7/site-packages/waitress/channel.py", lineno 349
    task.service()
  File "..../venv/lib/python3.7/site-packages/waitress/task.py", lineno 169
    self.execute()
  File "..../venv/lib/python3.7/site-packages/waitress/task.py", lineno 439
    app_iter = self.channel.server.application(environ, start_response)
  File "..../venv/lib/python3.7/site-packages/paste/translogger.py", lineno 69
    return self.application(environ, replacement_start_response)
  File "..../venv/lib/python3.7/site-packages/pyramid/router.py", lineno 270
    response = self.execution_policy(environ, self)
  File "..../venv/lib/python3.7/site-packages/pyramid/router.py", lineno 277
    return router.invoke_request(request)
  File "..../venv/lib/python3.7/site-packages/pyramid/router.py", lineno 249
    response = handle_request(request)
  File "..../venv/lib/python3.7/site-packages/pyramid_tm/__init__.py", lineno 143
    response = handler(request)
  File "..../venv/lib/python3.7/site-packages/pyramid/tweens.py", lineno 41
    response = handler(request)
  File "..../venv/lib/python3.7/site-packages/pyramid/router.py", lineno 148
    registry, request, context, context_iface, view_name
  File "..../venv/lib/python3.7/site-packages/pyramid/view.py", lineno 667
    response = view_callable(context, request)
  File "..../venv/lib/python3.7/site-packages/pyramid/viewderivers.py", lineno 514
    return view(context, request)
  File "/Users/user/nonsync/dev/maphub/web/server/maphub_web/security.py", lineno 94
    response = view(context, request)
  File "..../venv/lib/python3.7/site-packages/pyramid/viewderivers.py", lineno 436
    result = view(context, request)
  File "..../venv/lib/python3.7/site-packages/pyramid/viewderivers.py", lineno 144
    response = view(request)
  File "/Users/user/nonsync/dev/maphub/web/server/maphub_web/views/ajax/upload_download.py", lineno 36
    request.POST['file'].file
  File "..../venv/lib/python3.7/site-packages/webob/request.py", lineno 776
    self.make_body_seekable()
  File "..../venv/lib/python3.7/site-packages/webob/request.py", lineno 929
    self.copy_body()
  File "..../venv/lib/python3.7/site-packages/webob/request.py", lineno 979
    fileobj = self.make_tempfile()
  File "..../venv/lib/python3.7/site-packages/webob/request.py", lineno 1019
    return tempfile.TemporaryFile()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tempfile.py", lineno 622
    newline=newline, encoding=encoding)

In a less minimal form (closer to real world logic), the output was:

/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/decoder.py:353: ResourceWarning: unclosed file <_io.BufferedRandom name=18>
  obj, end = self.scan_once(s, idx)
Object allocated at (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", lineno 890
    self._bootstrap_inner()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", lineno 926
    self.run()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", lineno 870
    self._target(*self._args, **self._kwargs)
  File "venv/lib/python3.7/site-packages/waitress/task.py", lineno 84
    task.service()
  File "venv/lib/python3.7/site-packages/waitress/channel.py", lineno 349
    task.service()
  File "venv/lib/python3.7/site-packages/waitress/task.py", lineno 169
    self.execute()
  File "venv/lib/python3.7/site-packages/waitress/task.py", lineno 439
    app_iter = self.channel.server.application(environ, start_response)
  File "venv/lib/python3.7/site-packages/paste/translogger.py", lineno 69
    return self.application(environ, replacement_start_response)
  File "venv/lib/python3.7/site-packages/pyramid/router.py", lineno 270
    response = self.execution_policy(environ, self)
  File "venv/lib/python3.7/site-packages/pyramid/router.py", lineno 277
    return router.invoke_request(request)
  File "venv/lib/python3.7/site-packages/pyramid/router.py", lineno 249
    response = handle_request(request)
  File "venv/lib/python3.7/site-packages/pyramid_tm/__init__.py", lineno 143
    response = handler(request)
  File "venv/lib/python3.7/site-packages/pyramid/tweens.py", lineno 41
    response = handler(request)
  File "venv/lib/python3.7/site-packages/pyramid/router.py", lineno 148
    registry, request, context, context_iface, view_name
  File "venv/lib/python3.7/site-packages/pyramid/view.py", lineno 667
    response = view_callable(context, request)
  File "venv/lib/python3.7/site-packages/pyramid/viewderivers.py", lineno 514
    return view(context, request)
  File "/Users/user/nonsync/dev/maphub/web/server/maphub_web/security.py", lineno 94
    response = view(context, request)
  File "venv/lib/python3.7/site-packages/pyramid/viewderivers.py", lineno 436
    result = view(context, request)
  File "venv/lib/python3.7/site-packages/pyramid/viewderivers.py", lineno 144
    response = view(request)
  File "/Users/user/nonsync/dev/maphub/web/server/maphub_web/views/ajax/upload_download.py", lineno 37
    file_obj = request.POST['file'].file
  File "venv/lib/python3.7/site-packages/webob/request.py", lineno 776
    self.make_body_seekable()
  File "venv/lib/python3.7/site-packages/webob/request.py", lineno 929
    self.copy_body()
  File "venv/lib/python3.7/site-packages/webob/request.py", lineno 979
    fileobj = self.make_tempfile()
  File "venv/lib/python3.7/site-packages/webob/request.py", lineno 1019
    return tempfile.TemporaryFile()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tempfile.py", lineno 622
    newline=newline, encoding=encoding)

To Reproduce
Steps to reproduce the behavior:

Try to POST the above snippet endpoint multiple times.
The first upload is always good. The error happens on the 2nd .. 5th upload usually.

Expected behavior
Nothing like this should happen.

Additional context
This is really strange and it took me a while to reproduce this minimal case, it honestly makes no sense to me. But it's 100% reproducible, so I'm sharing it here as it is. The file I'm uploading is 2.8 MB in size.

Python 3.7 / macOS
pyramid==1.10.4
WebOb==1.8.6
waitress==1.4.4

The text was updated successfully, but these errors were encountered:

mmerickel · 2020-06-29T15:55:15Z

Opening a file without closing it... yep that'll trigger.

@view_config(route_name='file_upload', renderer='json')
def map_upload(request):
   with request.POST['file'].file as fp:
      pass

   with open('/some/path/on/local/disk/some.json') as infile:
        infile.read()
        return {}

mmerickel · 2020-06-29T16:15:03Z

I opened this issue on webob which could help in the future but the behavior you're seeing is expected right now (unfortunately).

Pylons/webob#415

hyperknot · 2020-06-29T17:16:13Z

Sorry that return was obviously a mistake in my pasted snippet. The problem is the same when return is outside the with block.

luhn · 2020-06-29T17:19:18Z

The issue issue isn't the return, it's the request.POST['file'].file. That opens up a file handler, but it's never closed, hence the bug.

@mmerickel sample puts that in a with statement, which will close the file handler properly.

hyperknot · 2020-06-29T19:28:05Z

I tried it with @mmerickel's with statement, it didn't help either. It took a lot of tries, like 6-8 but it triggered the warning:

    with request.POST['file'].file as fp:
        pass

    with open(
        'path.json',
        'rb',
    ) as infile:
        infile.read()

    return {}

.../venv/lib/python3.7/site-packages/redis/_compat.py:72: ResourceWarning: unclosed file <_io.FileIO name=32 mode='rb+' closefd=True>
  return sock.recv(*args, **kwargs)
Object allocated at (most recent call last):
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", lineno 890
    self._bootstrap_inner()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", lineno 926
    self.run()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", lineno 870
    self._target(*self._args, **self._kwargs)
  File ".../venv/lib/python3.7/site-packages/waitress/task.py", lineno 84
    task.service()
  File ".../venv/lib/python3.7/site-packages/waitress/channel.py", lineno 349
    task.service()
  File ".../venv/lib/python3.7/site-packages/waitress/task.py", lineno 169
    self.execute()
  File ".../venv/lib/python3.7/site-packages/waitress/task.py", lineno 439
    app_iter = self.channel.server.application(environ, start_response)
  File ".../venv/lib/python3.7/site-packages/paste/translogger.py", lineno 69
    return self.application(environ, replacement_start_response)
  File ".../venv/lib/python3.7/site-packages/pyramid/router.py", lineno 270
    response = self.execution_policy(environ, self)
  File ".../venv/lib/python3.7/site-packages/pyramid/router.py", lineno 277
    return router.invoke_request(request)
  File ".../venv/lib/python3.7/site-packages/pyramid/router.py", lineno 249
    response = handle_request(request)
  File ".../venv/lib/python3.7/site-packages/pyramid_tm/__init__.py", lineno 143
    response = handler(request)
  File ".../venv/lib/python3.7/site-packages/pyramid/tweens.py", lineno 41
    response = handler(request)
  File ".../venv/lib/python3.7/site-packages/pyramid/router.py", lineno 148
    registry, request, context, context_iface, view_name
  File ".../venv/lib/python3.7/site-packages/pyramid/view.py", lineno 667
    response = view_callable(context, request)
  File ".../venv/lib/python3.7/site-packages/pyramid/viewderivers.py", lineno 514
    return view(context, request)
  File "maphub_web/security.py", lineno 94
    response = view(context, request)
  File ".../venv/lib/python3.7/site-packages/pyramid/viewderivers.py", lineno 436
    result = view(context, request)
  File ".../venv/lib/python3.7/site-packages/pyramid/viewderivers.py", lineno 144
    response = view(request)
  File "maphub_web/views/ajax/upload_download.py", lineno 36
    with request.POST['file'].file as fp:
  File ".../venv/lib/python3.7/site-packages/webob/request.py", lineno 776
    self.make_body_seekable()
  File ".../venv/lib/python3.7/site-packages/webob/request.py", lineno 929
    self.copy_body()
  File ".../venv/lib/python3.7/site-packages/webob/request.py", lineno 979
    fileobj = self.make_tempfile()
  File ".../venv/lib/python3.7/site-packages/webob/request.py", lineno 1019
    return tempfile.TemporaryFile()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/tempfile.py", lineno 622
    newline=newline, encoding=encoding)
200 POST /ajax/map_upload

mmerickel · 2020-06-29T19:44:02Z

Sorry the context manager is actually on the cgi.FieldStorage object which is file-like and supports reading/writing. This may work better:

@view_config(route_name='file_upload', renderer='json')
def map_upload(request):
   with request.POST['file'] as field:
      data = field.value

   with open('/some/path/on/local/disk/some.json') as infile:
        infile.read()
   return {}

If you want to consume the file directly without loading it into a full memory buffer then you can read field.file directly as well. The important thing is to make sure you close the field when you're done with it via field.file.close() which the context manager will do. From there, no one should try to access the raw file again in that request.

hyperknot · 2020-06-29T20:02:29Z

The following still doesn't work (actually this was what I tried as a very first step):

    fp = request.POST['file'].file

    with open('disk.file', 'wb') as dst_file:
        shutil.copyfileobj(fp, dst_file)

    fp.close()

    with open(some_path, 'rb') as infile:
        infile.read()

    return {}

mmerickel · 2020-06-29T20:29:52Z

Apologies I’ll have to try to find a repro to offer more help. Closing the field file is usually enough.

archoversight · 2020-06-30T19:49:26Z

Closing the file returned from the FieldStorage is good enough. It's a red herring, looking at the traceback the temporary file that is the culprit is the one created by WebOb when it makes sure that the body of the request is seekable (so that it may be rewound to the beginning of the stream).

It looks like your WSGI server does not already give Pyramid a wsgi.input that is seekable, so WebOb does it for you, but it doesn't clean up after itself.

Edit: Actually WebOb doesn't check if the provided wsgi.input is seekable or not, only trusting itself, so always copying to make seekable.

mmerickel · 2020-06-30T21:09:24Z

Ah I missed that it was due to the tempfile created from make_body_seekable which is a separate issue. Unfortunately there's not a good way to clean that up automatically at this time.

hyperknot · 2020-06-30T22:08:40Z

It looks like your WSGI server does not already give Pyramid a wsgi.input that is seekable, so WebOb does it for you, but it doesn't clean up after itself.

This is in local development setup with Waitress.

digitalresistor · 2020-07-06T00:52:39Z

@hyperknot and waitress does, but WebOb doesn't adequately check for it. See the edit above by @archoversight.

mmerickel · 2020-07-07T14:52:02Z

Moving this issue to Pylons/webob#416.

mmerickel mentioned this issue Jul 7, 2020

detect seekable wsgi.input and use it instead of making another copy Pylons/webob#416

Open

mmerickel closed this as completed Jul 7, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ResourceWarning: unclosed file with request.POST['file'].file #3599

ResourceWarning: unclosed file with request.POST['file'].file #3599

hyperknot commented Jun 29, 2020 •

edited

Loading

mmerickel commented Jun 29, 2020

mmerickel commented Jun 29, 2020

hyperknot commented Jun 29, 2020 •

edited

Loading

luhn commented Jun 29, 2020

hyperknot commented Jun 29, 2020 •

edited

Loading

mmerickel commented Jun 29, 2020 •

edited

Loading

hyperknot commented Jun 29, 2020 •

edited

Loading

mmerickel commented Jun 29, 2020

archoversight commented Jun 30, 2020 •

edited

Loading

mmerickel commented Jun 30, 2020

hyperknot commented Jun 30, 2020

digitalresistor commented Jul 6, 2020

mmerickel commented Jul 7, 2020

ResourceWarning: unclosed file with request.POST['file'].file #3599

ResourceWarning: unclosed file with request.POST['file'].file #3599

Comments

hyperknot commented Jun 29, 2020 • edited Loading

mmerickel commented Jun 29, 2020

mmerickel commented Jun 29, 2020

hyperknot commented Jun 29, 2020 • edited Loading

luhn commented Jun 29, 2020

hyperknot commented Jun 29, 2020 • edited Loading

mmerickel commented Jun 29, 2020 • edited Loading

hyperknot commented Jun 29, 2020 • edited Loading

mmerickel commented Jun 29, 2020

archoversight commented Jun 30, 2020 • edited Loading

mmerickel commented Jun 30, 2020

hyperknot commented Jun 30, 2020

digitalresistor commented Jul 6, 2020

mmerickel commented Jul 7, 2020

hyperknot commented Jun 29, 2020 •

edited

Loading

hyperknot commented Jun 29, 2020 •

edited

Loading

hyperknot commented Jun 29, 2020 •

edited

Loading

mmerickel commented Jun 29, 2020 •

edited

Loading

hyperknot commented Jun 29, 2020 •

edited

Loading

archoversight commented Jun 30, 2020 •

edited

Loading