Missing IO_REPARSE_TAG_MOUNT_POINT in stat module in 3.9?

[Erotemic]

I'm getting an error in ubelt when testing against pypy on windows: https://github.com/Erotemic/ubelt/actions/runs/9105340373/job/25030757962

The error is that:

E                   AttributeError: module 'stat' has no attribute 'IO_REPARSE_TAG_MOUNT_POINT'

This is coming from shutil, the traceback is:

..\ubelt\util_path.py:1518: in copy
    dst = copytree(
C:\hostedtoolcache\windows\PyPy\3.9.19\x86\Lib\shutil.py:568: in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
C:\hostedtoolcache\windows\PyPy\3.9.19\x86\Lib\shutil.py:504: in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
C:\hostedtoolcache\windows\PyPy\3.9.19\x86\Lib\shutil.py:568: in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
C:\hostedtoolcache\windows\PyPy\3.9.19\x86\Lib\shutil.py:504: in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
C:\hostedtoolcache\windows\PyPy\3.9.19\x86\Lib\shutil.py:568: in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
C:\hostedtoolcache\windows\PyPy\3.9.19\x86\Lib\shutil.py:504: in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
C:\hostedtoolcache\windows\PyPy\3.9.19\x86\Lib\shutil.py:568: in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
C:\hostedtoolcache\windows\PyPy\3.9.19\x86\Lib\shutil.py:504: in _copytree
    copytree(srcobj, dstname, symlinks, ignore, copy_function,
C:\hostedtoolcache\windows\PyPy\3.9.19\x86\Lib\shutil.py:568: in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,

In writing this up, I'm getting confused, because the stat module in CPython 3.9 and 3.11 don't seem to have this attribute either, but in all cases, including PyPy 3.9, the shutil module references this attribute, but the stat module doesn't define it. I spun up a docker image for pypy:3.9, and python:3.9, as well as having a venv with CPython 3.11 on my local machine, and in each one I ran:

cat $(python -c "import stat; print(stat.__file__)")  | grep IO_RE
cat $(python -c "import shutil; print(shutil.__file__)")  | grep IO_RE

Where the first command prints nothing, and the second command does print something.

I opened up the stats.py module and see there is a from _stat import *, so perhaps that is where this attribute comes from? I don't have a windows machine to test on aside from the github actions CI, which is difficult to work with. Not sure if this is a pypy issue where it isn't exposing this attribute on windows or not.

Looking at the CPython source code, it looks like it does define IO_REPARSE_TAG_MOUNT_POINT: https://github.com/python/cpython/blob/3.9/Modules/_stat.c

There is some code in PyPy that references the attribute: https://github.com/pypy/pypy/blob/release-pypy3.9-v7.3.16/rpython/rlib/src/winreparse.c but it looks like it never adds it as an attribute to the "stats" module? Based on this, it seems like this might be a PyPy bug that should be patched. Thoughts?

[mattip]

Indeed that should be exported from _stat. I compared pypy to cpython, and these are missing {'IO_REPARSE_TAG_APPEXECLINK', 'IO_REPARSE_TAG_MOUNT_POINT', 'IO_REPARSE_TAG_SYMLINK'}.

[Erotemic]

Thanks for the confirmation. I've opened an issue now that I'm fairly sure this is a bug: #4953

[Erotemic]

@mattip

I've hacked in a monkey patch for pypy:

def _patch_win32_stats_on_pypy():
    """
    Handle [PyPyIssue4953]_.

    References:
        [PyPyIssue4953] https://github.com/pypy/pypy/issues/4953#event-12838738353
    """
    if not hasattr(stat, 'IO_REPARSE_TAG_MOUNT_POINT'):
        stat.IO_REPARSE_TAG_APPEXECLINK = 0x8000001b  # windows
        stat.IO_REPARSE_TAG_MOUNT_POINT = 0xa0000003  # windows
        stat.IO_REPARSE_TAG_SYMLINK = 0xa000000c      # windows

And then calling it like:

            if WIN32 and platform.python_implementation() == 'PyPy':
                _patch_win32_stats_on_pypy()

But I'm getting other errors that might be similar oversights? I'm not sure:

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

src = 'C:\\Users\\runneradmin\\AppData\\Local\\ubelt\\tests\\test_path\\64fb927b\\root\\links\\cyclic\\n1\\n2\\loop'
dst = 'C:\\Users\\runneradmin\\AppData\\Local\\ubelt\\tests\\test_path\\64fb927b\\our_copy\\links\\cyclic\\n1\\n2\\loop'

    def copystat(src, dst, *, follow_symlinks=True):
        """Copy file metadata
    
        Copy the permission bits, last access time, last modification time, and
        flags from `src` to `dst`. On Linux, copystat() also copies the "extended
        attributes" where possible. The file contents, owner, and group are
        unaffected. `src` and `dst` are path-like objects or path names given as
        strings.
    
        If the optional flag `follow_symlinks` is not set, symlinks aren't
        followed if and only if both `src` and `dst` are symlinks.
        """
        sys.audit("shutil.copystat", src, dst)
    
        def _nop(*args, ns=None, follow_symlinks=None):
            pass
    
        # follow symlinks (aka don't not follow symlinks)
        follow = follow_symlinks or not (_islink(src) and os.path.islink(dst))
        if follow:
            # use the real function if it exists
            def lookup(name):
                return getattr(os, name, _nop)
        else:
            # use the real function only if it exists
            # *and* it supports follow_symlinks
            def lookup(name):
                fn = getattr(os, name, _nop)
                if fn in os.supports_follow_symlinks:
                    return fn
                return _nop
    
        if isinstance(src, os.DirEntry):
            st = src.stat(follow_symlinks=follow)
        else:
            st = lookup("stat")(src, follow_symlinks=follow)
>       mode = stat.S_IMODE(st.st_mode)
E       AttributeError: 'NoneType' object has no attribute 'st_mode'

This is wrt to shutil.copystat. The run that I saw this in is: https://github.com/Erotemic/ubelt/actions/runs/9131194891/job/25109874480

I can work to find a MWE, but maybe the issue is apparent to you?

[mattip]

Thanks for following up. I fixed the root cause in b132295 which will in the end add "stats" to os.supports_follow_symlinks. You can patch this without the rpython fix in that changeset with

os.supports_follow_symlinks.add("stat")

[Erotemic]

Should that line be:

os.supports_follow_symlinks.add(os.stat)

?
as noted in: #4977

[mattip]

I don't think so We want to get to this result: On CPython

>>> os.supports_follow_symlinks
{<built-in function stat>}

and on PyPy 3.10 from June 8

>>>> os.supports_follow_symlinks
{<built-in function stat>}

Edit: Ahh, the problem is in the reproducer in #4977 and in my suggested fix in the comment from May 8 above, not in latest PyPY3.10.