Issue: Installation Failure of Netmiko on Python 3.11.4 Embedded Version

[glego]

Overview

I encountered an error while attempting to install netmiko using pip on the embedded Python version python-3.11.4-embed-amd64.

Key Notes

• The issue seems to be with the future package, as indicated by the error traceback.
• The problem doesn't seem to originate from pip, but from a subprocess during the package's metadata generation.

Reproduction Steps

The script named python-embed-future-issue.ps1 performs the following:

1. Downloads the Python 3.11.4 embedded binaries.
2. Install pip.
3. Modify the python311._pth to ensure the site-packages directory is considered.
4. Initiate the installation of netmiko using the command python.exe -m pip install netmiko.

Script content

# Define Variables
$pythonUrl = "https://www.python.org/ftp/python/3.11.4/python-3.11.4-embed-amd64.zip"
$pythonEmbedZip = "${PWD}\python-3.11.4-embed-amd64.zip"
$pythonEmbedDir = "${PWD}\python-3.11.4-embed-amd64"
$pythonExec = "${pythonEmbedDir}\python.exe"
$pythonPthFilePath = "${pythonEmbedDir}\python311._pth"

$getPipUrl = "https://bootstrap.pypa.io/get-pip.py"
$getPipFilePath = "${pythonEmbedDir}\get-pip.py"

# Cleanup existing files and paths
Remove-Item -Path $pythonEmbedZip -Force -ErrorAction "SilentlyContinue"
Remove-Item -Path $pythonEmbedDir -Force -Recurse -ErrorAction "SilentlyContinue"

# Download Embedded Python Version
(New-Object Net.WebClient).DownloadFile($pythonUrl, $pythonEmbedZip)
Expand-Archive $pythonEmbedZip 

# Download Pip Installer Script
(New-Object Net.WebClient).DownloadFile($getPipUrl, $getPipFilePath)

# Modify .pth file to include site-packages
$pthContent = Get-Content -Path $pythonPthFilePath
$newPthContent = $pthContent -replace "#import site", "import site"
Set-Content -Path $pythonPthFilePath -Value $newPthContent -Force

# Initiate Pip Installation
& $pythonExec "${getPipFilePath}"

# Begin netmiko Installation
& $pythonExec -m pip install netmiko

Output from the script execution:

Collecting pip
  Obtaining dependency information for pip from https://files.pythonhosted.org/packages/50/c2/e06851e8cc28dcad7c155f4753da8833ac06a5c704c109313b8d5a62968a/pip-23.2.1-py3-none-any.whl.metadata    
  Using cached pip-23.2.1-py3-none-any.whl.metadata (4.2 kB)
Collecting setuptools
  Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/c7/42/be1c7bbdd83e1bfb160c94b9cafd8e25efc7400346cf7ccdbdb452c467fa/setuptools-68.0.0-py3-none-any.whl.metadata
  Using cached setuptools-68.0.0-py3-none-any.whl.metadata (6.4 kB)
Collecting wheel
  Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/28/f5/6955d7b3a5d71ce6bac104f9cf98c1b0513ad656cdaca8ea7d579196f771/wheel-0.41.1-py3-none-any.whl.metadata
  Using cached wheel-0.41.1-py3-none-any.whl.metadata (2.2 kB)
Using cached pip-23.2.1-py3-none-any.whl (2.1 MB)
Using cached setuptools-68.0.0-py3-none-any.whl (804 kB)
Using cached wheel-0.41.1-py3-none-any.whl (64 kB)
Installing collected packages: wheel, setuptools, pip
  WARNING: The script wheel.exe is installed in 'C:\Users\GlennGoffin\Git\utils\python3-embed-win-x64\temp\python-3.11.4-embed-amd64\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts pip.exe, pip3.11.exe and pip3.exe are installed in 'C:\Users\GlennGoffin\Git\utils\python3-embed-win-x64\temp\python-3.11.4-embed-amd64\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-23.2.1 setuptools-68.0.0 wheel-0.41.1
Collecting netmiko
  Using cached netmiko-4.2.0-py3-none-any.whl (213 kB)
Collecting ntc-templates>=2.0.0 (from netmiko)
  Obtaining dependency information for ntc-templates>=2.0.0 from https://files.pythonhosted.org/packages/8a/64/b5d15f16ea6e5152fe1b56115b02efb87326db5b006bb907290fcdbf019b/ntc_templates-3.5.0-py3-none-any.whl.metadata
  Using cached ntc_templates-3.5.0-py3-none-any.whl.metadata (3.9 kB)
Collecting paramiko>=2.9.5 (from netmiko)
  Obtaining dependency information for paramiko>=2.9.5 from https://files.pythonhosted.org/packages/bb/8f/3cef65d3fe76e59f320405027d594a0332e44852fef722f0ee4e81e2e7e3/paramiko-3.3.1-py3-none-any.whl.metadata
  Using cached paramiko-3.3.1-py3-none-any.whl.metadata (4.4 kB)
Collecting pyserial>=3.3 (from netmiko)
  Using cached pyserial-3.5-py2.py3-none-any.whl (90 kB)
Collecting pyyaml>=5.3 (from netmiko)
  Obtaining dependency information for pyyaml>=5.3 from https://files.pythonhosted.org/packages/b3/34/65bb4b2d7908044963ebf614fe0fdb080773fc7030d7e39c8d3eddcd4257/PyYAML-6.0.1-cp311-cp311-win_amd64.whl.metadata
  Using cached PyYAML-6.0.1-cp311-cp311-win_amd64.whl.metadata (2.1 kB)
Collecting scp>=0.13.6 (from netmiko)
  Using cached scp-0.14.5-py2.py3-none-any.whl (8.7 kB)
Collecting textfsm>=1.1.3 (from netmiko)
  Using cached textfsm-1.1.3-py2.py3-none-any.whl (44 kB)
Collecting bcrypt>=3.2 (from paramiko>=2.9.5->netmiko)
  Using cached bcrypt-4.0.1-cp36-abi3-win_amd64.whl (152 kB)
Collecting cryptography>=3.3 (from paramiko>=2.9.5->netmiko)
  Obtaining dependency information for cryptography>=3.3 from https://files.pythonhosted.org/packages/30/56/5f4eee57ccd577c261b516bfcbe17492838e2bc4e2e92ea93bbb57666fbd/cryptography-41.0.3-cp37-abi3-win_amd64.whl.metadata
  Using cached cryptography-41.0.3-cp37-abi3-win_amd64.whl.metadata (5.3 kB)
Collecting pynacl>=1.5 (from paramiko>=2.9.5->netmiko)
  Using cached PyNaCl-1.5.0-cp36-abi3-win_amd64.whl (212 kB)
Collecting six (from textfsm>=1.1.3->netmiko)
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting future (from textfsm>=1.1.3->netmiko)
  Using cached future-0.18.3.tar.gz (840 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\GlennGoffin\AppData\Local\Temp\pip-install-copost0e\future_8c061b43807c4df1aed534bfb1fad12e\setup.py", line 86, in <module>
          import src.future
      ModuleNotFoundError: No module named 'src'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Attempts to fix

Update PIP and Setup Tools

python -m ensurepip --upgrade
python -m pip install --upgrade pip
python -m pip install --upgrade setuptools

References

• https://kb.paessler.com/en/topic/91446-error-installing-python-module-future
• Issue: Installation Failure of Netmiko on Python 3.11.4 Embedded Version PythonCharmers/python-future#623

[ktbyers]

PY3.11.0 worked fine for me on a fedora based system

Running setup.py install for future ... done
Successfully installed bcrypt-4.0.1 cffi-1.15.1 cryptography-41.0.3 future-0.18.3 netmiko-4.2.0 ntc-templates-3.5.0 paramiko-3.3.1 pycparser-2.21 pynacl-1.5.0 pyserial-3.5 pyyaml-6.0.1 scp-0.14.5 six-1.16.0 textfsm-1.1.3

The src.future referenced above exists--so it looks like an environment issue on your side.

https://github.com/PythonCharmers/python-future/tree/v0.18.3/src

What version of setuptools are you using?

I tested with both the setuptools versions below:

65.5.0
68.1.2

[glego]

Hi @ktbyers

I've tested it on a windows 11 OS which a fresh (embedded) python evironment 3.11.4. The versions can be retrieved from the log output above. When using the Python installer, i also dont have any issues...

Using cached pip-23.2.1-py3-none-any.whl (2.1 MB)
Using cached setuptools-68.0.0-py3-none-any.whl (804 kB)
Using cached wheel-0.41.1-py3-none-any.whl (64 kB)

[ktbyers]

My guess is that it is related to this step:

Modify the python311._pth to ensure the site-packages directory is considered.

There is a good chance that future is the only library that doesn't have a wheel file and for some reason that is tripping you up. Note, future does this:

https://github.com/PythonCharmers/python-future/blob/master/setup.py#L177

I wonder if that is causing a conflict with what you are trying to do.

I wonder if can build a wheel for future and just download/install that?

[glego]

Thanks for the direction, i'll have a look and report back