Add CFFI dependency to avoid oscrypto failures

[PoesRaven]

Modified to include CCFI during installation. dxlclient errors out without CCFI.

[jbarlow-mcafee]

If this failure is occurring with oscrypto, it seems like it would be better to try to address the issue with the oscrypto project rather than by including a new dependency in the OpenDXL Python client.

I have never seen this issue, so I wonder if it only occurs for specific environments / OS-distributions. This oscrypto issue sounds somewhat similar - wbond/oscrypto#18. As mentioned in that issue, oscrypto was intended to be designed to use cffi if available but fall back to ctypes when it is missing. In that issue, the problem was appearing with OpenBSD in particular and was resolved in a later commit, with some changes to support LibreSSL compatibility. It doesn't appear, however, that that change has gone into a new release of oscrypto to PyPI yet though.

Have you ensured that you have the latest version of oscrypto, 0.19.1, installed during your testing?

What OS distribution and Python version are you able to reproduce this problem with?

If this isn't the same issue as the OpenBSD one above, we might try raising a new issue with the oscrypto project with a simpler use case which reproduces the problem.

[PoesRaven]

Hi Jeremy.

This makes complete sense. I'm running on Mac High Sierra which has an update from Python. This is a common issue from the Mac (Darwin) a subset of the BSD family... so yeah. CFFI worked well for me probably because there is an issue in the way cTypes are handled in new releases. My concern about not putting this here is that we are allowing a project which is not managed by us to dictate dxlclient usability in Mac.

[jbarlow-mcafee]

Thanks, @PoesRaven. It sounds like from this related issue filed to oscrypto for High Sierra in particular, wbond/oscrypto#22, that you may be seeing the same problem.

Could you try running the OpenDXL provisioning operation on High Sierra with the latest sources on the master branch of oscrypto to confirm if the oscrypto fix does address the problem? Unfortunately, it sounds like it may be a while until a new oscrypto release is dropped with this fix.

If installing the cffi package or doing a Homebrew install of OpenSSL — as mentioned in this comment — avoid the issue, we might recommend these as workarounds until a new oscrypto release can be released. I'm still a little leery of including the cffi dependency into the OpenDXL Python client directly given that the oscrypto project intends to provide a broad abstraction layer for crypto operations across OS environments and it seems like the OpenDXL Python client shouldn't have to compensate for that.

[PoesRaven]

Gotcha! I completely understand.

I did upgrade the home-brew openssl which is no longer recommended by python as python now comes with its own up-to-date openssl library (within the last 60 days this is has been updated in 3.6). Still, no luck. I'll try to update the oscrypto library as time permits and report back to the git repo.

Thanks, @jbarlow-mcafee .

-J

[jbarlow-mcafee]

Thanks, @PoesRaven.

I'm less optimistic about whether the oscrypto fix will address the problem if you can still reproduce it when OpenSSL is being used. It seemed like the fix was LibreSSL-specific. I'll be curious to hear what you find after testing with the latest unreleased oscrypto changes.

I'll also try to get a Mac updated to High Sierra here and see if I can reproduce the issue.

[jbarlow-mcafee]

I was able to reproduce the issue on High Sierra. Basically, any time I tried to run any Python client CLI commands - python -m dxlclient provisionconfig ... - I would get an error like the following:

› python -m dxlclient provisionconfig sample myeposerver client
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/dxlclient/__main__.py", line 9, in <module>
    from dxlclient._cli import cli_run
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/dxlclient/_cli/__init__.py", line 13, in <module>
    from dxlclient._cli._cli_subcommands import \
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/dxlclient/_cli/_cli_subcommands.py", line 18, in <module>
    from dxlclient._cli._crypto import X509Name, validate_cert_pem, \
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/dxlclient/_cli/_crypto.py", line 15, in <module>
    from oscrypto import asymmetric
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/oscrypto/asymmetric.py", line 15, in <module>
    from .kdf import pbkdf2, pbkdf2_iteration_calculator
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/oscrypto/kdf.py", line 9, in <module>
    from .util import rand_bytes
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/oscrypto/util.py", line 10, in <module>
    from ._osx.util import rand_bytes
  File "/.virtualenv/py-2.7.10/lib/python2.7/site-packages/oscrypto/_osx/util.py", line 208, in <module>
    from .._openssl._libcrypto import libcrypto
  File "/virtualenv/py-2.7.10/lib/python2.7/site-packages/oscrypto/_openssl/_libcrypto.py", line 14, in <module>
    from ._libcrypto_ctypes import (
  File "/virtualenv/py-2.7.10/lib/python2.7/site-packages/oscrypto/_openssl/_libcrypto_ctypes.py", line 668, in <module>
    raise FFIEngineError('Error initializing ctypes')
oscrypto._ffi.FFIEngineError: Error initializing ctypes

I saw the error both with Python 2.7.10 and 3.7.0. I was also able to reproduce the issue with two different SSL versions - LibreSSL 2.2.7 (default SSL installed with High Sierra) and a Homebrew-installed OpenSSL 1.0.2p (14 Aug 2018). In each case, I had the latest oscrypto release, 0.19.1, installed.

In each case, I was able to perform the provisionconfig command successfully by performing either (or both) of the following:

• Run pip install cffi.
• Run pip uninstall oscrypto, followed by pip install git+https://github.com/wbond/oscrypto@af778bf.

oscrypto version af778bf was the head of the master branch at the time I ran the test.

This seems to confirm that it may not be possible to suggest that a non-LibreSSL install would be enough to work around the problem but does seem to show that when the new oscrypto release is delivered that it should fix the problem.

For a little better visibility to users who may encounter this problem, I'll create a separate issue in the GitHub repo that I'll point back to this PR.