tiiuae · henrirosten · Dec 5, 2023 · Dec 4, 2023 · Dec 4, 2023 · Dec 4, 2023
diff --git a/README.md b/README.md
@@ -9,7 +9,7 @@ SPDX-License-Identifier: CC-BY-SA-4.0
 This repository is home to various command line tools and Python libraries that aim to help with software supply chain challenges:
 - [`sbomnix`](#generate-sbom-based-on-derivation-file-or-out-path) is a utility that generates SBOMs given [Nix](https://nixos.org/) derivation or out path.
 - [`nixgraph`](./doc/nixgraph.md) helps query and visualize dependency graphs for [Nix](https://nixos.org/) derivation or out path.
-- [`vulnxscan`](./doc/vulnxscan/vulnxscan.md) is a vulnerability scanner demonstrating the usage of SBOMs in running vulnerability scans.
+- [`vulnxscan`](./doc/vulnxscan.md) is a vulnerability scanner demonstrating the usage of SBOMs in running vulnerability scans.
 - [`repology_cli`](./doc/replogoy_cli.md) and [`repology_cve`](./doc/replogoy_cli.md#repology-cve-search) are command line clients to [repology.org](https://repology.org/).
 - [`nix_outdated`](./doc/nix_outdated.md) is a utility that finds outdated nix dependencies for given out path, listing the outdated packages in priority order based on how many other packages depend on the given outdated package.
 
@@ -194,4 +194,4 @@ This project is licensed under the Apache-2.0 license - see the [Apache-2.0.txt]
 
 
 ## Acknowledgements
-`sbomnix` uses Nix store derivation scanner ([nix.py](sbomnix/nix.py) and [derivation.py](sbomnix/derivation.py)) originally from [vulnix](https://github.com/flyingcircusio/vulnix).
+`sbomnix` uses Nix store derivation scanner ([nix.py](src/sbomnix/nix.py) and [derivation.py](src/sbomnix/derivation.py)) originally from [vulnix](https://github.com/nix-community/vulnix).
diff --git a/doc/replogoy_cli.md b/doc/replogoy_cli.md
@@ -98,7 +98,7 @@ INFO     Repology package info, packages:5
 | debian_12 | perl:firefox-marionette     | 1.35    | newest   |           0            | 1.35                    |                       |
 ```
 
-Notice: using short search strings with `--pkg_search` might result a large number of matches and, thus, potentially a large number of queries to repology.org. To avoid spamming repology.org with such queries, `repology_cli` limits the number of requests sent to repology.org to at most one request per second. In addition, it caches all responses locally for 3600 seconds.
+Notice: using short search strings with `--pkg_search` might result a large number of matches and, thus, potentially a large number of queries to repology.org. To avoid spamming repology.org with such queries, `repology_cli` limits the number of requests sent to repology.org to at most one request per second. In addition, it caches all responses locally for two hours.
 
 ### Search by Package Names in SBOM
 Following query finds 'nix_unstable' packages that match the packages in the CycloneDX sbom 'wget.runtime.sbom.cdx.json':

diff --git a/doc/vulnxscan.md b/doc/vulnxscan.md
@@ -47,7 +47,7 @@ $ nix eval -f '<nixpkgs>' 'git.drvPath'
 ### Nix and OSV Vulnerability Database
 [OSV](https://osv.dev/) is a vulnerability database for open-source projects [initiated by Google](https://security.googleblog.com/2021/02/launching-osv-better-vulnerability.html). 
 
-[OSV database](https://osv.dev/list?ecosystem=) currently [does not support Nix ecosystem](https://ossf.github.io/osv-schema/#affectedpackage-field), so queries that specify Nix as ecosystem would not return any matches. For this reason `vulnxscan` currently does not use Google's official [OSV-Scanner](https://security.googleblog.com/2022/12/announcing-osv-scanner-vulnerability.html) front-end, but implements it's own OSV client demo in [osv.py](./osv.py).
+[OSV database](https://osv.dev/list?ecosystem=) currently [does not support Nix ecosystem](https://ossf.github.io/osv-schema/#affectedpackage-field), so queries that specify Nix as ecosystem would not return any matches. For this reason `vulnxscan` currently does not use Google's official [OSV-Scanner](https://security.googleblog.com/2022/12/announcing-osv-scanner-vulnerability.html) front-end, but implements it's own OSV client demo in [osv.py](../src/vulnxscan/osv.py).
 
 `osv.py` sends queries to [OSV API](https://osv.dev/docs/) without specifying the ecosystem, only the target package name and version. At the time of writing, such queries to OSV API return vulnerabilities that match the given package and version across all ecosystems. As a result, the OSV vulnerabilities for Nix ecosystem will include false positives. 
 
@@ -59,7 +59,7 @@ Also, it is worth mentioning that OSV queries without ecosystem are undocumented
 ### Vulnix
 [Vulnix](https://github.com/nix-community/vulnix) is a vulnerability scanner intended for Nix targets. It uses [NIST NVD](https://nvd.nist.gov/vuln) vulnerability database.
 
-Vulnix matches vulnerabilities based on [heuristic](https://github.com/flyingcircusio/vulnix/blob/f56f3ac857626171b95e51d98cb6874278f789d3/src/vulnix/derivation.py#L104), which might result more false positives compared to direct match. False positives due to rough heuristic are an [intended feature](https://github.com/flyingcircusio/vulnix#whitelisting) in vulnix. On the other hand, vulnix accounts [CVE patches](https://github.com/flyingcircusio/vulnix#cve-patch-auto-detection) applied on Nix packages when matching vulnerabilities, something currently not directly supported by other scanners.
+Vulnix matches vulnerabilities based on [heuristic](https://github.com/nix-community/vulnix/blob/f56f3ac857626171b95e51d98cb6874278f789d3/src/vulnix/derivation.py#L104), which might result more false positives compared to direct match. False positives due to rough heuristic are an [intended feature](https://github.com/nix-community/vulnix#whitelisting) in vulnix. On the other hand, vulnix accounts [CVE patches](https://github.com/nix-community/vulnix#cve-patch-auto-detection) applied on Nix packages when matching vulnerabilities, something currently not directly supported by other scanners.
 
 ## Vulnxscan Usage Examples
 
@@ -371,5 +371,5 @@ For now, consider `vulnxscan` as a demonstration. Some improvement ideas are lis
  - Nix ecosystem is not supported in OSV: the way `osv.py` makes use of OSV data for Nix targets -- as explained in section [Nix and OSV vulnerability database](#nix-and-osv-vulnerability-database) -- makes the reported OSV vulnerabilities include false positives.
 
 ### Other Future Work
-- [vulnxscan](./vulnxscan.py) uses vulnix from a [forked repository](https://github.com/henrirosten/vulnix), to include vulnix support for [scanning runtime-only dependencies](https://github.com/flyingcircusio/vulnix/compare/master...henrirosten:vulnix:master).
-- [vulnxscan](./vulnxscan.py) could include more scanners in addition to [vulnix](https://github.com/flyingcircusio/vulnix), [grype](https://github.com/anchore/grype), and [osv.py](../src/vulnxscan/osv.py). Suggestions for other open-source scanners, especially those that can digest CycloneDX or SPDX SBOMs are welcome. Consider e.g. [bombon](https://github.com/nikstur/bombon) and [cve-bin-tool](https://github.com/intel/cve-bin-tool). Adding cve-bin-tool to vulnxscan was [demonstrated](https://github.com/tiiuae/sbomnix/pull/75) earlier, but not merged due to reasons explained in the [PR](https://github.com/tiiuae/sbomnix/pull/75#issuecomment-1670958503).
+- [vulnxscan](../src/vulnxscan/vulnxscan_cli.py) uses vulnix from a [forked repository](https://github.com/henrirosten/vulnix), to include vulnix support for [scanning runtime-only dependencies](https://github.com/nix-community/vulnix/compare/master...henrirosten:vulnix:master).
+- [vulnxscan](../src//vulnxscan/vulnxscan_cli.py) could include more scanners in addition to [vulnix](https://github.com/nix-community/vulnix), [grype](https://github.com/anchore/grype), and [osv.py](../src/vulnxscan/osv.py). Suggestions for other open-source scanners, especially those that can digest CycloneDX or SPDX SBOMs are welcome. Consider e.g. [bombon](https://github.com/nikstur/bombon) and [cve-bin-tool](https://github.com/intel/cve-bin-tool). Adding cve-bin-tool to vulnxscan was [demonstrated](https://github.com/tiiuae/sbomnix/pull/75) earlier, but not merged due to reasons explained in the [PR](https://github.com/tiiuae/sbomnix/pull/75#issuecomment-1670958503).
diff --git a/nix/checks.nix b/nix/checks.nix
@@ -33,7 +33,7 @@
           } ''
             cd ${../.}
             export HOME=/tmp
-            pylint --disable duplicate-code -rn $(find . -name "*.py" ! -path "*venv*" ! -path "*eggs*")
+            pylint --enable=useless-suppression --fail-on=I0021 --disable=duplicate-code -rn $(find . -name "*.py" ! -path "*venv*" ! -path "*eggs*")
             touch $out
           '';
       }

diff --git a/nix/devshell.nix b/nix/devshell.nix
@@ -36,6 +36,8 @@
       # invoking entrypoints from inside the devshell.
       shellHook = ''
         export PYTHONPATH="$PYTHONPATH:$(pwd)/src"
+        # https://github.com/NixOS/nix/issues/1009:
+        export TMPDIR="/tmp"
       '';
     };
   };

diff --git a/nix/packages.nix b/nix/packages.nix
@@ -160,9 +160,9 @@
           requests
           requests-cache
           requests-ratelimiter
+          setuptools
           tabulate
           venvShellHook
-          wheel
 
           # dev dependencies
           jsonschema

diff --git a/requirements.txt b/requirements.txt
diff --git a/setup.py b/setup.py
@@ -2,7 +2,7 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-# pylint: disable=invalid-name, import-error, missing-function-docstring
+# pylint: disable=missing-function-docstring
 
 """ setup.py for setuptools """
 
@@ -23,12 +23,18 @@ def project_path(*names):
 
 
 requires = [
+    "beautifulsoup4",
     "colorlog",
     "graphviz",
+    "numpy",
+    "pandas",
     "packageurl-python",
     "packaging",
-    "pandas",
     "reuse",
+    "requests",
+    "requests-cache",
+    "requests-ratelimiter",
+    "setuptools",
     "tabulate",
 ]
 

diff --git a/src/common/utils.py b/src/common/utils.py
@@ -2,7 +2,7 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-# pylint: disable=invalid-name
+# pylint: disable=invalid-name, abstract-method
 
 """ sbomnix utils """
 
@@ -19,6 +19,10 @@
 from colorlog import ColoredFormatter, default_log_colors
 import pandas as pd
 
+from requests import Session
+from requests_cache import CacheMixin
+from requests_ratelimiter import LimiterMixin
+
 ###############################################################################
 
 LOG_SPAM = logging.DEBUG - 1
@@ -280,6 +284,13 @@ def check_positive(val):
     return intval
 
 
+class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
+    """
+    Session class with caching and rate-limiting
+    https://requests-cache.readthedocs.io/en/stable/user_guide/compatibility.html
+    """
+
+
 ################################################################################
 
 set_log_verbosity(1)

diff --git a/src/nixupdate/nix_outdated.py b/src/nixupdate/nix_outdated.py
@@ -82,7 +82,7 @@ def _generate_sbom(target_path, runtime=True, buildtime=False):
     suffix = ".cdx.json"
     with NamedTemporaryFile(delete=False, prefix=prefix, suffix=suffix) as f:
         sbomdb.to_cdx(f.name, printinfo=False)
-        return f.name
+        return pathlib.Path(f.name)
 
 
 def _run_repology_cli(sbompath):
@@ -105,7 +105,7 @@ def _run_nix_visualize(targt_path):
     with NamedTemporaryFile(delete=False, prefix=prefix, suffix=suffix) as f:
         cmd = "nix-visualize " f"--output={f.name} {targt_path}"
         exec_cmd(cmd.split())
-        return f.name
+        return pathlib.Path(f.name)
 
 
 def _nix_visualize_csv_to_df(csvpath):
@@ -257,16 +257,21 @@ def main():
     exit_unless_nix_artifact(target_path_abs, force_realise=runtime)
 
     sbom_path = _generate_sbom(target_path_abs, runtime, args.buildtime)
-    LOG.info("Using SBOM '%s'", sbom_path)
+    LOG.debug("Using SBOM '%s'", sbom_path)
 
     df_repology = _run_repology_cli(sbom_path)
+    if LOG.level > logging.DEBUG:
+        sbom_path.unlink(missing_ok=True)
     df_log(df_repology, LOG_SPAM)
 
     if not args.buildtime:
         nix_visualize_out = _run_nix_visualize(target_path_abs)
-        LOG.info("Using nix-visualize out: '%s'", nix_visualize_out)
+        LOG.debug("Using nix-visualize out: '%s'", nix_visualize_out)
         df_nix_visualize = _nix_visualize_csv_to_df(nix_visualize_out)
         df_log(df_nix_visualize, LOG_SPAM)
+        if LOG.level > logging.DEBUG:
+            # Remove temp file unless verbosity is DEBUG or more verbose
+            nix_visualize_out.unlink(missing_ok=True)
     else:
         LOG.info("Not running nix-visualize due to '--buildtime' argument")
         df_nix_visualize = None

diff --git a/src/repology/repology_cli.py b/src/repology/repology_cli.py
@@ -4,9 +4,9 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-# pylint: disable=invalid-name, import-error, unexpected-keyword-arg,
-# pylint: disable=abstract-method, too-few-public-methods, too-many-statements
-# pylint: disable=too-many-instance-attributes, too-many-locals,
+# pylint: disable=invalid-name
+# pylint: disable=too-few-public-methods, too-many-statements
+# pylint: disable=too-many-instance-attributes, too-many-locals
 
 """ Command-line interface to repology.org """
 
@@ -16,9 +16,6 @@
 import re
 import urllib.parse
 from argparse import ArgumentParser, ArgumentTypeError, SUPPRESS
-from requests import Session
-from requests_cache import CacheMixin
-from requests_ratelimiter import LimiterMixin
 from bs4 import BeautifulSoup
 import numpy as np
 import pandas as pd
@@ -32,6 +29,7 @@
     df_regex_filter,
     nix_to_repology_pkg_name,
     parse_version,
+    CachedLimiterSession,
 )
 
 ###############################################################################
@@ -104,12 +102,6 @@ def getargs(args=None):
 ################################################################################
 
 
-class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
-    """Session class with caching and rate-limiting"""
-
-    # See: https://requests-cache.readthedocs.io/en/stable/user_guide/compatibility.html
-
-
 class Repology:
     """Query and parse Repology package data"""
 

diff --git a/src/repology/repology_cve.py b/src/repology/repology_cve.py
@@ -4,7 +4,7 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-# pylint: disable=invalid-name abstract-method too-many-locals
+# pylint: disable=invalid-name too-many-locals
 
 """ Command-line interface to query CVE info from repology.org """
 
@@ -13,9 +13,6 @@
 import re
 import urllib.parse
 from argparse import ArgumentParser, ArgumentTypeError
-from requests import Session
-from requests_cache import CacheMixin
-from requests_ratelimiter import LimiterMixin
 from bs4 import BeautifulSoup
 import numpy as np
 import pandas as pd
@@ -27,6 +24,7 @@
     set_log_verbosity,
     df_to_csv_file,
     parse_version,
+    CachedLimiterSession,
 )
 
 ###############################################################################
@@ -60,12 +58,6 @@ def getargs():
 ################################################################################
 
 
-class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
-    """Session class with caching and rate-limiting"""
-
-    # See: https://requests-cache.readthedocs.io/en/stable/user_guide/compatibility.html
-
-
 def _parse_cve_resp(resp, pkg_name, pkg_version):
     soup = BeautifulSoup(resp.text, "html.parser")
     tables = soup.find_all("table")

diff --git a/src/sbomnix/derivation.py b/src/sbomnix/derivation.py
@@ -4,7 +4,7 @@
 
 # SPDX-FileCopyrightText: 2022-2023 Technology Innovation Institute (TII)
 
-# pylint: disable=unnecessary-pass, invalid-name, eval-used
+# pylint: disable=invalid-name, eval-used
 # pylint: disable=too-many-instance-attributes
 
 """ Nix derivation, originally from https://github.com/flyingcircusio/vulnix """

diff --git a/src/vulnxscan/osv.py b/src/vulnxscan/osv.py
@@ -4,8 +4,6 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-# pylint: disable=import-error
-
 """ Demonstrate querying OSV db for vulnerabilities based on cdx SBOM """
 
 import argparse

diff --git a/src/vulnxscan/vulnxscan_cli.py b/src/vulnxscan/vulnxscan_cli.py
@@ -4,8 +4,7 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-# pylint: disable=invalid-name, import-error, too-many-arguments
-# pylint: disable=singleton-comparison, abstract-method
+# pylint: disable=invalid-name
 # pylint: disable=too-many-return-statements
 
 """
@@ -25,9 +24,6 @@
 from tempfile import NamedTemporaryFile
 from shutil import which
 
-from requests import Session
-from requests_cache import CacheMixin
-from requests_ratelimiter import LimiterMixin
 import pandas as pd
 import numpy as np
 
@@ -49,6 +45,7 @@
     nix_to_repology_pkg_name,
     parse_version,
     version_distance,
+    CachedLimiterSession,
 )
 
 ###############################################################################
@@ -393,10 +390,6 @@ def report(self, args, sbom_csv):
 # Triage
 
 
-class CachedLimiterSession(CacheMixin, LimiterMixin, Session):
-    """Session class with caching and rate-limiting"""
-
-
 _repology_cve_dfs = {}
 _repology_cli_dfs = {}
 # Rate-limited and cached session. For github api rate limits, see:
@@ -754,7 +747,7 @@ def _generate_sbom(target_path, runtime=True, buildtime=False):
     ) as fcsv:
         sbomdb.to_cdx(fcdx.name, printinfo=False)
         sbomdb.to_csv(fcsv.name, loglevel=logging.DEBUG)
-        return fcdx.name, fcsv.name
+        return pathlib.Path(fcdx.name), pathlib.Path(fcsv.name)
 
 
 def _is_json(path):
@@ -894,12 +887,16 @@ def main():
         sbom_cdx_path, sbom_csv_path = _generate_sbom(
             target_path_abs, runtime, args.buildtime
         )
-        LOG.info("Using cdx SBOM '%s'", sbom_cdx_path)
-        LOG.info("Using csv SBOM '%s'", sbom_csv_path)
+        LOG.debug("Using cdx SBOM '%s'", sbom_cdx_path)
+        LOG.debug("Using csv SBOM '%s'", sbom_csv_path)
         scanner.scan_vulnix(target_path_abs, args.buildtime)
     scanner.scan_grype(sbom_cdx_path)
     scanner.scan_osv(sbom_cdx_path)
     scanner.report(args, sbom_csv_path)
+    if not args.sbom and LOG.level > logging.DEBUG:
+        # Remove generated temp files unless verbosity is DEBUG or more verbose
+        sbom_cdx_path.unlink(missing_ok=True)
+        sbom_csv_path.unlink(missing_ok=True)
 
 
 if __name__ == "__main__":

diff --git a/tests/compare_deps.py b/tests/compare_deps.py
@@ -4,7 +4,7 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
-# pylint: disable=invalid-name, too-many-locals, import-error
+# pylint: disable=invalid-name, too-many-locals
 
 """ Python script that compares dependencies between sbomnix and nixgraph """