Merge pull request #21 from Cisco-Talos/dev

Release accompagnying SigAnalyzer blog post

README.md

@@ -14,16 +14,40 @@ engineers in creating ClamAV NDB and LDB signatures from IDA Pro's Disassembly
 or Strings view.
 
 CASC should run on any platform that supports IDA Pro 6.7 and higher.
-Limited functionality is available for IDA Pro 6.6
+Limited functionality is available for IDA Pro 6.6.
+The signature analyzer part of CASC has only been tested on IDA 6.9.
 
 README with pictures can be found on our wiki:
     [https://github.com/vrtadmin/casc/wiki](https://github.com/vrtadmin/casc/wiki)
 
-## Installation
+## Building
+You need a working build system to build the yara-python library. On Ubuntu,
+you can get that by installing `sudo apt-get install -y build-essential
+libpython2.7-dev libpython2.7-dev:i386 gcc-multilib libssl-dev libssl-dev:i386`
+on a 64 bit system (The build process has been developed for a 64 bit machine,
+and will not work out of the box on a 32 bit machine).
 
-The ClamAV Signature Creator (CASC) is easy to install. Simply copy and paste 
-the Python script (`clamav_sig_creator.py`) to IDA Pro’s plug-in directory
+Run `python package.py --output <output-dir>` to build the plugin zip archives.
+This will build all combinations of 32/64 versions for Windows and Linux.
 
+## Installation
+If you are using your system's python for IDA Pro (probably the case if you're
+on a 32 bit Linux system, or on a 64 bit Linux system and you're using IDA 7.0
+or higher), you can install the packages _ply_, _yara-python_ and
+_ida-netnode_, and then unzip the _casc-\*-universal.zip_ archive into your IDA
+Pro directory.
+
+Otherwise, if you use the python bundled with IDA Pro, you'll need to install
+the libraries in this python. Either you can do this by yourself (e.g., by
+following the instructions from [hexblog](http://www.hexblog.com/?p=726)), or
+you use the archives with bundled dependencies that we provide. Those archives
+are built as described in the [building][#Building] step above, by running
+_package.py_. To install an archive, simply pick the _casc-\*-fat.zip_
+corresponding to your system, and unzip it in your IDA Pro directory.
+
+In case you don't want to install additional libraries, the plugin will degrade
+gracefully and hide the _"Analyze"_ functionality which requires the libraries to
+be installed.
 
 | Operating System               | IDA Pro Plug-in Path                     |
 | ------------------------------ | -----------------------------------------|
@@ -42,6 +66,8 @@ Tested on
 ---------
 | IDA Pro Version | OK | Notes                                              |
 | ----- | --------------- | ---------------------------------------------------|
+| 7.0 | Y            |                                                    |
+| 6.95| Y            |                                                    |
 | 6.7 | Y            |                                                    |
 | 6.6 | Y            | Doesn't support right click option in IDA View or Strings Windows |
 | 6.5 | N            | IDA doesn't provide PySide and Qt support          |
@@ -144,8 +170,20 @@ Once the “Create ClamAV Signature” button is click a dialog box with a
 formatted email will be displayed for the user to send to ClamAV’s 
 community-sigs list. Selecting the [[email protected]]([email protected]) hyperlink  is a mailto: link. It will attempt to copy the signature information displayed  to the systems default mail client. Keep in mind if any special characters are  used then the email’s contents may not be correct and will need to be manually  copied over.
 
-## Bugs and Support
+## Analyzing ClamAV signatures
+In the main plugin panel, use the tabbed pane to switch to the _"Analyze"_
+mode. Note that this tab is not available if you don't have the required
+libraries installed.  You can now paste a ClamAV .ldb or .ndb signature in the
+text field above the _"Add Signature"_ button. Once you then click _Add
+signature_, the signature will appear in the left top list (if it was well
+formatted and parsed by the plugin, otherwise check the IDA Pro output window
+for errors). Now you can double-click on the signature in the list. For .ndb
+signatures this will directly bring you to the matched part of the binary,
+and color the match in red. For .ldb signatures, the subsignatures will be
+displayed in the right list element. If you double-click any of the sub
+signature entries, it will bring you to the match for this subsignature.
 
+## Bugs and Support
 There is no support provided with CASC. 
 
 If you think you've found a bug, please report it at:

package.py

@@ -0,0 +1,162 @@
+#!/usr/bin/env python
+
+import sys
+import os
+import argparse
+import tempfile
+import zipfile
+import shutil
+import subprocess
+import urllib
+import shutil
+import errno    
+
+from functools import partial
+
+
+def mkdir_p(path):
+    try:
+        os.makedirs(path)
+    except OSError as exc:  # Python >2.5
+        if exc.errno == errno.EEXIST and os.path.isdir(path):
+            pass
+        else:
+            raise
+
+class GitPythonDependency(object):
+    def __init__(self, url):
+        self.dir = tempfile.mkdtemp()
+        self.url = url
+        self.fetch()
+
+    def destroy(self):
+        shutil.rmtree(self.dir)
+
+    def fetch(self):
+        cmd = ("git", "clone", self.url, self.dir)
+        subprocess.check_call(cmd)
+
+    def install(self, path, bits):
+        env = {"PYTHONPATH": os.path.join(path, "lib/python2.7/site-packages")}
+        cmd = ("python", "setup.py", "install", "--prefix", path)
+        subprocess.check_call(cmd, cwd = self.dir, env = env)
+
+class YaraPythonWinDependency(object):
+    YARA_PYTHON_32BIT_URL = "https://www.dropbox.com/sh/umip8ndplytwzj1/AADuLJ_5Sa279u0fKplRbkOZa/yara-python-3.6.1.win32-py2.7.exe?dl=1"
+    YARA_PYTHON_64BIT_URL = "https://www.dropbox.com/sh/umip8ndplytwzj1/AADetRlpZd-Nd4slSkud78Qxa/yara-python-3.6.3.win-amd64-py2.7.exe?dl=1"
+
+    def __init__(self):
+        handle, self.installer_32bit = tempfile.mkstemp()
+        os.close(handle)
+        handle, self.installer_64bit = tempfile.mkstemp()
+        os.close(handle)
+        cmd = ("curl", "-L", "-o", self.installer_32bit, self.YARA_PYTHON_32BIT_URL)
+        subprocess.check_call(cmd)
+        cmd = ("curl", "-L", "-o", self.installer_64bit, self.YARA_PYTHON_64BIT_URL)
+        subprocess.check_call(cmd)
+
+    def destroy(self):
+        os.unlink(self.installer_32bit)
+        os.unlink(self.installer_64bit)
+
+    def install(self, path, bits):
+        if bits == 32:
+            cmd = ("unzip", "-j", "-d", os.path.join(path, "lib/python2.7"), self.installer_32bit, "PLATLIB/yara.pyd")
+        else:
+            cmd = ("unzip", "-j", "-d", os.path.join(path, "lib/python2.7"), self.installer_64bit, "PLATLIB/yara.pyd")
+        # unzip will return status 1 because the file is a self-extracting archive, where the self extractor
+        # is detected as junk 
+        subprocess.call(cmd)
+
+class YaraPythonLinuxDependency(object):
+    YARA_PYTHON_GIT_URL = "https://github.com/VirusTotal/yara-python"
+
+    def __init__(self):
+        self.yara_python_dir = tempfile.mkdtemp()
+        cmd = ("git", "clone", "--recursive", self.YARA_PYTHON_GIT_URL, self.yara_python_dir)
+        subprocess.check_call(cmd)
+
+    def destroy(self):
+        shutil.rmtree(self.yara_python_dir)
+
+    def install(self, path, bits):
+        env = os.environ.copy()
+        env["PYTHONPATH"] = os.path.join(path, "lib/python2.7/site-packages")
+        if bits == 32:
+            env["CFLAGS"] = "-m32"
+            env["CXXFLAGS"] = "-m32"
+            env["LDFLAGS"] = "-m32"
+        cmd = ("python", "setup.py", "clean")
+        subprocess.check_call(cmd, cwd = self.yara_python_dir)
+        if os.path.exists(os.path.join(self.yara_python_dir, "build")):
+            shutil.rmtree(os.path.join(self.yara_python_dir, "build"))
+        cmd = ("python", "setup.py", "install", "--prefix", path)
+        subprocess.check_call(cmd, cwd = self.yara_python_dir, env = env)
+
+def get_plugin_git_version():
+    try:
+        return subprocess.check_output(["git", "describe", "--always", "--tags"]).split("\n")[0].strip() or "devel"
+    except OSError:
+        return "unknown"
+
+def main(args):
+    my_directory = os.path.split(os.path.abspath(__file__))[0]
+    version = get_plugin_git_version()
+
+    ply = GitPythonDependency("https://github.com/dabeaz/ply")
+    idann = GitPythonDependency("https://github.com/williballenthin/ida-netnode")
+
+    yara_python_win = YaraPythonWinDependency()
+    yara_python_linux = YaraPythonLinuxDependency()
+
+    def fetch_windows_dependencies(path, bits):
+        ply.install(os.path.join(path, "python"), bits)
+        idann.install(os.path.join(path, "python"), bits)
+        yara_python_win.install(os.path.join(path, "python"), bits)
+
+    def fetch_linux_dependencies(path, bits):
+        ply.install(os.path.join(path, "python"), bits)
+        idann.install(os.path.join(path, "python"), bits)
+        yara_python_linux.install(os.path.join(path, "python"), bits)
+
+    architectures = [
+        ("universal", lambda x: None),
+        ("windows_32bit_fat", partial(fetch_windows_dependencies, bits = 32)),
+        ("windows_64bit_fat", partial(fetch_windows_dependencies, bits = 64)),
+        ("linux_32bit_fat", partial(fetch_linux_dependencies, bits = 32)),
+        ("linux_64bit_fat", partial(fetch_linux_dependencies, bits = 64))]
+
+
+    for arch, fetcher in architectures:
+        directory = tempfile.mkdtemp()
+        mkdir_p(os.path.join(directory, "plugins"))
+        mkdir_p(os.path.join(directory, "python/lib/python2.7/site-packages"))
+        print("my_directory: {}".format(my_directory))
+        plugin_files = subprocess.check_output(("git", "ls-files"), \
+                    cwd = os.path.join(my_directory, "plugin"), 
+                    env = os.environ).strip().split("\n")
+        for f in plugin_files:
+            destpath = os.path.join(directory, "plugins", os.path.split(f)[0])
+            mkdir_p(destpath)
+            shutil.copy2(os.path.join(my_directory, "plugin", f), os.path.join(directory, "plugins", f))
+        fetcher(directory)
+        files = [os.path.join(d, f) for d, _, files in os.walk(directory) for f in files]
+        with zipfile.ZipFile(os.path.join(args.output, "casc_{}_{}.zip".format(version, arch)), "w") as archive:
+            for path in files:
+                archive.write(path, os.path.relpath(path, directory))
+        shutil.rmtree(directory)
+
+    ply.destroy()
+    idann.destroy()
+    yara_python_win.destroy()
+    yara_python_linux.destroy()
+
+def parse_args():
+    parser = argparse.ArgumentParser(description = "Create an installable archive of the CASC plugin")
+    parser.add_argument("--output", type = str, default = os.getcwd(), help = "Output directory")
+
+    return parser.parse_args()
+
+if __name__ == "__main__":
+    main(parse_args())
+