Installation instructions are missing a step

[JulianBirch]

On ubuntu 12.04, just referencing the plugin and lib as in the readme gives you the following:

Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library '/usr/java/packages/lib/amd64/libv8.so.clj-v8': /usr/java/packages/lib/amd64/libv8.so.clj-v8: cannot open shared object file: No such file or directory

[pbiggar]

Good point. I just committed a fix in ee68022

[JulianBirch]

I'm afraid this isn't working for me. Same error :(

** deleted a bit that was a brain failure on my part **

Oh, and you need a closing " on the jvm-opts.

[JulianBirch]

I tried

["-Djava.library.path=target/native/macosx/x86_64:target/native/linux/x86_64:target/native/linux/x86:native/macosx/x86_64:native/linux/x86_64:native/linux/x86:"]

as well (ripped from clj-v8). That didn't work either. Checked the jar, and the files are definitely there.

[pbiggar]

Fix the typo, thanks!

I have no idea about the other problem sorry. It works for us in production on 64bit 12:04 :(

[magnars]

I have a similar problem when deploying an uberjar.

It looks to me like clj-v8 reads the binaries from the file system. (https://github.com/circleci/clj-v8/blob/master/clj-v8/src/v8/core.clj#L20-L23)

Shouldn't these be read off of the class path?

[magnars]

@pbiggar Does it also work for you in an uberjar? It seems like it is looking for the binaries in /usr/java/packages/lib/amd64/libv8.so.clj-v8, and of course there's nothing there. They're under native/linux/x86_64 on the class path.

The comment on https://github.com/circleci/clj-v8/blob/master/clj-v8/src/v8/core.clj#L17-L19 seems to imply that it is a problem with version numbers that require the absolute path lookup. Can that be mended in some other way, like using only digits and dots for versions?

[magnars]

It might not have anything to do with linux either, because just running an uberjar on the command line in OSX reveals similar problems:

> java -jar our-project-0.1.0-SNAPSHOT-standalone.jar
Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'v8wrapper': dlopen(libv8wrapper.dylib, 9): image not found

Does clj-v8 support having the binaries in jars? Does jna?

[pbiggar]

We're not using an uberjar in production, so I wouldn't be surprised if that's the problem.

Can that be mended in some other way, like using only digits and dots for versions?

Maybe, but we need to have a version which doesn't conflict with another possible version number, since libv8 will likely exist on the system.

But I agree with your diagnosis of where the problem likely lies.

[magnars]

It seems like this StackOverflow question has the solution, basically use System.load(File) instead of System.loadLibrary(String).

http://stackoverflow.com/questions/2937406/how-to-bundle-a-native-library-and-a-jni-library-inside-a-jar?rq=1

Is this something you are interested in supporting?

[pbiggar]

Yes please!

[magnars]

Ok, I'll give it a go.

[magnars]

This is what I have so far:

(require '[clojure.java.io :as io])
(import '[java.io File FileOutputStream])

(defn load-library-from-class-path
  [name extension]
  (let [tmp (File/createTempFile name extension)
        lib (io/resource (str "native/macosx/x86_64/" name extension))
        in (.openStream lib)
        out (FileOutputStream. tmp)]
    (io/copy in out)
    (.close out)
    (.close in)
    (System/load (.getAbsolutePath tmp))))

(load-library-from-class-path "libv8" ".dylib.clj-v8")
(load-library-from-class-path "libv8wrapper" ".dylib")

Loading libv8 works, but I still get

java.lang.UnsatisfiedLinkError: /private/var/folders/11/2gxd8c_13zx2l_3qkkby86jh0000gn/T/libv8wrapper3107856195926061800.dylib: dlopen(/private/var/folders/11/2gxd8c_13zx2l_3qkkby86jh0000gn/T/libv8wrapper3107856195926061800.dylib, 1): Library not loaded: @loader_path/libv8.dylib.clj-v8
  Referenced from: /private/var/folders/11/2gxd8c_13zx2l_3qkkby86jh0000gn/T/libv8wrapper3107856195926061800.dylib
  Reason: image not found

when trying to load libv8wrapper. I guess it's because the loader doesn't see that its dependency is already loaded.

I'll continue beating at it, but if you have any ideas or tips, I'm all ears. :)

[magnars]

I just realized this belongs in the clj-v8 repo. If you would like to move there, I'd be happy to open an issue.

[magnars]

What is the function of libv8wrapper btw? Would it be a possible strategy to just use libv8 directly? Seems like that would simplify the loading quite a bit.

[magnars]

The Java ZeroMQ wrapper has this working, by the way. It doesn't have the issue with one native lib depending on another, tho.

https://github.com/zeromq/jzmq/blob/master/src/org/zeromq/EmbeddedLibraryTools.java

[magnars]

Ok, moving along at a halting pace. :)

If I avoid File/createTempFile and instead create it myself with the proper name, then I can load both libv8 and libv8wrapper. Success!

[magnars]

@pbiggar

Sorry about the rubberducking. :)

Would you mind giving this a go?

(require '[clojure.java.io :as io])
(import '[java.io File FileOutputStream])

(defn- find-binary-path
  []
  (let [os-name (System/getProperty "os.name")
        os-arch (System/getProperty "os.arch")
        id (case os-name
             "Mac OS X" "macosx"
             "Linux" "linux"
             (throw (Exception. (str "Unsupported OS: " os-name))))
        arch (case os-arch
               "x86" "x86"
               "x86_64" "x86_64"
               "amd64" "x86_64" ;; need to compile own binaries for this?
               (throw (Exception. (str "Unsupported archetype: " os-arch))))]
    (str id "/" arch)))

(defn load-library-from-class-path
  [name extension]
  (let [tmp (File. (File. (System/getProperty "java.io.tmpdir")) (str name extension))
        lib (io/resource (str "native/" (find-binary-path) "/" name extension))
        in (.openStream lib)
        out (FileOutputStream. tmp)]
    (io/copy in out)
    (.close out)
    (.close in)
    (System/load (.getAbsolutePath tmp))))

(try
  (System/loadLibrary "v8wrapper")
  (catch UnsatisfiedLinkError e
    (load-library-from-class-path "libv8" ".dylib.clj-v8")
    (load-library-from-class-path "libv8wrapper" ".dylib")))

(def LIBRARY (com.sun.jna.NativeLibrary/getInstance "v8wrapper"))

[magnars]

I'd be happy to do it myself, but running lein test in clj-v8/clj-v8 gives me a very familiar error out of the box:

Caused by: java.lang.UnsatisfiedLinkError: Unable to load library 'v8wrapper': dlopen(/Users/magnars/stuff/clj-v8/clj-v8/target/native/macosx/x86_64/libv8wrapper.dylib, 9): Library not loaded: /usr/local/lib/libv8.dylib
  Referenced from: /Users/magnars/stuff/clj-v8/clj-v8/target/native/macosx/x86_64/libv8wrapper.dylib
  Reason: image not found

So I'm not sure how you're running your tests.

[JulianBirch]

Ironically, this is another of the issues I've reported: circleci/clj-v8#6

[magnars]

The code above only works on Mac atm, since I included dylib. I'm working on a version that uses so on linux.

[magnars]

Here's the code that finds .so instead of .dylib on linux.

(defn- find-file-path-fragments
  []
  (let [os-name (System/getProperty "os.name")
        os-arch (System/getProperty "os.arch")]
    (case [os-name os-arch]
      ["Mac OS X" "x86_64"] ["macosx/x86_64/" ".dylib"]
      ["Linux" "x86_64"]    ["linux/x86_64/" ".so"]
      ["Linux" "amd64"]     ["linux/x86_64/" ".so"]
      ["Linux" "x86"]       ["linux/x86/" ".so"]
      (throw (Exception. (str "Unsupported OS/archetype: " os-name " " os-arch))))))

(defn load-library-from-class-path
  [name path-postfix]
  (let [[binary-path binary-extension] (find-file-path-fragments)
        file-name (str name binary-extension path-postfix)
        tmp (File. (File. (System/getProperty "java.io.tmpdir")) file-name)
        lib (io/resource (str "native/" binary-path file-name))
        in (.openStream lib)
        out (FileOutputStream. tmp)]
    (io/copy in out)
    (.close out)
    (.close in)
    (System/load (.getAbsolutePath tmp))))

(try
  (System/loadLibrary "v8wrapper")
  (catch UnsatisfiedLinkError e
    (load-library-from-class-path "libv8" ".clj-v8")
    (load-library-from-class-path "libv8wrapper" "")))

(def LIBRARY (com.sun.jna.NativeLibrary/getInstance "v8wrapper"))

Right now it works spendidly on my Mac, using the native files from the Jar. On my Ubuntu it loads both libraries from the class path without error, but then JNA fails to load v8wrapper anyway. I'm kinda stuck at this point. Maybe a way forward is to replace JNA with a hand-made JNI interface.

If you have any ideas, please do advice. :)

[magnars]

Okay, so here's the deal:

• the jna.NativeLibrary loader requires that the file is to be found on disk.
• if it is on disk, but already loaded, then it uses the one that's already loaded.

So, to get this to work on both OSX and Linux, just add:

(System/setProperty "jna.library.path" (System/getProperty "java.io.tmpdir"))

right before

(def LIBRARY (com.sun.jna.NativeLibrary/getInstance "v8wrapper"))

It doesn't feel good, but that's what it takes.

I guess this is a peculiarity of JNA, meaning you could probably avoid this sort of trickery by switching to plain JNI - but that is pretty painful as well.

I'll see if I can get the clj-v8 tests to run using this. Otherwise I hope you can pick up the ball from here, @pbiggar - unless you feel this sort of mess is not worth having clj-v8 support living in an uberjar.

[pbiggar]

I'm really not sure what to do here. AFAICT, this fixes some uberjar problems, but adds some other problems? Is my understanding correct?

In general in these cases, I'll prefer to leave the patch unmerged and wait for a complete solution. I know that's a pain in the arse to solve your use case though :(

[magnars]

The comments above chronicle my blundering steps towards the solution, but the pull request works in all cases that I know of. There might be regressions that I don't know about, since I am unable to run the tests (with or without the patch), and since there are no instructions on how to get the tests running. However, with this fix it now works both as a regular dependency, and bundled in an uberjar.

[magnars]

Let me add that I've tested the pull request code on both OSX and Linux (Ubuntu), both with clj-v8 as a regular dependency and bundled in an uberjar. So as far as I can tell, it works. :) I'm just not able to run the tests.