Embedding docs out of date

[samuelcolvin]

https://github.com/JuliaLang/julia/blob/master/doc/manual/embedding.rst

The advice on compiling the test:

gcc -o test -I$JULIA_DIR/include/julia -L$JULIA_DIR/usr/lib -ljulia test.c

is wrong. For a start the directory $JULIA_DIR/include/julia doesn't exist.

This also affects the embedding example https://github.com/JuliaLang/julia/blob/master/examples/embedding.c as the manual page above seems to be the goto advice on how to compile it.

See google user group question.

[samuelcolvin]

Maybe at least partly a duplicate of #5834 however following their instructions I still couldn't get test.c to compile.

[tknopp]

The embedding.c example is build by running make in the examples/ directory. Seems to work here.
$JULIA_DIR has to be replaced by the appropriate directory path.

[samuelcolvin]

Thanks Tobias for your help - I had missed the Makefile in examples. Running make -n printed the actual gcc and g++ commands, basically the first gcc command compiles without linking then a g++ command links (I think). The vital thing is that gcc uses -c - without linking, if you add that to my gcc original command:

gcc -o test -I$JULIA_DIR/src -I$JULIA_DIR/src/support -I$JULIA_DIR/usr/include -c test.c

test.c compiles but can't be executed because it's not linked(??). I ended up getting test.c to compile with clang (after reading this), using the following command:

sudo ldconfig $JULIA_DIR/usr/lib/
clang -o test -I$JULIA_DIR/src/ -I$JULIA_DIR/src/support/ -I$JULIA_DIR/usr/include/ -L$JULIA_DIR/usr/lib/ -ljulia test.c

I also had to alter the code:

jl_init("/path/to/julia/usr/lib/");

To get it to run.

Basically the issue stands: the docs on embedding are wrong and the solution is not obvious. Either they need to be fixed to use clang, and the right paths; or fixed to use gcc correctly (somehow??), and the right paths.

[tknopp]

yes you are right that docu issues should be fixed. But I am not entirely sure what the issue is. When you compile test.c, the libjulia.so has to be in the shared library path when you use it. Of course it would be helpful when this is mentioned in the docu.

I am not sure about the other issue though. Maybe this could be solved by #5045.

[samuelcolvin]

Personally all I'm really looking for is an example of how to compile test.c using gcc - eg. without the need to install clang.

My eventual aim is to build a python library for calling Julia that does all the grunt work in c. Eg. like pyjulia but written predominently in c not python. For this I need to compile a julia library using setuptools.

[tknopp]

Ok, if you do

gcc -o test -I$JULIA_DIR/src -I$JULIA_DIR/src/support -I$JULIA_DIR/usr/include -c test.c

you are not linking against libjulia. The original line from the docu was

gcc -o test -I$JULIA_DIR/include/julia -L$JULIA_DIR/usr/lib -ljulia test.c

which links against libjulia and tells it where the header files are and where libjulia is located. It seems that the issue is that there is an usr missing in that line. Could you try

gcc -o test -I$JULIA_DIR/usr/include/julia -L$JULIA_DIR/usr/lib -ljulia test.c

and if this does not work either additionally put -I$JULIA_DIR/src/ -I$JULIA_DIR/src/support/ to the line?

[tknopp]

And just as a side note: When I wrote the embedding docu (which was unitl that point non-existent) I was aware that its quite hard to educate how to compile C code in a cross platform way. Everybody uses a different tool for building C code (Makefile, CMake, Autotools, Visual Studio, ...) and this gcc-comand line I put to the docu was more meant as a example.

[jakebolewski]

@samuelcolvin are you using Ubuntu by chance? If you are you need to set up the paths differently, search the julia-dev list for details about how link to libjulia in this case.

Offtopic, but have yoiu read all the pros/cons about what you intend to do on the julia-users mailing list? This has been brought up multiple times as a GSOC project.

[tknopp]

@jakebolewski good point. Do you know if the directory structure is somehow different for the Ubuntu binaries? In jl_init there is a heuristic where to search for the system image and it is based on the julia home dir. I not entirely sure how all this work, but it might be more robust to take some relative path to the libjulia.so path which can be determined quite easily (see #5045)

[samuelcolvin]

Sorry, yes Ubuntu 13.10 64 bit.
@tknopp I tried that command and get fatal error: julia.h: No such file or directory

If I add those two includes I get fatal error: uv.h: No such file or directory.

uv.h is in usr/include/ if I add that path I get back to the error I described on the user group:

gcc -o test -I$JULIA_DIR/usr/include/julia -I$JULIA_DIR/usr/include -I$JULIA_DIR/src/ -I$JULIA_DIR/src/support/ -L$JULIA_DIR/usr/lib -ljulia test.c
/tmp/ccpB8Jrh.o: In function `main':
test.c:(.text+0x15): undefined reference to `jl_init'
test.c:(.text+0x1c): undefined reference to `jl_root_task'
test.c:(.text+0x2e): undefined reference to `jl_root_task'
test.c:(.text+0x4f): undefined reference to `jl_handle_stack_switch'
test.c:(.text+0x59): undefined reference to `jl_eval_string'

And round in circles we go. As reference, the make commands which do work for embedding.c are:

gcc  -std=gnu99 -pipe -fPIC -fno-strict-aliasing -D_FILE_OFFSET_BITS=64 -O3 -falign-functions -momit-leaf-frame-pointer -Wall -Wno-strict-aliasing -fno-omit-frame-pointer -I/home/samuel/julia/src -I/home/samuel/julia/src/support -I/home/samuel/julia/usr/include -std=gnu99 -pipe -fPIC -fno-strict-aliasing -D_FILE_OFFSET_BITS=64  -c embedding.c -o embedding.o

g++  -O3 -falign-functions -momit-leaf-frame-pointer -Wall -Wno-strict-aliasing -fno-omit-frame-pointer -I/home/samuel/julia/src -I/home/samuel/julia/src/support -I/home/samuel/julia/usr/include -std=gnu99 -pipe -fPIC -fno-strict-aliasing -D_FILE_OFFSET_BITS=64  embedding.o -o /home/samuel/julia/usr/bin/embedding -L/home/samuel/julia/usr/lib/julia -L/home/samuel/julia/usr/lib -ljulia -Wl,-Bdynamic  -Wl,--no-whole-archive -L/home/samuel/julia/usr/lib  -lz -ldl -lm  -ldl -lrt -lpthread -Wl,--export-dynamic -Wl,--version-script=/home/samuel/julia/src/julia.expmap -Wl,--no-whole-archive /home/samuel/julia/usr/lib/libunwind-generic.a /home/samuel/julia/usr/lib/libunwind.a -Wl,-rpath,'$ORIGIN/../lib/julia' -Wl,-rpath,'$ORIGIN/../lib' -Wl,-z,origin

@jakebolewski Yes I've read numerous things, I just thought it would be worth trying a julia python lib written in c for the sake of speed. Which pros/cons did you mean exactly?

[jakebolewski]

Here in the link to the relevant comment https://groups.google.com/d/msg/julia-dev/36fFFLW25uk/MHciMbopHHYJ

But it looks like you are not using the Ubuntu binaries, so this is a moot point.
I thought that Julia ships with a patched version of libuv? That might be the root of your problem.

[tknopp]

I am a little suprised that the Makefile uses g++ for linking.

But still I do not understand why the jl_* functions are not found in libjulia.so. Unfortunatly I don't have a linux box so that I cannot track down why this happens. Would be great if someone other having access to linux could have a look at this.

[samuelcolvin]

In the mean time shouldn't someone with such permissions give this labels of doc and linux?

[cmundi]

@samuelcolvin OK, this may help.

I'm going to assume

export JULIA=<your-top-level-julia-dir>

and just to have a specific case to work with, put your test.c (which I will assume is taken verbatim from the Julia manual) in $JULIA/examples. Now do this:

gcc -c test.c -o test.o -I$JULIA/src -I$JULIA/src/support -I$JULIA/usr/include
gcc test.o -o $JULIA/usr/bin/test -L$JULIA/usr/lib -ljulia -Wl,-rpath,'$ORIGIN/../lib/julia' -Wl,-rpath,'$ORIGIN/../lib' -Wl,-z,origin

Now you should be able to do

../usr/bin/test
1.4142135623730951

Assuming all went, pause to enjoy the moment.

So what's going on here? First, I make no claim that these options should be used in production; they shouldn't be! But this is about as minimal an example as I could create and definitely not optimal in any other sense. (Please, no wagering. It's a demonstration, not a competition.) All I've done is seriously strip down the Makefile in the examples directory. The only things that make this a little tricky are the linker -rpath options passed with -Wl,* and that is pretty tricky stuff.

Specifying the libs (with -L and -l) is enough to let the linker resolve the libjulia.so symbols. But if we want the loader to be able to resolve the lib at runtime, then we need those miserable -rpath options which hard-code the location of the lib relative to the executable (i.e., $ORIGIN) into the executable test. You can convince yourself of what's going on by running ldd test on test where it is supposed to be ($JULIA/usr/bin/test) and again in any location which is not and "up one and over" sibling of that location. Just for fun you could change the linker options to look in the executable's folder and drop a copy of libjulia.so in there. That's ugly, so you might not want to make it a habit.

I'd suggest RTFM, but that would make me a jerk. Instead, you might want to read this blog post which is probably the best discussion I've seen on the topic.

One final note... master this example before you try building any shared libs which link with libjulia.so. Just sayin'.

[tknopp]

Shouldn't it be sufficient to put the libjulia directory to the LD_LIBRARY_PATH?

The three include dirs are really only necessary when trying to compile something against the julia build dir. After installation there is one include folder where all the necessary files are in.

[cmundi]

Yes. LD_LIBRARY_PATH is arguably a brittle solution but I admit to using it
when I'm in a hurry.

Yes. The docs should assume an installation. My impression was that the OP
was using the build dir but I might be wrong about that.
On Apr 5, 2014 11:02 PM, "Tobias Knopp" [email protected] wrote:

Shouldn't it be sufficient to put the libjulia directory to the
LD_LIBRARY_PATH?

The three include dirs are really only necessary when trying to compile
something against the julia build dir. After installation there is one
include folder where all the necessary files are in.

Reply to this email directly or view it on GitHubhttps://github.com//issues/6327#issuecomment-39659482
.

[tknopp]

Right, but this bug report is about the docs beeing wrong and I try to find out what we can do to make them more clear. Maybe we should remove the gcc command from the docs and just say that the include and library path should be properly set.

[cmundi]

I think I agree, provided the docs actually tell the reader what includes
and libs are required, and then the reader can take whatever steps are
appropriate to his or her platform toolchain. Eventually, there will be
Julia tutorials and howtos. I see build/embedding instructions as something
better for a readme. But then again, I only ever read the appendix to K&R
so maybe I'm weird.
On Apr 7, 2014 12:37 AM, "Tobias Knopp" [email protected] wrote:

Right, but this bug report is about the docs beeing wrong and I try to
find out what we can do to make them more clear. Maybe we should remove the
gcc command from the docs and just say that the include and library path
should be properly set.

Reply to this email directly or view it on GitHubhttps://github.com//issues/6327#issuecomment-39703021
.

[tknopp]

When I wrote this I wanted to give this a "hands on" touch to show that it is really no magic to embed Julia. But of course without knowledge of C and how to build C programs in practice the embedding docu is not useful anyway.

[cmundi]

I still think it is useful to show how easy embedding can be, for people
who know their tools. So I would keep the code example, remove the gcc
command line, and just tell people

1. What files they need to compile/link against.
2. Please refer to the toolchain docs for their platform.

Done. Easy and effortless don't need to mean the same thing. :)
On Apr 7, 2014 8:15 AM, "Tobias Knopp" [email protected] wrote:

When I wrote this I wanted to give this a "hands on" touch to show that it
is really no magic to embed Julia. But of course without knowledge of C and
how to build C programs in practice the embedding docu is not useful anyway.

Reply to this email directly or view it on GitHubhttps://github.com//issues/6327#issuecomment-39742978
.

[samuelcolvin]

Thanks a lot for your help.

The key here I think is to make it clear that the process is significantly different for different platforms, and state what files they need to compile/link against.

After that I don't see the problem with giving a demonstration for particular OS's; since it won't take long to write, why not give as much guidance as possible?

For reference, with test.c changed to include jl_init("/path/to/julia/usr/lib/"); the build process can be as simple as:

export JULIA=/path/to/julia
gcc -c test.c -o test.o -I$JULIA/src -I$JULIA/src/support -I$JULIA/usr/include
gcc test.o -o test -L$JULIA/usr/lib -ljulia -Wl,-rpath,'/path/to/julia/usr/lib'

With that test can be run from anywhere. Just adding that to the docs would be a great start.

[samuelcolvin]

Turns out it can be put into one command to be even simpler:

export JULIA=/path/to/julia
gcc test.c -o test -I$JULIA/src -I$JULIA/src/support -I$JULIA/usr/include -L$JULIA/usr/lib -ljulia -Wl,-rpath,'/path/to/julia/usr/lib'

In short it was the -rpath options I was missing.

[cmundi]

Super! Glad to see the progress. File thus under "easy in retrospect."

Maybe you'd be willing to try your hand at adding some helpful notes to the
docs? I suspect these sorts of things will eventually get their own place
(and this thread may hasten the day) but until then the manual is probably
where most people will look.
On Apr 7, 2014 9:05 AM, "Samuel Colvin" [email protected] wrote:

Turns out it can be put into one command to be even simpler:

export JULIA=/path/to/julia
gcc test.c -o test -I$JULIA/src -I$JULIA/src/support -I$JULIA/usr/include -L$JULIA/usr/lib -ljulia -Wl,-rpath,'/path/to/julia/usr/lib'

In short it was the -rpath options I was missing.

Reply to this email directly or view it on GitHubhttps://github.com//issues/6327#issuecomment-39749260
.

[samuelcolvin]

Will do when I'm less manic, might be a couple of weeks but it's on my TODO list.

[mdcfrancis]

With the latest 0.4 release I'm getting similar linker errors

gcc -o test -I$JULIA_DIR/include/julia -L$JULIA_DIR/lib/julia -ljulia test.cpp -Wl,-rpath,'$JULIA_DIR/lib/julia'
/tmp/ccj36oRo.o: In function main': test.cpp:(.text+0x15): undefined reference tojl_init'
test.cpp:(.text+0x1f): undefined reference to jl_eval_string' test.cpp:(.text+0x29): undefined reference tojl_atexit_hook'
collect2: error: ld returned 1 exit status

[ihnorton]

@mdcfrancis the link targets need to be after the source file. The following works for me (on Ubuntu 14.04):

cd usr/lib
gcc -g -o embedding -I../../src -I../../src/support/ -I../../usr/include -L./ ../../examples/embedding.c -ljulia -lm -lLLVM-3.8svn
./embedding

[tkelman]

I think the documentation and example makefiles are up-to-date now with #21299. Leave a comment or reopen if not.