Timeout while compiling kiwix-build on my system

[mgautierfr]

[This issue has been reported at #346 by @Aditya-Sood, but as the thread has move to another discussion, I prefer to move the "timeout" problem in another issue]

I'm working on fixing kiwix/kiwix-android/#765 for which I had to compile kiwix-lib into the app, which I was told required using kiwix-build

During setup, upon running kiwix-build kiwix-lib -v, I obtained the following output:

.
.
.
[==========] Running 8 tests from 1 test case.
[----------] Global test environment set-up.
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[==========] Running 1 test from 1 test case.
[==========] Running 7 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 8 tests from DirentTest
[----------] 3 tests from UuidTest
[----------] Global test environment set-up.
[----------] 5 tests from ClusterTest
[ RUN      ] DirentTest.set_get_data_dirent
[ RUN      ] UuidTest.construct
[----------] 1 test from HeaderTest
[ RUN      ] ClusterTest.create_cluster
[       OK ] DirentTest.set_get_data_dirent (0 ms)
[       OK ] UuidTest.construct (0 ms)
[ RUN      ] HeaderTest.read_write_header
[       OK ] ClusterTest.create_cluster (0 ms)
[ RUN      ] DirentTest.read_write_article_dirent
[ RUN      ] UuidTest.generate
[       OK ] HeaderTest.read_write_header (0 ms)
[ RUN      ] ClusterTest.read_write_cluster
[       OK ] DirentTest.read_write_article_dirent (0 ms)
[----------] 1 test from HeaderTest (1 ms total)

[       OK ] ClusterTest.read_write_cluster (0 ms)
[ RUN      ] DirentTest.read_write_article_dirent_unicode
[----------] Global test environment tear-down
[ RUN      ] ClusterTest.read_write_empty
[       OK ] DirentTest.read_write_article_dirent_unicode (0 ms)
[==========] 1 test from 1 test case ran. (2 ms total)
[  PASSED  ] 1 test.
[       OK ] ClusterTest.read_write_empty (0 ms)
[ RUN      ] DirentTest.read_write_redirect_dirent
[ RUN      ] ClusterTest.read_write_clusterZ
[       OK ] DirentTest.read_write_redirect_dirent (0 ms)
[ RUN      ] DirentTest.read_write_linktarget_dirent
[       OK ] ClusterTest.read_write_clusterZ (2 ms)
[       OK ] DirentTest.read_write_linktarget_dirent (0 ms)
[ RUN      ] ClusterTest.read_write_clusterLzma
[ RUN      ] DirentTest.read_write_deleted_dirent
[       OK ] DirentTest.read_write_deleted_dirent (0 ms)
[ RUN      ] DirentTest.dirent_size
[       OK ] DirentTest.dirent_size (0 ms)
[ RUN      ] DirentTest.redirect_dirent_size
[       OK ] ClusterTest.read_write_clusterLzma (37 ms)
[       OK ] DirentTest.redirect_dirent_size (0 ms)
[----------] 5 tests from ClusterTest (91 ms total)

[----------] 8 tests from DirentTest (91 ms total)

[----------] 2 tests from CluterTest
[----------] Global test environment tear-down
[ RUN      ] CluterTest.read_write_extended_cluster
[==========] 8 tests from 1 test case ran. (93 ms total)
[  PASSED  ] 8 tests.
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from TemplateTest
[ RUN      ] TemplateTest.ZeroTemplate
[       OK ] TemplateTest.ZeroTemplate (0 ms)
[ RUN      ] TemplateTest.Token
[       OK ] TemplateTest.Token (0 ms)
[ RUN      ] TemplateTest.Link
[       OK ] TemplateTest.Link (0 ms)
[----------] 3 tests from TemplateTest (0 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran. (1 ms total)
[  PASSED  ] 3 tests.
[       OK ] UuidTest.generate (1000 ms)
[ RUN      ] UuidTest.output
[       OK ] UuidTest.output (0 ms)
[----------] 3 tests from UuidTest (1001 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran. (1002 ms total)
[  PASSED  ] 3 tests.
cluster time out (After 480 seconds)
1/5 cluster                                 TIMEOUT 488.94 s 
2/5 dirent                                  OK       0.74 s 
3/5 header                                  OK       0.72 s 
4/5 uuid                                    OK       1.65 s 
5/5 template                                OK       0.05 s 

Ok:                    4
Expected Fail:         0
Fail:                  1
Unexpected Pass:       0
Skipped:               0
Timeout:               1


Stopping build due to errors

The entire output of the command is here

There was a test libzim : run command 'meson test --verbose -t 8' line in the middle of that output, running which gave this error:

Only ninja backend is supported to rebuild tests before running them.
Meson test encountered an error:

Directory $'.../Kiwix/kiwix-build' does not seem to be a Meson build directory.

I'm guessing this is related to the timeout bit in #188

I am basically looking for a way to test the changes I have made to kiwix-lib (commit) & kiwix-android (commit)

Any help is much appreciated!

[mgautierfr]

@Aditya-Sood, what is your system ?

[kelson42]

@Aditya-Sood Without feedback from you on this ticket we won't be able to help you and I will have to close the ticket.

[Aditya-Sood]

@mgautierfr sorry for not responding here! I didn't know you had created a separate issue for this, I've been replying on the previous one for the last couple of days

@kelson42 I have wiped the previous local kiwix-build repo and started over. Things are okay as of now, the kiwix-build kiwix-lib-app ... has been downloading stuff and running for about 2 hrs now

I'll update by evening of the progress

[Aditya-Sood]

@mgautierfr do I need to build kiwix-lib before I put it in the SOURCE directory of kiwix-build for building?
I haven't done that yet, which might be the reason why I get this cryptic error when running kiwix-build kiwix-lib-app --target-platform android --verbose:

FAILED: src/25a6634@@kiwix@sha/searcher.cpp.o 
/mnt/1800D5F100D5D5BA/Mamba/Kiwix/Build/kiwix-build/BUILD_android_arm/android-ndk-r13b/bin/arm-linux-androideabi-g++ -Isrc/25a6634@@kiwix@sha -Isrc -I../../SOURCE/kiwix-lib/src -Iinclude -I../../SOURCE/kiwix-lib/include -Istatic -Isrc/android -I/mnt/1800D5F100D5D5BA/Mamba/Kiwix/Build/kiwix-build/BUILD_android_arm/INSTALL/include -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -Werror -std=c++11 -O2 -g -fPIC -DCURL_STATICLIB -pthread -I/mnt/1800D5F100D5D5BA/Mamba/Kiwix/Build/kiwix-build/BUILD_android_arm/INSTALL/include -march=armv7-a -MD -MQ 'src/25a6634@@kiwix@sha/searcher.cpp.o' -MF 'src/25a6634@@kiwix@sha/searcher.cpp.o.d' -o 'src/25a6634@@kiwix@sha/searcher.cpp.o' -c ../../SOURCE/kiwix-lib/src/searcher.cpp
../../SOURCE/kiwix-lib/src/searcher.cpp: In member function 'std::string kiwix::Searcher::getHtml()':
../../SOURCE/kiwix-lib/src/searcher.cpp:363:30: error: 'search_result_tmpl' is not a member of 'RESOURCE'
   std::string template_str = RESOURCE::search_result_tmpl;
                              ^
ninja: build stopped: subcommand failed.
ERROR
Stopping build due to errors

[mhutti1]

@Aditya-Sood Delete SOURCE and all BUILD_ directories and try again?

[Aditya-Sood]

If I delete SOURCE then won't it use the kiwix-lib it clones from github?
Because the build completed without my copy of kiwix-lib in SOURCE, this error happened only after I copied my local repo into it

[mhutti1]

Ok what is the difference between the two?

[Aditya-Sood]

Basically my local repo has the changes for issue 765 that I want to test

[mhutti1]

Well then your changes must have cause it?... Can you debug what might have happened?

[Aditya-Sood]

@mhutti1 so I deleted all the BUILD_ folders, copied my local kiwix-lib repo to SOURCE and started compilation again. This time I'm getting a

FileNotFoundError: [Errno 2] No such file or directory: '.../kiwix-build/SOURCE/kiwix-lib/android-kiwix-lib-publisher'

I was getting this error last time as well before I started over with a fresh clone. So I guess I need to build the local kiwix-lib repo (with changes for issue 765) before copying it into kiwix-build's SOURCE directory?

[mhutti1]

No... There should not be any build files in SOURCE, it is literally just source code. I would imagine that that error is because you need to rebase your changes ontop of kiwix-lib master. Once you do this then the file it is complaining about will exist.

[Aditya-Sood]

So just git rebase master and then copy the local repo into SOURCE?

[mhutti1]

Yes

…

On Sun, 18 Aug 2019, 10:44 Aditya-Sood, ***@***.***> wrote: So just git rebase master and then copy the local repo into SOURCE? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#354?email_source=notifications&email_token=AAROKWMQCH5F7LL2QIMMVSDQFEKWPA5CNFSM4IH2D2YKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4Q4KQA#issuecomment-522306880>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAROKWMBMEHYE27UQZRWCDTQFEKWPANCNFSM4IH2D2YA> .

[Aditya-Sood]

The build completed successfully after the rebase
Regarding ln kiwix-build/BUILD_android/kiwix-lib-app/kiwixLibAndroid/build/outputs/aar kiwix-android/app/libs, I'm getting a 'hard link not allowed' error
Do I basically copy the aar folder into kiwix-android/app/ and rename it to libs?

[Aditya-Sood]

I'm having trouble getting the JNIKiwixReader updated. Even after commenting out the previous implementations that I could find, the reader somehow still uses the previous implementation. (I've kept the kiwix-lib .aar file in app/libs)
What can I do to change this? Do I delete the kiwix-android/kiwixlib to force this?

EDIT: I merged develop again and manually deleted the kiwix-android/kiwixlib directory
I'm now getting a Unsupported Modules Detected: Compilation is not supported for following modules: kiwixlib. Unfortunately you can't have non-Gradle Java modules and Android-Gradle modules in one project. error

[mhutti1]

@Aditya-Sood You may need to restart and invalidate your cache. Other than that just check your build.gradle. The kiwixlib dependency is there pretty clearly so it should be trivial to see what it is trying to load.

[Aditya-Sood]

Invalidating caches didn't help earlier
I'll check out the Gradle file

EDIT: Importing the project again removed the error but is still not updating the JNIKiwixReader

[mhutti1]

@Aditya-Sood Can you share your build.gradle?

[Aditya-Sood]

build.gradle - I haven't made any changes to it though

[mhutti1]

What if you comment out https://github.com/Aditya-Sood/kiwix-android/blob/b1530a2777d8a67abf400bb5e6672a85fc55635f/app/build.gradle#L56 ?

[Aditya-Sood]

I'm getting a Cannot resolve symbol 'kiwixlib' error then
So file("./libs").exists() is returning false for some reason, even though the directory exists. I also tried file("./libs/") and file("./libs/kiwixLibAndroid-release.aar"), but those didn't work either.

As a side note: Could you have a look at my kiwix-lib .aar file once? There are a couple of empty directories in it, which doesn't make sense to me.

[mhutti1]

Is this file("./libs").exists() a line in your gradle or have you added it? I am not really familiar with aars so I can't comment too much.

[Aditya-Sood]

It's used in the shouldUseLocalVersion() function in build.gradle

[mhutti1]

Can you give it an absolute path and try that?

[Aditya-Sood]

I figured out the problem with the .aar - basically kiwix-build uses the master branch of the kiwix-lib source code when building. Merging my changes into the master branch did the trick, the updated method is shown in JNIKiwixReader now.
(Seems logical in hindsight, but I think I'll create a PR to the readme all the same about merging changes to master for local testing)

I'll get to testing now

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

@Aditya-Sood Do you still have this issue?

[Aditya-Sood]

@kelson42 it's been fixed, I'd made a mistake somewhere during setup. Repeating the setup afresh had fixed it.
Issue can be closed
Thank you @mgautierfr & @mhutti1 for helping me out