compile libzim timeout

[hamkido]

There are still some problem when compiling libzim.
I have tested it on three machines.
VPS: debian9
Laptop: debian10
Desktop: archlinux

[kelson42]

@hamkido Please put all the details (environnement, commands, output) on one system, otherwise we can not help you.

[hamkido]

At first I thought it was my laptop processor was too slow.
Then I tested it on vps and the same test on the desktop.
I have read the timeout issue #354
and libzim compile issue #379
But didn't help.

[hamkido]

Sorry for my misspelling
detail output：
$ kiwix-build libzim
[INSTALL PACKAGES]

• automake : SKIP
• libtool : SKIP
• cmake : SKIP
• git : SKIP
• subversion : SKIP
• ccache : SKIP
• pkg-config : SKIP
• gcc : SKIP
• autopoint : SKIP
• libbz2-dev : SKIP
• libmagic-dev : SKIP
• zlib1g-dev : SKIP
• uuid-dev : SKIP
  SKIP, No package to install.
  [SETUP PLATFORMS]
  [PREPARE]
  prepare sources lzma :
  download lzma : SKIP
  extract lzma : SKIP
  prepare sources xapian-core :
  download xapian-core : SKIP
  extract xapian-core : SKIP
  patch xapian-core : SKIP
  prepare sources icu4c :
  download icu4c : SKIP
  extract icu4c : SKIP
  patch icu4c : SKIP
  prepare sources libzim :
  gitclone libzim : SKIP
  gitupdate libzim : OK (4.9s)
  [BUILD]
  build lzma (native_dyn):
  configure lzma : OK (19.5s)
  compile lzma : .OK (32.3s)
  install lzma : OK (5.0s)
  build xapian-core (native_dyn):
  configure xapian-core : OK (26.2s)
  compile xapian-core : ......OK (194.7s)
  install xapian-core : OK (2.6s)
  build icu4c (native_dyn):
  configure icu4c : OK (14.9s)
  compile icu4c : ...............OK (473.1s)
  install icu4c : OK (1.9s)
  build libzim (native_dyn):
  configure libzim : ....OK (136.2s)
  compile libzim : ..OK (88.6s)
  test libzim : .ERROR
  run command 'meson test --verbose -t 8'
  current directory is '/home/hamki/kiwix/BUILD_native_dyn/libzim'
  env is :
  SSH_AUTH_SOCK : '/tmp/ssh-YG2QgwQXLjLb/agent.911'
  SSH_AGENT_PID : '981'
  LANG : 'en_US.UTF-8'
  XDG_CURRENT_DESKTOP : 'i3'
  COLORTERM : 'truecolor'
  XDG_GREETER_DATA_DIR : '/var/lib/lightdm/data/hamki'
  GPG_AGENT_INFO : '/run/user/1000/gnupg/S.gpg-agent:0:1'
  QT_IM_MODULE : 'fcitx'
  USER : 'hamki'
  HOME : '/home/hamki'
  XDG_SESSION_PATH : '/org/freedesktop/DisplayManager/Session0'
  DESKTOP_SESSION : 'i3'
  DBUS_SESSION_BUS_ADDRESS : 'unix:path=/run/user/1000/bus'
  XDG_VTNR : '7'
  XDG_SEAT : 'seat0'
  I3SOCK : '/run/user/1000/i3/ipc-socket.911'
  XDG_DATA_DIRS : '/home/hamki/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share'
  GTK_IM_MODULE : 'fcitx'
  XDG_SESSION_DESKTOP : 'i3'
  QT_ACCESSIBILITY : '1'
  VTE_VERSION : '5802'
  XDG_SEAT_PATH : '/org/freedesktop/DisplayManager/Seat0'
  LOGNAME : 'hamki'
  PATH : '/home/hamki/kiwix/BUILD_native_dyn/INSTALL/bin:/usr/lib/ccache:/home/hamki/kiwix/bin:/home/hamki/.local/bin:/home/hamki/.npm-global/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/hamki/go/bin'
  XDG_RUNTIME_DIR : '/run/user/1000'
  XMODIFIERS : '@im=fcitx'
  XDG_SESSION_ID : '2'
  XDG_SESSION_TYPE : 'x11'
  WINDOWID : '60817411'
  SHELL : '/bin/zsh'
  _JAVA_OPTIONS : '-Dawt.useSystemAAFontSettings=on -Dswing.aatext=true'
  GDM_LANG : 'en_US.utf8'
  QT_QPA_PLATFORMTHEME : 'qt5ct'
  XAUTHORITY : '/home/hamki/.Xauthority'
  PWD : '/home/hamki/kiwix'
  XDG_SESSION_CLASS : 'user'
  DISPLAY : ':0'
  GDMSESSION : 'i3'
  TERM : 'xterm-256color'
  SHLVL : '1'
  OLDPWD : '/home/hamki'
  ZSH : '/home/hamki/.oh-my-zsh'
  PAGER : 'less'
  LESS : '-R'
  LSCOLORS : 'Gxfxcxdxbxegedabagacad'
  LS_COLORS : 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.zst=01;31:.tzst=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.wim=01;31:.swm=01;31:.dwm=01;31:.esd=01;31:.jpg=01;35:.jpeg=01;35:.mjpg=01;35:.mjpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:*.xspf=00;36:'
  GOPATH : '/home/hamki/go'
  EDITOR : 'emacsclient -t'
  VIRTUAL_ENV : '/home/hamki/kiwix'
  PS1 : '(kiwix) ╭─$(virtualenv_prompt_info)%{$terminfo[bold]$fg[green]%}%n@%m %{$reset_color%}%{$terminfo[bold]$fg[blue]%}%~ %{$reset_color%}$(ruby_prompt_info)$(git_prompt_info)\n╰─%B$%b '
  _ : '/home/hamki/kiwix/bin/kiwix-build'
  PKG_CONFIG_PATH : ':/home/hamki/kiwix/BUILD_native_dyn/INSTALL/lib/x86_64-linux-gnu/pkgconfig'
  LD_LIBRARY_PATH : ':/home/hamki/kiwix/BUILD_native_dyn/INSTALL/lib:/home/hamki/kiwix/BUILD_native_dyn/INSTALL/lib/x86_64-linux-gnu'
  QMAKE_CXXFLAGS : '-I/home/hamki/kiwix/BUILD_native_dyn/INSTALL/include '
  CPPFLAGS : '-I/home/hamki/kiwix/BUILD_native_dyn/INSTALL/include '
  QMAKE_LFLAGS : '-L/home/hamki/kiwix/BUILD_native_dyn/INSTALL/lib -L/home/hamki/kiwix/BUILD_native_dyn/INSTALL/lib/x86_64-linux-gnu '
  LDFLAGS : '-L/home/hamki/kiwix/BUILD_native_dyn/INSTALL/lib -L/home/hamki/kiwix/BUILD_native_dyn/INSTALL/lib/x86_64-linux-gnu '
  ninja: Entering directory `/home/hamki/kiwix/BUILD_native_dyn/libzim'
  ninja: no work to do.
  [==========] Running 7 tests from 2 test cases.
  [----------] Global test environment set-up.
  [----------] 5 tests from ClusterTest
  [ RUN ] ClusterTest.create_cluster
  [ OK ] ClusterTest.create_cluster (0 ms)
  [ RUN ] ClusterTest.read_write_cluster
  [ OK ] ClusterTest.read_write_cluster (0 ms)
  [ RUN ] ClusterTest.read_write_empty
  [ OK ] ClusterTest.read_write_empty (0 ms)
  [ RUN ] ClusterTest.read_write_clusterZ
  [ OK ] ClusterTest.read_write_clusterZ (2 ms)
  [ RUN ] ClusterTest.read_write_clusterLzma
  [==========] Running 8 tests from 1 test case.
  [----------] Global test environment set-up.
  [----------] 8 tests from DirentTest
  [ RUN ] DirentTest.set_get_data_dirent
  [ OK ] DirentTest.set_get_data_dirent (0 ms)
  [ RUN ] DirentTest.read_write_article_dirent
  [ OK ] DirentTest.read_write_article_dirent (0 ms)
  [ RUN ] DirentTest.read_write_article_dirent_unicode
  [ OK ] DirentTest.read_write_article_dirent_unicode (0 ms)
  [ RUN ] DirentTest.read_write_redirect_dirent
  [ OK ] DirentTest.read_write_redirect_dirent (0 ms)
  [ RUN ] DirentTest.read_write_linktarget_dirent
  [ OK ] DirentTest.read_write_linktarget_dirent (0 ms)
  [ RUN ] DirentTest.read_write_deleted_dirent
  [ OK ] DirentTest.read_write_deleted_dirent (0 ms)
  [ RUN ] DirentTest.dirent_size
  [ OK ] DirentTest.dirent_size (0 ms)
  [ RUN ] DirentTest.redirect_dirent_size
  [ OK ] DirentTest.redirect_dirent_size (0 ms)
  [----------] 8 tests from DirentTest (0 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 1 test case ran. (1 ms total)
[ PASSED ] 8 tests.
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from HeaderTest
[ RUN ] HeaderTest.read_write_header
[ OK ] HeaderTest.read_write_header (0 ms)
[----------] 1 test from HeaderTest (0 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (0 ms total)
[ PASSED ] 1 test.
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from UuidTest
[ RUN ] UuidTest.construct
[ OK ] UuidTest.construct (0 ms)
[ RUN ] UuidTest.generate
[ OK ] ClusterTest.read_write_clusterLzma (849 ms)
[----------] 5 tests from ClusterTest (851 ms total)

[----------] 2 tests from CluterTest
[ RUN ] CluterTest.read_write_extended_cluster
[ OK ] UuidTest.generate (1000 ms)
[ RUN ] UuidTest.output
[ OK ] UuidTest.output (0 ms)
[----------] 3 tests from UuidTest (1000 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran. (1000 ms total)
[ PASSED ] 3 tests.
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from TemplateTest
[ RUN ] TemplateTest.ZeroTemplate
[ OK ] TemplateTest.ZeroTemplate (0 ms)
[ RUN ] TemplateTest.Token
[ OK ] TemplateTest.Token (0 ms)
[ RUN ] TemplateTest.Link
[ OK ] TemplateTest.Link (0 ms)
[----------] 3 tests from TemplateTest (0 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran. (7 ms total)
[ PASSED ] 3 tests.
1/5 cluster FAIL 194.95 s (killed by signal 9 SIGKILL)
2/5 dirent OK 0.06 s
3/5 header OK 0.03 s
4/5 uuid OK 1.03 s
5/5 template OK 0.04 s

Ok: 4
Expected Fail: 0
Fail: 1
Unexpected Pass: 0
Skipped: 0
Timeout: 0

Stopping build due to errors

[kelson42]

@mgautierfr ?

[hamkido]

As a result, Kiwix-desktop that depend on libzim cannot compile.
Although I can use flatpak or appimage,
But this does not quite agree with the K.I.S.S principle.
This problem has troubled me for a long time.
If you need test or output results, I will provide.

[kelson42]

@hamkido Have you tried this? openzim/libzim#147 (comment)

[hamkido]

@kelson42 Thanks, this fixed the issue. You are a hacker.
The cause of the problem is that all my machine memory is only 15.4G.
I think the readme.md file should add hardware requirements. Because as long as someone compiles libzim or other projects that require libzim, even using export SKIP_BIG_MEMORY_TEST = 1 requires more than 12G of memory.
Thank you very much.

[ynikitenko]

I had a similar problem and it took me a while to figure that out. It doesn't seem that readme.md was fixed since then, and I think it was wrong that @hamkido closed this issue.

[kelson42]

I have updated the documentation of libzim:
openzim/libzim@2e98509