Skip to content

Commit

Permalink
Dockerfile: upgrade to Ubuntu 24.04
Browse files Browse the repository at this point in the history 
Switch to Ubuntu 24.04, bump some
versions of dependencies, and use
some of the packages from Ubuntu
instead of building OpenCV manually.

There is also an ubuntu user/group
by default in the base image, so
use that instead of creating
a separate docker user like before.

Refs #3
  • Loading branch information
@pjonsson
pjonsson committed Dec 2, 2024
1 parent a6a3449 commit e863dfb
Showing 1 changed file with 16 additions and 36 deletions.
52 changes: 16 additions & 36 deletions Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# Fabian Lehmann
# Wilfried Weber

FROM ubuntu:20.04 AS builder
FROM ubuntu:24.04 AS builder

# disable interactive frontends
ENV DEBIAN_FRONTEND=noninteractive
Expand Down Expand Up @@ -54,29 +54,23 @@ apt-get -y install \
libglpk-dev \
libavcodec-dev \
libavformat-dev \
libopencv-dev \
libswscale-dev \
python3.8 \
python3.12 \
python3-pip \
python-is-python3 \
pandoc \
parallel \
libudunits2-dev \
r-base \
aria2 && \
dpkg -r parallel && \
wget http://de.archive.ubuntu.com/ubuntu/pool/universe/p/parallel/parallel_20210822+ds-2_all.deb && \
dpkg -i parallel_20210822+ds-2_all.deb && \
rm parallel_20210822+ds-2_all.deb && \
# Set python aliases for Python 3.x
echo 'alias python=python3' >> ~/.bashrc \
&& echo 'alias pip=pip3' >> ~/.bashrc \
&& . ~/.bashrc && \
#
# NumPy is needed for OpenCV, gsutil for level1-csd, landsatlinks for level1-landsat (requires gdal/requests/tqdm)
pip3 install --no-cache-dir --upgrade pip && \
pip3 install --no-cache-dir \
numpy==1.18.1 \
pip3 install --break-system-packages --no-cache-dir \
numpy==1.26.4 \
gsutil \
scipy==1.6.0 \
gdal==$(gdal-config --version | awk -F'[.]' '{print $1"."$2}') \
scipy==1.14.1 \
gdal==$(gdal-config --version) \
git+https://github.com/ernstste/landsatlinks.git && \
#
# Install R packages
Expand All @@ -91,22 +85,12 @@ apt-get clean && rm -r /var/cache/ /root/.cache /tmp/Rtmp*

# Install folder
ENV INSTALL_DIR=/opt/install/src \
HOME=/home/docker \
PATH="$PATH:/home/docker/bin"
HOME=/home/ubuntu \
PATH="$PATH:/home/ubuntu/bin"

# Build OpenCV from source
RUN mkdir -p $INSTALL_DIR/opencv && cd $INSTALL_DIR/opencv && \
wget https://github.com/opencv/opencv/archive/4.1.0.zip \
&& unzip 4.1.0.zip && \
mkdir -p $INSTALL_DIR/opencv/opencv-4.1.0/build && \
cd $INSTALL_DIR/opencv/opencv-4.1.0/build && \
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/local .. \
&& make -j7 \
&& make install \
&& make clean && \
#
# Build SPLITS from source
cd $INSTALL_DIR && \
RUN mkdir -p $INSTALL_DIR && cd $INSTALL_DIR && \
git clone https://bitbucket.org/smader/splits.git \
&& cd splits && \
libtoolize --force \
Expand All @@ -123,14 +107,10 @@ libtoolize --force \
rm -rf $INSTALL_DIR
#RUN apt-get purge -y --auto-remove apt-utils cmake git build-essential software-properties-common

RUN chgrp ubuntu /usr/local/bin && \
install -d -o ubuntu -g ubuntu -m 755 /home/ubuntu/bin

# Create a dedicated 'docker' group and user
RUN groupadd docker && \
useradd -m docker -g docker -p docker && \
chmod 0777 /home/docker && \
chgrp docker /usr/local/bin && \
mkdir -p /home/docker/bin && chown docker /home/docker/bin
# Use this user by default
USER docker
USER ubuntu

WORKDIR /home/docker
WORKDIR /home/ubuntu

7 comments on commit e863dfb

@kelewinska
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using an ubuntu user and user group will lead to the need to overhaul existing setups.
For example, on our server users have docker privileges only when they belong to a docker group. In order to run force in a container from this image, a new image group will need to be created, assuming ubuntu group does not exist already, which could lead to potential conflicts.
My suggestion is to keep the user and user group as docker. It would allow for a seamless transition.

@pjonsson
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docker user/group in the Ubuntu 20.04 image had uid/gid 1000. It will have uid/gid 1001 in Ubuntu 24.04, since the ubuntu user/group already has uid/gid 1000.

Most Linux users have a single-user system, so their local user has uid/gid 1000, and they can write files from inside docker into their local home directory seamlessly. If the user inside the container suddenly has uid/gid 1001 it's going to cause permission problems. (Users who don't have uid/gid 1000 has that problem already, but those usrs are much more rare.) Docker Desktop on MacOS has some kind of built-in support for mapping uid/gid between the host and container, so that doesn't show this problem.

Gitlab runs docker-in-docker because CI jobs frequently build docker images. I can't see why you would run docker inside a Force image though, can you describe the use-case you have for running docker inside docker?

@kelewinska
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i do not use a docker inside the docker.
I built the base image using the spec above and then try to build force with this updated image: https://github.com/davidfrantz/force/pull/336/files#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557.
It crushes on building FORCE. I had a bit more luck when I changed user and user group to docker. Still, I was not able to run force in the container. We do not spin force containers as root users.

@davidfrantz
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, on our server users have docker privileges only when they belong to a docker group.

This is normal. Otherwise you are not allowed to run Docker. The group within the container is unrelated to this. It just serves the purpose that the container cannot be called as root.

@kelewinska
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the image built based on this Dockerfile is missing libgdal.* in /lib/. I see the libgdal-dev is listed in line 35, but only the following libraries are available in /lib/

drwxr-xr-x 8 root root 4096 Feb 13 15:41 R
drwxr-xr-x 2 root root 4096 Apr 8 2024 X11
drwxr-xr-x 1 root root 4096 Jan 27 02:03 apt
drwxr-xr-x 2 root root 4096 Feb 13 15:40 bfd-plugins
drwxr-xr-x 2 root root 4096 Feb 13 15:39 binfmt.d
drwxr-xr-x 2 root root 4096 Feb 13 15:40 compat-ld
lrwxrwxrwx 1 root root 21 Jan 31 2024 cpp -> /etc/alternatives/cpp
drwxr-xr-x 2 root root 4096 Feb 13 15:39 dbus-1.0
drwxr-xr-x 3 root root 4096 Jan 27 02:03 dpkg
drwxr-xr-x 2 root root 4096 Feb 13 15:39 environment.d
drwxr-xr-x 2 root root 4096 Feb 13 15:40 file
drwxr-xr-x 2 root root 4096 Feb 13 15:41 fltk
drwxr-xr-x 3 root root 4096 Feb 13 15:40 gcc
drwxr-xr-x 2 root root 4096 Feb 13 15:39 girepository-1.0
drwxr-xr-x 3 root root 12288 Feb 13 15:40 git-core
drwxr-xr-x 2 root root 4096 Feb 13 15:40 gnupg
drwxr-xr-x 2 root root 4096 Feb 13 15:39 gnupg2
drwxr-xr-x 2 root root 4096 Feb 13 15:40 gold-ld
drwxr-xr-x 3 root root 4096 Feb 13 15:40 groff
drwxr-xr-x 2 root root 4096 Jan 27 02:09 init
drwxr-xr-x 2 root root 4096 Feb 13 15:41 jni
drwxr-xr-x 3 root root 4096 Feb 13 15:39 kernel
lrwxrwxrwx 1 root root 13 Mar 31 2024 libR.so -> R/lib/libR.so
lrwxrwxrwx 1 root root 18 Mar 31 2024 libarmadillo.so -> libarmadillo.so.12
lrwxrwxrwx 1 root root 22 Mar 31 2024 libarmadillo.so.12 -> libarmadillo.so.12.6.7
-rw-r--r-- 1 root root 73728 Mar 31 2024 libarmadillo.so.12.6.7
drwxr-xr-x 2 root root 4096 Feb 13 15:41 libpsm1
drwxr-xr-x 1 root root 4096 Mar 30 2024 locale
drwxr-xr-x 1 root root 4096 Jan 27 02:09 lsb
drwxr-xr-x 2 root root 4096 Feb 13 15:40 man-db
drwxr-xr-x 1 root root 4096 Feb 4 2024 mime
drwxr-xr-x 2 root root 4096 Feb 13 15:39 modprobe.d
drwxr-xr-x 8 root root 4096 Feb 13 15:39 networkd-dispatcher
drwxr-xr-x 2 root root 4096 Feb 13 15:40 openssh
-rw-r--r-- 1 root root 400 Feb 5 16:08 os-release
drwxr-xr-x 2 root root 4096 Feb 13 15:39 pam.d
drwxr-xr-x 5 root root 4096 Feb 13 15:39 pcrlock.d
drwxr-xr-x 2 root root 4096 Feb 13 15:41 pkgconfig
drwxr-xr-x 2 root root 4096 Feb 13 15:39 policykit-1
drwxr-xr-x 2 root root 4096 Feb 13 15:39 polkit-1
drwxr-xr-x 3 root root 4096 Feb 13 15:39 python3
drwxr-xr-x 33 root root 12288 Feb 13 15:41 python3.12
drwxr-xr-x 2 root root 4096 Aug 9 2024 sasl2
drwxr-xr-x 2 root root 4096 Feb 13 15:39 software-properties
drwxr-xr-x 3 root root 4096 Feb 13 15:39 ssl
drwxr-xr-x 1 root root 4096 Feb 13 15:39 sysctl.d
drwxr-xr-x 2 root root 4096 Feb 13 15:40 sysstat
drwxr-xr-x 1 root root 4096 Feb 13 15:39 systemd
drwxr-xr-x 2 root root 4096 Feb 13 15:39 sysusers.d
drwxr-xr-x 3 root root 4096 Feb 13 15:41 tcltk
drwxr-xr-x 1 root root 4096 Feb 13 15:40 tmpfiles.d
drwxr-xr-x 1 root root 4096 Feb 13 15:41 udev
drwxr-xr-x 2 root root 4096 Feb 13 15:41 valgrind
drwxr-xr-x 1 root root 86016 Feb 13 15:41 x86_64-linux-gnu

the base image based on ubuntu20.04 has the following libs:

drwxr-xr-x 8 root root 4096 Nov 19 11:58 R
drwxr-xr-x 2 root root 4096 Feb 8 2020 X11
drwxr-xr-x 1 root root 4096 Oct 11 02:09 apt
drwxr-xr-x 2 root root 4096 Nov 19 11:58 bfd-plugins
drwxr-xr-x 2 root root 4096 Jun 17 2024 binfmt.d
drwxr-xr-x 2 root root 4096 Nov 19 11:58 compat-ld
lrwxrwxrwx 1 root root 21 Nov 19 11:58 cpp -> /etc/alternatives/cpp
drwxr-xr-x 2 root root 4096 Nov 19 11:57 dbus-1.0
drwxr-xr-x 3 root root 4096 Oct 11 02:03 dpkg
drwxr-xr-x 3 root root 4096 Nov 19 11:58 emacsen-common
drwxr-xr-x 2 root root 4096 Nov 19 11:56 environment.d
drwxr-xr-x 2 root root 4096 Nov 19 11:57 file
drwxr-xr-x 2 root root 4096 Nov 19 11:58 fltk
drwxr-xr-x 3 root root 4096 Nov 19 11:58 gcc
drwxr-xr-x 2 root root 4096 Nov 19 11:58 gettext
drwxr-xr-x 2 root root 4096 Nov 19 11:57 girepository-1.0
drwxr-xr-x 3 root root 4096 Nov 19 11:58 git-core
drwxr-xr-x 2 root root 4096 Nov 19 11:57 gnupg
drwxr-xr-x 2 root root 4096 Nov 19 11:57 gnupg2
drwxr-xr-x 2 root root 4096 Nov 19 11:58 gold-ld
drwxr-xr-x 4 root root 4096 Nov 19 11:57 groff
drwxr-xr-x 2 root root 4096 Oct 11 02:08 init
drwxr-xr-x 3 root root 4096 Nov 19 11:56 kernel
lrwxrwxrwx 1 root root 13 Nov 3 14:59 libR.so -> R/lib/libR.so
lrwxrwxrwx 1 root root 17 Mar 23 2020 libarmadillo.so -> libarmadillo.so.9
lrwxrwxrwx 1 root root 23 Mar 23 2020 libarmadillo.so.9 -> libarmadillo.so.9.800.4
-rw-r--r-- 1 root root 73616 Mar 23 2020 libarmadillo.so.9.800.4
-rw-r--r-- 1 root root 1120672 Apr 10 2020 libdfalt.a
-rw-r--r-- 1 root root 933 Apr 10 2020 libdfalt.la
lrwxrwxrwx 1 root root 17 Apr 10 2020 libdfalt.so -> libdfalt.so.0.0.0
lrwxrwxrwx 1 root root 17 Apr 10 2020 libdfalt.so.0 -> libdfalt.so.0.0.0
-rw-r--r-- 1 root root 503888 Apr 10 2020 libdfalt.so.0.0.0
-rw-r--r-- 1 root root 50018498 Mar 29 2020 libgdal.a
lrwxrwxrwx 1 root root 17 Mar 29 2020 libgdal.so -> libgdal.so.26.0.4
lrwxrwxrwx 1 root root 17 Mar 29 2020 libgdal.so.26 -> libgdal.so.26.0.4
-rw-r--r-- 1 root root 20197488 Mar 29 2020 libgdal.so.26.0.4
-rw-r--r-- 1 root root 1133 Apr 10 2020 libhdf4.settings
-rw-r--r-- 1 root root 274934 Apr 10 2020 libmfhdfalt.a
-rw-r--r-- 1 root root 954 Apr 10 2020 libmfhdfalt.la
lrwxrwxrwx 1 root root 20 Apr 10 2020 libmfhdfalt.so -> libmfhdfalt.so.0.0.0
lrwxrwxrwx 1 root root 20 Apr 10 2020 libmfhdfalt.so.0 -> libmfhdfalt.so.0.0.0
-rw-r--r-- 1 root root 161888 Apr 10 2020 libmfhdfalt.so.0.0.0
lrwxrwxrwx 1 root root 14 Sep 5 2019 libogdi.so -> libogdi.so.4.1
lrwxrwxrwx 1 root root 14 Sep 5 2019 libogdi.so.4 -> libogdi.so.4.1
-rw-r--r-- 1 root root 109000 Sep 5 2019 libogdi.so.4.1
lrwxrwxrwx 1 root root 13 Sep 5 2019 libvpf.so -> libvpf.so.4.1
lrwxrwxrwx 1 root root 13 Sep 5 2019 libvpf.so.4 -> libvpf.so.4.1
-rw-r--r-- 1 root root 293296 Sep 5 2019 libvpf.so.4.1
drwxr-xr-x 3 root root 4096 Apr 14 2020 locale
drwxr-xr-x 1 root root 4096 Oct 11 02:08 lsb
drwxr-xr-x 2 root root 4096 Nov 19 11:57 man-db
drwxr-xr-x 1 root root 4096 Nov 19 11:56 mime
drwxr-xr-x 2 root root 4096 Nov 19 11:56 modprobe.d
drwxr-xr-x 2 root root 4096 Jun 17 2024 modules-load.d
drwxr-xr-x 8 root root 4096 Nov 19 11:57 networkd-dispatcher
drwxr-xr-x 2 root root 4096 Nov 19 11:58 ogdi
drwxr-xr-x 2 root root 4096 Nov 19 11:57 openssh
-rw-r--r-- 1 root root 382 Jan 2 2024 os-release
drwxr-xr-x 2 root root 4096 Nov 19 11:57 packagekit
-rw-r--r-- 1 root root 17 Feb 7 2020 pkg-config.multiarch
drwxr-xr-x 2 root root 4096 Nov 19 11:58 pkgconfig
drwxr-xr-x 2 root root 4096 Nov 19 11:57 policykit-1
drwxr-xr-x 3 root root 4096 Nov 19 11:57 python2.7
drwxr-xr-x 3 root root 4096 Nov 19 11:56 python3
drwxr-xr-x 30 root root 4096 Nov 19 11:58 python3.8
drwxr-xr-x 4 root root 4096 Nov 19 11:58 python3.9
drwxr-xr-x 2 root root 4096 Feb 15 2022 sasl2
drwxr-xr-x 2 root root 4096 Nov 19 11:57 software-properties
drwxr-xr-x 3 root root 4096 Nov 19 11:57 ssl
drwxr-xr-x 1 root root 4096 Nov 19 11:56 sysctl.d
drwxr-xr-x 2 root root 4096 Nov 19 11:58 sysstat
drwxr-xr-x 1 root root 4096 Nov 19 11:57 systemd
drwxr-xr-x 2 root root 4096 Nov 19 11:57 sysusers.d
drwxr-xr-x 3 root root 4096 Nov 19 11:58 tcltk
drwxr-xr-x 16 root root 4096 Feb 26 2020 terminfo
drwxr-xr-x 1 root root 4096 Nov 19 11:57 tmpfiles.d
drwxr-xr-x 1 root root 4096 Oct 11 02:09 udev
drwxr-xr-x 2 root root 4096 Nov 19 11:56 valgrind
drwxr-xr-x 1 root root 36864 Nov 19 11:58 x86_64-linux-gnu

Not sure whether because of this or due to additional issues, I am not able to build force based on this image.

Do you have any thoughts on how to resolve it? Thanks in advance!

@pjonsson
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Packages suffixed with -dev contain things required for compilation, such as header files (*.h, etc.) and static libraries (*.a).

The x86-64 GDAL libraries reside under /usr/lib/x86_64-linux-gnu in the Ubuntu 24 package: https://packages.ubuntu.com/noble/amd64/libgdal34t64/filelist

If you want to build force on the Ubuntu 24.04-based image, I suggest you use the patches in davidfrantz/force#336. That PR builds, but it still needs to be tested, so use at your own risk.

@kelewinska
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dear @pjonsson. Thank you for coming back to me. I am doing exactly as you have suggested. I first build the base image as specified above (e863dfb), and then try to build the force image (3.8.0) from davidfrantz/force#336 on top of it. Sadly, the force fails to build. Therefore, I compared the libraries in the image base image based on the 20.04 version and the one above looking for the differences that could be a reason for the failing build.
Of course, you are correct, libgdal.* are in the /usr/lib/x86_64-linux-gnu.

I guess, I should move then my inquiry to the davidfrantz/force#336. I see the pull passes all the checks, yet for some reason, the force image does not build successfully (exit error 2) on my system.

Please sign in to comment.