Create a Docker image for netpkg-tool

[philippgille]

Some people might want to use the tool, but don't want to install all the requirements (.NET Core SDK, FUSE for appimagetool). Docker containers are perfect for that - they contain all dependencies and after usage they can be thrown away.

Usage would be something like this (on a machine with only git and Docker installed):

1. git clone https://github.com/phil-harmoniq/aspnet-src
2. docker run --rm -v ${PWD}/aspnet-src:/root/src -v ${PWD}/aspnet-out:/root/out phil-harmoniq/netpkg-tool netpkg-tool /root/src /root/out -n aspnet-pkg
3. ./aspnet-out/aspnet-pkg

The Docker image should probably be based on the official microsoft/dotnet Docker image (tag 2.0-sdk), but normal appimagetool usage depends on FUSE, which doesn't work in Docker containers, see this AppImage wiki page and the AppImage wiki page I wrote regarding packaging .NET Core apps. So the following is probably required:

1. netpkg-tool needs to check if it's inside a container. If not: use appimagetool as is, otherwise unpack it and use the binary inside its AppDir
2. libglib2.0-0 needs to be installed in the Docker image so unpacking the appimagetool works

I've done something similar for philippgille/hello-netcoreapp, so I know how to do this and I would like to create a PR if you think this would be a nice feature.

[phil-harmoniq]

This sounds like a good idea! I'm not very familiar with Docker, so a PR would be great.

[phil-harmoniq]

Pull Request #50 was merged into the development branch. I'm going to keep this issue open until the changes are merged into the master branch.

Docker is pretty cool, thanks for the PR! I'm going to be working on getting this and a couple of other small changes into the next version over the next couple of days. I'm sure I'll have more questions as I dig deeper into Docker, but one thing I noticed is that you copied netpkg-tool.csproj into the Docker container by itself and restored before copying the rest of the source folder. Is there a specific purpose for this? Does it have something to do with Docker layering?

[phil-harmoniq]

I finally merged the changes into the master branch and automated Docker images are now available @philippgille. 😄 However, I can't seem to figure out why Docker Hub is using the main README.md instead of the one inside of the docker directory.

[philippgille]

Thanks for merging @phil-harmoniq!

Couple of things:

• Yes, copying netpkg-tool.csproj and restoring the packages on its own, before copying the rest of the app, is for utilizing Docker's layer caching mechanisms.
  • Each COPY instruction in the Dockerfile leads to a new layer. COPY also checks if the contents of what's being copied changed. If not, then a previously cached layer is used.
  • RUN doesn't check any file contents - instead when building a Docker image and there's a RUN command, only the command itself is checked for change. But RUN also leads to a cached layer.
  • So, for example: You build a Docker image for the first time. The .csproj file is copied and the dotnet restore has to be executed, because the layer cache is empty. After that, a layer exists and is in the cache. Now you rarely change the .csproj file, but instead you often change Program.cs. But as long as .csproj doesn't change, dotnet restore doesn't have to be executed anymore when building a Docker image, because Docker can re-use the layers of the COPY .csproj and RUN dotnet restore commands.
  • Try it: cd into your repository, run docker build -f docker/Dockerfile -t local/netpkg-tool .. Run it again and see that nothing gets rebuilt, instead only cached layers are used. Then change the .csproj and then run the docker build ... command again. You see Step 5/13 is running dotnet restore. Step 7/13 also gets executed again. Execute the dotnet build ... command again and see that only cached layers get used. Now change Program.cs and then rebuild the Docker image. Step 5/13 is now using a cached layer, because there's nothing new to restore. Instead, only 7/13 gets executed again.
• Your Docker Hub repo isn't configured as automated build. This means you need to manually push built images to the Docker Hub. Travis CI can automatically build the image and push it to Docker Hub, but some niceties are still missing then. Docker Hub repositories with automated builds are marked as such and some users filter for automated builds when searching for dockerized apps. And only repos with automated builds have the Dockerfile, build history and link to the originating GitHub repository on their page! I think this might also be the reason why the Docker image description isn't the one from the docker/README.md file. There might still be more to do for the correct README to be shown, I head issues with that as well.
• When the docker/README.md shows up on the Docker Hub, you probably want it to look like most Docker image descriptions on Docker Hub look like. I'd suggest:
  • Remove the pull section. docker run implicitly pulls Docker images, just like dotnet run implicitly calls dotnet restore since .NET Core 2.0. Looking at popular Docker images, you won't find docker pull in any READMEs.
  • Add links to the Dockerfile. Pretty much all popular Docker image descriptions start with a section "Supported tags and respective Dockerfile links" - see some popular images here.
  • The Usage section has some commands that still use the local/netpkg-tool image, but that was only needed when the image wasn't on Docker Hub yet and you had to build it with docker build -f docker/Dockerfile -t local/netpkg-tool . (any tag would have worked, but local/something is a nice hint that the image was built locally and not pulled from any Docker registry). You should replace that now with philhamroniq/netpkg-tool.

[phil-harmoniq]

@philippgille, I've been trying to figure out what's going on but I'm completely stumped. So, I do have automated builds set up with Docker Cloud (they happen when pushes happen to either the develop or master branch), but Docker Hub doesn't show it as an automated build.

As a test, I tried to make a repo directly in Docker Hub (instead of Docker Cloud) because it was able to tag the repo as an automated build, but now the Dockerfile fails when attempting the first copy (netpkg-tool.csproj). I can't seem to find any helpful issues in the Docker Github issues page. Building works locally and through Docker Cloud, but neither of them seem to actually tag the repo as an automated build.

For reference, here is one of the logs with the copy error. Building locally works just fine, so I'm not sure what's going wrong.

[philippgille]

Ah I see. You forgot to add the log of the copy error, but I guess it's something like this: Build failed: COPY failed: stat /var/lib/docker/tmp/docker-builder423847332/src: no such file or directory?

When I set up a Docker Hub build for my project about 2 month ago and used multi-stage builds for the first time, I had the same issue. Back then I found a post on the Docker forums where a Docker employee said they would update Docker Hub to a newer Docker version in July, because they were using a Docker version older than 17.05, which introduced multi-stage builds.

That's why I used Docker Cloud as well. And instead of removing the automatic Docker Hub build, I just turned the automatic builds off, like this:

That way, the image still shows up as automated build in Docker Hub.

Because they said they're going to update in July, I thought they had done so and you could use Docker Hub for the multi-stage build. There's a Docker issue for this: docker/hub-feedback#1039

[phil-harmoniq]

Thanks for the readme pull request. Yea that was exactly the error. Thanks so much for your help! Automated builds are now working properly. The only thing left is Docker Hub still using the root README.md and not docker/README.md. I found this issue, but it's currently closed.

Edit: Unfortunately, it seems that build logs aren't showing up on Docker Hub using this method.

[philippgille]

Regarding the build logs: Sooner or later Docker Hub will be updated to a Docker version that supports multi-stage builds. When that happens you can switch from building the images on Docker Cloud to Docker Hub and everything should be fine.

Regarding the README: I had the issue myself and what helped me was manually pasting the new README in the description field on Docker Cloud once (link should be something like https://cloud.docker.com/swarm/philhamroniq/repository/docker/philhamroniq/netpkg-tool/general, then in the general tab there's a ReadMe section with an edit button in the form of a pencil). After that, when changing the README in the GitHub repository, the Docker Cloud and Docker Hub image descriptions where updated accordingly. I have issues with this in my repository as well.