Intermittent issues after migration

[tws101]

Recently migrated from Docker running in a debian jail to Native Docker in TrueNAS Scale

Running version 20241106_101604

Nextcloud intermittently becomes unresponsive

Logs

nextcloud-aio-apache       | Waiting for Nextcloud to start...
nextcloud-aio-database     | Setting postgres values...
nextcloud-aio-database     | 
nextcloud-aio-database     | PostgreSQL Database directory appears to contain a database; Skipping initialization
nextcloud-aio-database     | 
nextcloud-aio-notify-push  | nc: getaddrinfo for host "nextcloud-aio-nextcloud" port 9001: Name does not resolve
nextcloud-aio-notify-push  | Waiting for Nextcloud to start...
nextcloud-aio-notify-push  | Waiting for Nextcloud to start...
nextcloud-aio-database     | 2024-11-12 17:26:32.891 CST [14] LOG:  starting PostgreSQL 16.4 on x86_64-pc-linux-musl, compiled by gcc (Alpine 13.2.1_git20240309) 13.2.1 20240309, 64-bit
nextcloud-aio-database     | 2024-11-12 17:26:32.891 CST [14] LOG:  listening on IPv4 address "0.0.0.0", port 5432
nextcloud-aio-database     | 2024-11-12 17:26:32.891 CST [14] LOG:  listening on IPv6 address "::", port 5432
nextcloud-aio-nextcloud    | Connection to nextcloud-aio-database (172.24.20.4) 5432 port [tcp/postgresql] succeeded!
nextcloud-aio-nextcloud    |               now              
nextcloud-aio-database     | 2024-11-12 17:26:33.071 CST [14] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
nextcloud-aio-database     | 2024-11-12 17:26:33.383 CST [24] LOG:  database system was shut down at 2024-11-12 17:25:39 CST
nextcloud-aio-database     | 2024-11-12 17:26:33.391 CST [14] LOG:  database system is ready to accept connections
nextcloud-aio-redis        | Redis has started
nextcloud-aio-nextcloud    | -------------------------------
nextcloud-aio-nextcloud    |  2024-11-12 17:26:37.921291-06
nextcloud-aio-nextcloud    | (1 row)
nextcloud-aio-nextcloud    | 
nextcloud-aio-nextcloud    | + '[' -f /dev-dri-group-was-added ']'
nextcloud-aio-nextcloud    | ++ find /dev -maxdepth 1 -mindepth 1 -name dri
nextcloud-aio-nextcloud    | + '[' -n '' ']'
nextcloud-aio-nextcloud    | + set +x
nextcloud-aio-nextcloud    | Enabling Imagick...
nextcloud-aio-nextcloud    | WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.20/main: No such file or directory
nextcloud-aio-nextcloud    | WARNING: opening from cache https://dl-cdn.alpinelinux.org/alpine/v3.20/community: No such file or directory
nextcloud-aio-nextcloud    | Connection to nextcloud-aio-redis (172.24.20.3) 6379 port [tcp/redis] succeeded!
nextcloud-aio-notify-push  | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | Applying one-click-instance settings...
nextcloud-aio-apache       | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | System config value one-click-instance set to boolean true
nextcloud-aio-notify-push  | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | System config value one-click-instance.user-limit set to integer 100
nextcloud-aio-nextcloud    | System config value one-click-instance.link set to string https://nextcloud.com/all-in-one/
nextcloud-aio-nextcloud    | support already enabled
nextcloud-aio-nextcloud    | Adjusting log files...
nextcloud-aio-nextcloud    | System config value upgrade.cli-upgrade-link set to string https://github.com/nextcloud/all-in-one/discussions/2726
nextcloud-aio-apache       | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | System config value logfile set to string /var/www/html/data/nextcloud.log
nextcloud-aio-notify-push  | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | Config value were not updated
nextcloud-aio-nextcloud    | System config value updatedirectory set to string /nc-updater
nextcloud-aio-nextcloud    | System config value maintenance_window_start set to integer 100
nextcloud-aio-nextcloud    | Applying network settings...
nextcloud-aio-nextcloud    | System config value allow_local_remote_servers set to boolean true
nextcloud-aio-apache       | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | System config value davstorage.request_timeout set to integer 3600
nextcloud-aio-notify-push  | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | System config value trusted_domains => 1 set to string nextcloud.domainredacted.net
nextcloud-aio-nextcloud    | System config value overwrite.cli.url set to string https://nextcloud.domainredacted.net/
nextcloud-aio-nextcloud    | System config value htaccess.RewriteBase set to string /
nextcloud-aio-nextcloud    | .htaccess has been updated
nextcloud-aio-apache       | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | System config value dbpersistent set to boolean false
nextcloud-aio-notify-push  | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | System config value auth.bruteforce.protection.enabled set to boolean true
nextcloud-aio-nextcloud    | System config value ratelimit.protection.enabled set to boolean true
nextcloud-aio-nextcloud    | System config value files_external_allow_create_new_local set to boolean false
nextcloud-aio-apache       | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | No such app enabled: nextcloud-aio
nextcloud-aio-notify-push  | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | notify_push is up-to-date or no updates could be found
nextcloud-aio-nextcloud    | System config value trusted_proxies => 0 set to string 127.0.0.1
nextcloud-aio-apache       | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | System config value trusted_proxies => 1 set to string ::1
nextcloud-aio-notify-push  | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | System config value trusted_proxies => 2 set to string 172.24.0.0/23
nextcloud-aio-nextcloud    | System config value trusted_proxies => 10 set to string 172.24.20.0/16
nextcloud-aio-nextcloud    | Config value were not updated
nextcloud-aio-nextcloud    | + '[' '' = true ']'
nextcloud-aio-nextcloud    | + set +x
nextcloud-aio-apache       | Waiting for Nextcloud to start...
nextcloud-aio-notify-push  | Waiting for Nextcloud to start...
nextcloud-aio-nextcloud    | [12-Nov-2024 17:27:16] NOTICE: fpm is running, pid 356
nextcloud-aio-nextcloud    | [12-Nov-2024 17:27:16] NOTICE: ready to handle connections
nextcloud-aio-apache       | Connection to nextcloud-aio-nextcloud (172.24.20.5) 9000 port [tcp/*] succeeded!
nextcloud-aio-notify-push  | Connection to nextcloud-aio-nextcloud (172.24.20.5) 9001 port [tcp/*] succeeded!
nextcloud-aio-notify-push  | notify-push was started
nextcloud-aio-apache       | [Tue Nov 12 17:27:21.487272 2024] [mpm_event:notice] [pid 67:tid 67] AH00489: Apache/2.4.62 (Unix) configured -- resuming normal operations
nextcloud-aio-apache       | [Tue Nov 12 17:27:21.487342 2024] [core:notice] [pid 67:tid 67] AH00094: Command line: '/usr/local/apache2/bin/httpd -D FOREGROUND'
nextcloud-aio-apache       | {"level":"info","ts":1731454041.5182633,"msg":"using config from file","file":"/tmp/Caddyfile"}
nextcloud-aio-apache       | {"level":"info","ts":1731454041.5205348,"msg":"adapted config to JSON","adapter":"caddyfile"}

Logs appear completely clean... If the containers are destroyed and rebuild it may more may not come up with no noticeable difference in the logs

Docker Compose (Collabra, Only Office and Full text search were removed for trouble shooting

networks:
  nextcloud:
    name: nextcloud
    ipam:
      driver: default
      config:
        - subnet: 172.24.20.0/24
          ip_range: 172.24.20.0/24
          gateway: 172.24.20.1
  proxy:
    name: proxy
    external: true
services:
  nextcloud-aio-apache:
    container_name: nextcloud-aio-apache
    depends_on:
      nextcloud-aio-nextcloud:
        condition: service_started
        required: false
      nextcloud-aio-notify-push:
        condition: service_started
        required: false
    image: nextcloud/aio-apache:${VERSION}
    user: 33:0
    init: true
    expose:
      - 11000
    environment:
      - NC_DOMAIN=${NC_DOMAIN}
      - NEXTCLOUD_HOST=${NEXTCLOUD_HOST}
      - APACHE_HOST=${APACHE_HOST}
      - APACHE_PORT=${APACHE_PORT}
      - TZ=${TIMEZONE}
      - APACHE_MAX_SIZE=${APACHE_MAX_SIZE}
      - APACHE_MAX_TIME=${NEXTCLOUD_MAX_TIME}
      - NOTIFY_PUSH_HOST=${NOTIFY_HOST}
    volumes:
      - ${NEXTCLOUD_DATA}:/var/www/html:ro
      - ${APACHE_DATA}:/mnt/data:rw
    restart: unless-stopped
    networks:
      - proxy
      - nextcloud
    labels:
      - traefik.enable=true
      - traefik.http.routers.nextcloud.entrypoints=websecure
      - traefik.http.routers.nextcloud.rule=Host(`${NC_DOMAIN}`)
      - traefik.http.routers.nextcloud.tls=true
      - traefik.http.services.nextcloud.loadbalancer.server.port=11000
      - traefik.http.routers.nextcloud.middlewares=nextcloud-secure-headers
      - traefik.http.middlewares.nextcloud-secure-headers.headers.hostsProxyHeaders=X-Forwarded-Host
      - traefik.http.middlewares.nextcloud-secure-headers.headers.referrerPolicy=same-origin
    read_only: true
    tmpfs:
      - /var/log/supervisord
      - /var/run/supervisord
      - /usr/local/apache2/logs
      - /tmp
      - /home/www-data
    cap_drop:
      - NET_RAW
    deploy:
      resources:
        limits:
          cpus: "2"
          memory: 4G
  nextcloud-aio-database:
    container_name: nextcloud-aio-database
    image: nextcloud/aio-postgresql:${VERSION}
    user: 999:0
    init: true
    expose:
      - 5432
    volumes:
      - ${DATABASE_DATA}:/var/lib/postgresql/data:rw
      - ${DATABASE_DUMP_DATA}:/mnt/data:rw
    environment:
      - POSTGRES_PASSWORD=${DATABASE_PASSWORD}
      - POSTGRES_DB=nextcloud_database
      - POSTGRES_USER=nextcloud
      - TZ=${TIMEZONE}
      - PGTZ=${TIMEZONE}
    stop_grace_period: 1800s
    restart: unless-stopped
    shm_size: 268435456
    networks:
      - nextcloud
    read_only: true
    tmpfs:
      - /var/run/postgresql:uid=999,gid=0,mode=0755
    cap_drop:
      - NET_RAW
    deploy:
      resources:
        limits:
          cpus: "2"
          memory: 4G
  nextcloud-aio-nextcloud:
    container_name: nextcloud-aio-nextcloud
    depends_on:
      nextcloud-aio-database:
        condition: service_started
        required: false
      nextcloud-aio-redis:
        condition: service_started
        required: false
    image: nextcloud/aio-nextcloud:${VERSION}
    user: 0:0
    init: true
    expose:
      - 9000
      - 9001
    volumes:
      - ${NEXTCLOUD_DATA}:/var/www/html:rw
      - ${NEXTCLOUD_DATADIR}:/mnt/ncdata:rw
      #- ${NEXTCLOUD_MOUNT}:${NEXTCLOUD_MOUNT}:rw
      #- ${NEXTCLOUD_TRUSTED_CACERTS_DIR}:/usr/local/share/ca-certificates:ro
    environment:
      - POSTGRES_HOST=${DATABASE_HOST}
      - POSTGRES_PORT=5432
      - POSTGRES_PASSWORD=${DATABASE_PASSWORD}
      - POSTGRES_DB=nextcloud_database
      - POSTGRES_USER=nextcloud
      - REDIS_HOST=${REDIS_HOST}
      - REDIS_HOST_PASSWORD=${REDIS_PASSWORD}
      - NC_DOMAIN=${NC_DOMAIN}
      - ADMIN_USER=admin
      - ADMIN_PASSWORD=${NEXTCLOUD_PASSWORD}
      - NEXTCLOUD_DATA_DIR=/mnt/ncdata
      - OVERWRITEHOST=${NC_DOMAIN}
      - OVERWRITEPROTOCOL=https
      - TURN_SECRET=${TURN_SECRET}
      - SIGNALING_SECRET=${SIGNALING_SECRET}
      - ONLYOFFICE_SECRET=${ONLYOFFICE_SECRET}
      - CLAMAV_ENABLED=${CLAMAV_ENABLED}
      - ONLYOFFICE_ENABLED=${ONLYOFFICE_ENABLED}
      - COLLABORA_ENABLED=${COLLABORA_ENABLED}
      - COLLABORA_HOST=${COLLABORA_HOST}
      - TALK_ENABLED=${TALK_ENABLED}
      - ONLYOFFICE_HOST=${ONLYOFFICE_HOST}
      - UPDATE_NEXTCLOUD_APPS=${UPDATE_NEXTCLOUD_APPS}
      - TZ=${TIMEZONE}
      - IMAGINARY_ENABLED=${IMAGINARY_ENABLED}
      - IMAGINARY_HOST=${IMAGINARY_HOST}
      - PHP_UPLOAD_LIMIT=${NEXTCLOUD_UPLOAD_LIMIT}
      - PHP_MEMORY_LIMIT=${NEXTCLOUD_MEMORY_LIMIT}
      - FULLTEXTSEARCH_ENABLED=${FULLTEXTSEARCH_ENABLED}
      - FULLTEXTSEARCH_HOST=${FULLTEXT_HOST}
      - PHP_MAX_TIME=${NEXTCLOUD_MAX_TIME}
      - STARTUP_APPS=${NEXTCLOUD_STARTUP_APPS}
      - ADDITIONAL_APKS=${NEXTCLOUD_ADDITIONAL_APKS}
      - ADDITIONAL_PHP_EXTENSIONS=${NEXTCLOUD_ADDITIONAL_PHP_EXTENSIONS}
      - INSTALL_LATEST_MAJOR=${INSTALL_LATEST_MAJOR}
      - FULLTEXTSEARCH_PASSWORD=${FULLTEXTSEARCH_PASSWORD}
      - REMOVE_DISABLED_APPS=${REMOVE_DISABLED_APPS}
      - APACHE_PORT=${APACHE_PORT}
      - IMAGINARY_SECRET=${IMAGINARY_SECRET}
      - ADDITIONAL_TRUSTED_PROXY=${ADDITIONAL_TRUSTED_PROXY}
    stop_grace_period: 600s
    restart: unless-stopped
    networks:
      - nextcloud
    cap_drop:
      - NET_RAW
    deploy:
      resources:
        limits:
          cpus: "4"
          memory: 8G
  nextcloud-aio-notify-push:
    container_name: nextcloud-aio-notify-push
    image: nextcloud/aio-notify-push:${VERSION}
    user: 33:0
    init: true
    expose:
      - 7867
    volumes:
      - ${NEXTCLOUD_DATA}:/nextcloud:ro
    environment:
      - NC_DOMAIN=${NC_DOMAIN}
      - NEXTCLOUD_HOST=${NEXTCLOUD_HOST}
      - REDIS_HOST=${REDIS_HOST}
      - REDIS_HOST_PASSWORD=${REDIS_PASSWORD}
      - POSTGRES_HOST=${DATABASE_HOST}
      - POSTGRES_PORT=5432
      - POSTGRES_PASSWORD=${DATABASE_PASSWORD}
      - POSTGRES_DB=nextcloud_database
      - POSTGRES_USER=nextcloud
    restart: unless-stopped
    networks:
      - nextcloud
    read_only: true
    cap_drop:
      - NET_RAW
    deploy:
      resources:
        limits:
          cpus: "2"
          memory: 4G
  nextcloud-aio-redis:
    container_name: nextcloud-aio-redis
    image: nextcloud/aio-redis:${VERSION}
    user: 999:0
    init: true
    expose:
      - 6379
    environment:
      - REDIS_HOST_PASSWORD=${REDIS_PASSWORD}
      - TZ=${TIMEZONE}
    volumes:
      - ${REDIS_DATA}:/data:rw
    restart: unless-stopped
    networks:
      - nextcloud
    read_only: true
    cap_drop:
      - NET_RAW
    deploy:
      resources:
        limits:
          cpus: "2"
          memory: 4G

Questions

1. Could this be a kernel issue?
   fs.inotify.max_user_watches set to 4962130
   fs.inotify.max_user_instances set to 8000
   all others default for the OS

2. Could this be a apache issue? I have been looking into just removing apache all together but I believe the nextcloud AIO app must have apache or it wont work all efforts to address nextcloud directly and not through apache are failing. If anyone knows the answer to this second question specifically can I use the nextcloud AIO stack without apache this would help troubleshoot.

[szaimen]

Hi, please post this at https://help.nextcloud.com. Thanks!