59 deploying to dev (#64)

* 59-deploying-to-dev

- Added a root level .env file to allow overriding config further downstream by having a nested file structure.
- Moved env/.env* to env/local/.env*, defaulting ENVIRONMENT=local
- Created suitable config in env/dev for all services to bootstrap
- Modified docker-compose.yml to take all configs from env files, with the ENVIRONMENT dir prefixing them.
- Added some TODO's in docker-compose.yml for ops concerns (will do those later)
- Changed _ to - in "trademodule_postgress" as this was the only docker service with underscores.
- Changed the postgres health check so that postgress doesn't spam logs.
- Updated project readme.md for deployment instructions.

* 59-deploying-to-dev

- Corrected readme.md

---------

Co-authored-by: Dave Robertson <[email protected]>

.env

@@ -0,0 +1 @@
+ENVIRONMENT=local

docker-compose.yml

@@ -3,6 +3,7 @@ version: "3.8"
 name: "edpn-backend"
 services:
   zookeeper:
+  ## TODO: Tag this image locally
     image: "bitnami/zookeeper:3.8"
     restart: on-failure
     environment:
@@ -17,13 +18,11 @@ services:
       start_period: 0s
 
   kafka:
+  ## TODO: Tag this image locally
     image: "bitnami/kafka:3.4"
     restart: on-failure
-    environment:
-      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
-      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
-      - ALLOW_PLAINTEXT_LISTENER=yes
-      - KAFKA_CFG_COMPRESSION_TYPE=gzip
+    env_file:
+          - env/${ENVIRONMENT}/.env_kafka
     volumes:
       - "kafka:/bitnami/kafka"
     healthcheck:
@@ -36,15 +35,16 @@ services:
       zookeeper:
         condition: service_healthy
 
-  trademodule_postgres:
+  trademodule-postgres:
+  ## TODO: Tag this image locally
     image: "postgres:15-alpine"
     restart: on-failure
     env_file:
-      - env/.env_trademodule
+      - env/${ENVIRONMENT}/.env_trademodule
     volumes:
       - "trademodule_postgres:/var/lib/postgresql/data"
     healthcheck:
-      test: pg_isready
+      test: ["CMD-SHELL", "pg_isready -d trademodule -U edpn_trademodule"]
       interval: 10s
       timeout: 5s
       retries: 5
@@ -53,7 +53,7 @@ services:
     image: nexus.edpn.io/edpn/backend/eddn-message-listener:latest-snapshot
     restart: on-failure
     env_file:
-      - env/.env_eddn_message_listener
+      - env/${ENVIRONMENT}/.env_eddn_message_listener
     healthcheck:
       test: "wget -qO- localhost:8080/actuator/health | grep UP || exit 1"
       interval: 20s
@@ -68,7 +68,7 @@ services:
     image: nexus.edpn.io/edpn/backend/modulith:latest-snapshot
     restart: on-failure
     env_file:
-      - env/.env_modulith
+      - env/${ENVIRONMENT}/.env_modulith
     ports:
       - "8080:8080"
     healthcheck:

env/dev/.env_eddn_message_listener

@@ -0,0 +1,7 @@
+# .env_eddn_message_listener file
+
+#eddn
+EDDN_URI=tcp://eddn.edcd.io:9500:9500
+
+# kafka
+KAFKA_URL=edpn-backend-kafka-1:9092

env/dev/.env_kafka

@@ -0,0 +1,4 @@
+KAFKA_CFG_ZOOKEEPER_CONNECT=edpn-backend-zookeeper-1:2181
+KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://edpn-backend-kafka-1:9092
+ALLOW_PLAINTEXT_LISTENER=yes
+KAFKA_CFG_COMPRESSION_TYPE=gzip

env/dev/.env_modulith

@@ -0,0 +1,9 @@
+# .env_boot file
+
+# kafka
+KAFKA_URL=edpn-backend-kafka-1:9092
+
+# tradeModule
+TRADEMODULE_DB_URL=jdbc:postgresql://edpn-backend-trademodule-postgres-1:5432/trademodule
+TRADEMODULE_DB_USERNAME=edpn_trademodule
+TRADEMODULE_DB_PASSWORD=MSDVq372rNk3AwRq

env/local/.env_kafka

@@ -0,0 +1,4 @@
+KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
+KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
+ALLOW_PLAINTEXT_LISTENER=yes
+KAFKA_CFG_COMPRESSION_TYPE=gzip

env/local/.env_trademodule

@@ -0,0 +1,6 @@
+# .env_trademodule file
+
+# PostgreSQL
+POSTGRES_USER=edpn_trademodule
+POSTGRES_PASSWORD=MSDVq372rNk3AwRq
+POSTGRES_DB=trademodule

readme.md

@@ -8,6 +8,7 @@ The Elite Dangerous Pilot Network Backend (EDPN Backend) project provides a REST
     - [Code structure](#code-structure)
 - [Installation](#installation)
 - [Local Development](#local-development)
+- [Deploying](#deploying)
 - [Data flow](#data-flow)
 - [Reporting Issues](#reporting-issues)
 - [Contributing](#contributing)
@@ -19,7 +20,7 @@ ___
 ## Technologies Used
 The EDPN Backend project is built using the following technologies:
 
-- [Sp[readme.md](readme.md)ring Boot](https://spring.io/projects/spring-boot): An open-source Java-based framework used to create stand-alone, production-grade Spring applications quickly and easily.
+- [Spring Boot](https://spring.io/projects/spring-boot): An open-source Java-based framework used to create stand-alone, production-grade Spring applications quickly and easily.
 - [Maven](https://maven.apache.org/): A build automation tool used to manage dependencies and build Java projects.
 - [Kafka](https://kafka.apache.org/): A distributed streaming platform used to build real-time data pipelines and streaming applications.
 - [Postgres](https://www.postgresql.org/): An open-source relational database management system used to store data for the EDPN Backend project.
@@ -128,6 +129,19 @@ to run the stack for local development, follow these steps:
 
 in short: docker compose up the stack, run both of the projects from IDE
 
+---
+## Deploying
+Note to Ops, you need the following files/folders from the repository:
+
+- docker-compose.yml
+- .env
+- env/dev (if deploying to a hosted dev environment)
+
+Save the files, using the existing folder structure from this repository, starting at `/opt/edpn-backend` - set file permissions appropriately (ops group, and 660 on files). Next, edit the `.env` file, setting `ENVIRONMENT=dev`. 
+
+Finally, in that directory, `docker compose up`. Everything should come up and show healthy within a few minutes.
+
+In future, we should automate this via pipeline, etc.
 ___
 ## Reporting Issues
 To report an issue with the EDPN Backend project or to request a feature, please open an issue on the project's GitHub repository. You can also join the [discord](https://discord.gg/RrhRmDQD) and make a suggestion there in `ideas` section.