Skip to content

Commit

Permalink
Inital devdocs (#144)
Browse files Browse the repository at this point in the history 
These Docs include a README.md for initial navigation.

Docs on:
- The Architecture
- Pipeline setup
- Future Development suggestions
- Filestructur
- Build/scripts & cmd

you can find the rest IN the docs

---------

Co-authored-by: mnilsen <[email protected]>
Co-authored-by: sebmid <[email protected]>
  • Loading branch information
@Trivinyx @ostepizza @sebmid
3 people authored May 2, 2024
1 parent 9588a0f commit 2e50fdf
Show file tree
Hide file tree
Showing 10 changed files with 551 additions and 0 deletions.
24 changes: 24 additions & 0 deletions docs/Architecture.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Application Architecture

[Back to README](README.md)

---
![Arcitecture Image](./img/Img2map%20Architecture.svg)

In the image above you can see how the current (02.05.24) architechture is designed. Starting with the client connecting to the serever on herkoku, and getting the pages.

From the inital page the lines to other pages indicate navigation, while lines to components shows firstly which components are in use, and how it flows througout the components.

The componets have two further connections idicating back and fourth communication. The first on the bottom of the components linking outside to "Mapbox libraries", this is what we found to be most descriptive. Inn the components linking to "Mapbox libraries", there are internal packages in use that is making a digital map, and interacting with the "Library". Now to the second, this is an internal to Axios. Axios is a package that is used to communicate with the backend over HTTP requests as seen.

Going over to the sepratly hosted backend on heroku, we have a uvicorn server running with a FastAPI application. This application recives the request and routes them according to the url specifcations.

Going with conversion router, it will send it to the core and return the processed result over http as a response to the frontend, going through axios to the apropriate component for display.

Heading trough the project/georef we are sent to the core for proceccing. The core logic will have one or multiple iterations of interactions with the file storage or the database handler, before finally returning in to the router for formatting a response, from here it is same path as conversion.

The database handler when reciving calls from the core logic will execute based on the call operations on a postgresSql DB through its connection. The DB in the instance is part of heroku addons, which are services provided by heroku to or in the instance.

The final part to look at is the file Storage handler, Bucketeer and Amazone S3 bucket. File storage handler behaves much in the same way as the DB handler executing calls request based on the type, where it differs is the connection. It recives its credentials from bucketeer (Heroku addon) which creates and manage the Amazone S3 bucket.

With the credatials from bucketeer, the file storage handler can do file operations inside of the bucket, this completes the architecture.
157 changes: 157 additions & 0 deletions docs/FutureDevSuggestions.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
# Further Development Suggestions

[Back to README](README.md)

---
> **Authors:** *Tom A. S. Myre, Markus Nilsen, Marius Evensen, Sebastian Midtskogen & Lukas A. Andersen*
> **Date:** 02.05.2024
This document is intended to provide our insight on further devolpment after our handover. As a helping guide we will provide the suggestions on genral improvments, new features and expansions.
> **Note:** There is also enhancments in Issues which are not mentioned here.
Defining the general structure of the suggestion, The topp line will consist of a title and a Type Category. Next will have two fields/keywords describing how difficult and our amount of reasearch & testing done. Below is a list of the categories and a short note on what we mean, further down is an example.

1. Type:

* **Front-end** *(The Next.js application / website)*
* **Back-end** *(The FastApi / processing / backend connected Services)*
* **Fullstack** *(Encompassing both)*
* **unknown** *(Unsure of where to place or the current types are not descriptive)*

2. Difficulity:

* **Low** *(Reletivly easy, considering context)*
* **Medium** *(Some difficulity expected/ time)*
* **High** *(Expected high difficulity / high timeconsumption)*
* **unknown** *(We can not place it/ we are unsure)*

3. Researched:

* **None** *(Only based on our intuition, given our work with the project)*
* **Low** *(Some testing and/or Reading some source)*
* **Medium** *(Decent amount of testing and/or Reading Sources)*
* **High** *(Good amount of testing and/or Good source reading)*

---

**Example, Front-End:**
>
> * `Difficulity: Medium`
> * `Research: Medium`
>
> **Description**: what, why & possibly a theoretical suggested solution.
## High priority

**Point error calculation, Fullstack:**

> * `Difficulity: High`
> * `Research: Medium`
>
> **Description**: Calculates how far off your original points where from their real positions. This is supposed to be done when users adds more then the minimum amounts of points to a map for the goerefrence. This will then improved accuracy, and they would be able to see how far off their original marker placements where compared to the re-referenced points on the map.
>
> A suggestion on this is to create an algorithm that crates a [affine transformation](https://se.mathworks.com/discovery/affine-transformation.html) matrix, to have more control of calculation parameters. In this case it needs to be [accepted](https://rasterio.readthedocs.io/en/stable/topics/transforms.html#using-affine-transformation-matrix) by rasterio.
>
> Another possible solution is to convert to the base use of gdal for tranformation, that option has not been as thoroughly explored.
### Good Starter

**Logging Middleware, Front-End:**
>
> * `Difficulity: Medium`
> * `Research: low`
>
> **Description**: Implement a standarized logging middleware to log crucial information during runtime, and generate better metrics. This would also be beneficial to have different loglevels in the code to differentiate good devolopment information and staged runtime logs. This is also good for getting to know the codebase.
## Medium priority

**Rotation of Image or PDF With map, Fullstack:**
>
> * `Difficulity: Low`
> * `Research: Low`
>
> **Description**: Simply put, to be able to synchronously rotate the image side along with the map (split view) when it has been georefrenced. This is for ease of use, for the user, to be able to align the side by side and recognition of features.
>
> What we know is that it theoreticaly can be done after georefrencing, by getting the corner bounds of the image to calculate the line to north from center, then adjust the start by rotating to north with getting the degrees between the north line and center to top. From this point it is to extract the rotatation from mapbox and mimic.
>
> **Subfeature:** To be able to lock the image upright after triggering min requriments for the feature above.
**Refactor ProjectHanlder, Back-End:**

> * `Difficulity: Medium`
> * `Research: low`
>
> **Description**: Seprate projecthandler into smaller classes. The Reasone for this is the current readablility of codebase and that currently the projecthandlerclass have to much resposebility.
>
> A Suggestion on our part would be to split out general point handeling to one class and handeling of georefenicing to another.
>
> A sub suggestion to the privious is to have the new georefrencing handler know of the project handler, which in turn uses point handler. The router will use all three.
>
**Suggested points, Fullstack:**
>
> * `Difficulity: High`
> * `Research: Medium`
>
> **Description**: With the inital georefrence, the software would suggest points on the map to be placed. This would greatly improve the user experience and the learning curve for new users of this kind of software. But this might be quite a difficult task as it would probably have to include some kind of artificial Intelligence to or advanced algorithm to achieve this at the level we want.
>
> In terms of an algorithm this would then most likely depend on different reworks like the error point calculations.
### Good starters

**A General HttexceptionHandler, Back-End:**
>
> * `Difficulity: Low`
> * `Research: Medium`
>
> **Description**: Create a common exeption class to respond with the apropriate Http Exeptions in response, this is to consolidate the error response for the routers (Easier readability). A possible way to do this is with python decorators, a fastapi dependecy injection or a combination of both.
**Rotate Image, Front-End:**
>
> * `Difficulity: Low`
> * `Research: Medium`
>
> **Description**: Simply to rotate the uploaded image, this is usefull when a user have a pdf where the Image is rotated sideways. This is not a feature we have previously prioritized but is more prevelent now, Should be relativly easy to implement.
**Automated Testing, Back-end:**
>
> * `Difficulity: Medium`
> * `Research: Low`
>
> **Description**: Create unit test for the core logic classes in the Back-end API, This is a bit of work but good for familiarization with the code. This also have the added benefit of a more thoroughly checking of the code which gives more cofidence to making changes and ensuring that the application works as intended.
## Low priority

**Clock to run on heroku, Back-End:**
>
> * `Difficulity: Medium`
> * `Research: Medium`
>
> **Description**: The clock scrpits intention is to run perioadic tasks on the database,this is to clean the stroages of stale project.
>
> This would be an full implementation of the projectSelf destruct where the clock would fetch from the database the projects which are over the limit, say once every 10 mins, and delete the ones that are over. We would also recommend to set new selfdestruct time each time a project is updated, to prevent active projects from being deleted. A source on [heroku clock setup](https://devcenter.heroku.com/articles/scheduled-jobs-custom-clock-processes)
**Workers, Back-End:**
>
> * `Difficulity: High`
> * `Research: High`
>
> **Description**: Creation of a worker to offload the more processing heavy task from the FastAPI, This is to better the application for scalability and making it more redundant. A worker should handle all the file saving, manipulation and creation. For communication between it is recommended to use a Message Broaker.
>
> A suggestion from is to use a combination of [RabbitMQ](https://www.rabbitmq.com/) (Message Broaker) and [Celary](https://docs.celeryq.dev/en/stable/getting-started/introduction.html) to implement this. Why RabbitMQ, it might be more time consuming and complex to setupp, but it got innbuilt redundancy in the queues and more features, hence the complexity. But the on of the alternatives Redis queue is more bare bones and is possible to setup with features like Rabbit but is wary manual, and another good thing heroku has a addon for a [RabbitMq server](https://elements.heroku.com/addons/cloudamqp), it has built in metrics.
**Edition of placed Marker Pairs(Points), Front-End:**
>
> * `Difficulity: Low`
> * `Research: Medium`
>
> **Description**: To be able to select a marker pair from the cordinates table, adjust them and have them updated. Better usability, user might have confirmed a pair before properly checking and might want to adjust either the cordinates (Map) or pixel placement (image). This is low becouse the exsitance of deletion.
>
> The backend endpoint for updating a point pair already exsist and the data to do so is in the front-end, but the user interaction and flow is missing.
**Multiple Image Georefrencing, Fullstack:**
>
> * `Difficulity: High`
> * `Research: None`
>
> **Description**: In theory it would have users be able to upload multiple images at once and georefrence them together, and see them on overlay. Another is bunch upload and sepratly have a work lits to georefrence trough each one then be able to see them all on overlay. This would require a more substantial change of the backeds way of storing project filepaths, and how it handels them, but also on how to manage and view them on the frontend would reqire substantial rework.
41 changes: 41 additions & 0 deletions docs/PipelinesSetup.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Pipelines setup

[Back to README](README.md)

---
This doc will outline in further detail how the pipeline[s] (Github Actions) are working, the external setup of repository secrets and variables, and the general purpose the pipelines.

## Deploy to Heroku

>This action is triggered by a tag matching the regular expression `v.0.[0-9]+.[0-9]`, but not if trailed by `-dev`.
This action has 2 jobs (these can be split later). When the action is triggered, the application is built based on the code of the current branch and commit that triggered the action. Each job then builds their respective Docker-images and -containers, then use a third party action to deploy the respective images to the Heroku apps.

### Github Secrets & Variables

Following are the names of the GitHub action secrets and variables, and explaining their purpose and/or need, related to this pipeline/action.

> **Secrets:**
* **BACKEND_API_URL :** The URL to the hosted instance of the Img2map-Backend. Needed environment variable for the frontend to communicate with the backend API. Needed for app function.

`Note: If the URL has a trailing '/', remove it`

`Note: This could possibly have been in vars`

* **MAPBOX_ACCESS_TOKEN :** This gives the frontend access to Mapbox's API-resources, used to render digital maps and access other features provided by Mapbox. Needed for app function.
* **HEROKU_API_KEY :** Heroku account- or team-specific key. Needed for deployment authorization.
* **HEROKU_EMAIL :** The registered email for the Heroku account. Needed for deployment authorization.
* **HEROKU_API_APP_NAME :** The Heroku app-name for the backend on Heroku. This gives the action the target location on Heroku (where it deploys to).
* **HEROKU_WEB_APP_NAME :** Same as above, but for the frontend application.

> **Variables:**
* **CORS_ORIGINS :** List of which URLs that are allowed to talk to the hosted backend.

Example: `http://example.com,http://example2.com`

`Note: If the URL has a trailing '/', remove it`
* **NEXT_PUBLIC_PDF_URLS :** A list of domains that will display "PDF2Map" rather than "Image2Map".

Example: `example.com,example2.com`, where example3.com will show Image2Map and example2.com will show PDF2Map.
23 changes: 23 additions & 0 deletions docs/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# DOCS :D

Welcome to the docs, below you will find verious links to different topics.

## General

[Architecture](Architecture.md)

[Setting Up Pipelines](PipelinesSetup.md)

[Future Development Suggestions](FutureDevSuggestions.md)

## Backend

[Backend File Struckture](./backend/FileStructure.md)

[Backend Scripts, cmd & build](./backend/ScriptsBuildCMD.md)

## Frontend

[Front-end File Struckture](./frontend/FileStructure.md)

[Front-end Scripts, cmd & build](./frontend/ScriptsBuildCMD.md)
146 changes: 146 additions & 0 deletions docs/backend/FileStructure.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
# Filestructure

[Back to README](../README.md)

---
The intention of this file is to give a general overview of the filestruckture, and explaining our thought process on the responsibility and purpose of/for a file.

## Current structur (with tree cmd)

Command: `PS <path> tree /f /a`

```cmd
c://backend
| .dockerignore
| .env.example
| main.py
| pyproject.toml
| requirements.txt
|
+---docker
| dev.Dockerfile
| docker-compose.backend.yml
| prod.Dockerfile
|
\---img2mapAPI
| Img2mapAPI.py
| __init__.py
|
+---routers
| converters.py
| georefProject.py
| server.py
| __init__.py
|
\---utils
| projectHandler.py
| __init__.py
|
+---core
| | FileHelper.py
| | georefHelper.py
| | ImageHelper.py
| |
| \---helper
| postgresSqlHelper.py
| sqliteHelper.py
| __init__.py
|
+---models
| point.py
| pointList.py
| project.py
| __init__.py
|
\---storage
| __init__.py
|
+---data
| postgresSqlHandler.py
| sqliteLocalStorage.py
| storageHandler.py
| __init__.py
|
\---files
fileStorage.py
localFileStorage.py
s3FileStorage.py
__init__.py
```

## Root level files

* `.dockerignore` : Tells Docker-Deamon what files to ignore when building.

* `.env.example` : Evironment varibles example file

* `main.py` : The entrypoint script for the application. [More Info](ScriptsBuildCMD)

* `pyproject.toml`: Project package configuration and setup.

* `requirements.txt`: Local requirments file for running

## Directories

Showing only the Dirs:

```shell
+---docker
\---img2mapAPI
+---routers
\---utils
+---core
| \---helper
+---models
\---storage
+---data
\---files
```

`docker`: Contains the Docker build files and docker-compose file spesificly for the backend.

### Img2mapApi (top package dir)

This is the top package directory, it contains the module which holds the FastAPI app and Setup `Img2mapAPI.py`, and below these directories.

#### Routers

> Contain two api routers.
#### Utils

1. `core` : Conatins the sub dir below, and the core modules for georef function (`georefHelper.py`), Image managment functions(`imageHelper.py`) & general temp file managment help functions(`FileHelper.py`).

* `helper` : Contains additional helper modules, (aka other functions)

2. `models` : Contains the model classes used by the application.

3. `storage` : Contains two sub dirs, collection for classes and functions dealing with storage of files or data.

* `data` : Contains the abstract class for datahandling and other classe who inherith from this base class and implement spesifik handlers for database interaction.
* `files` : Same principel as above, just handeling file storage instead.

## Major files

This is mainly to cover more on files (Modules) that need more description then what is given by the previous section.

> **Note:** `__init__.py` files is for module structure and importing.
1. `projectHandler.py`

This module is part of the core and is resposible for pre-prossesing, preparing models, calling needed functions and classes. It handels all things related to projects, this includes projects, points & images related to the project.

2. `converters.py` & `georefProject.py`

They contain the routing endpoint logic, converters is image cropping, pdf to image and image to image and georefProject handels endpoint logic and mostly call `projecthandler`.

georefProject does some dependency injection to give `projectHandeler` the deps it need to store files and data (see `storage` description).

3. `storageHandler.py` & `fileStorage.py`

They are modules conatining abstract classes with abstract functions, mimicing a interface. They are bases to be used as types for dependecy injection, and implementers to inherit.

`storageHandler` is for Data storage, commonly manage connection to a database.

`fileStorage` is for file storage, example implementiation is a class connecting to Amazone S3, saving, deleting, fecthing etc.
Loading

0 comments on commit 2e50fdf

Please sign in to comment.