Merge branch '#101-introduce-Atlas'

arc42 · Jan 27, 2024 · 8f5c255 · 8f5c255
2 parents ccdc853 + 2c2c0d4
commit 8f5c255
Show file tree

Hide file tree

Showing 24 changed files with 325 additions and 483 deletions.
diff --git a/.idea/dataSources.xml b/.idea/dataSources.xml
diff --git a/documentation/3-context-status-arc42-org4canvas.drawio.png b/documentation/3-context-status-arc42-org4canvas.drawio.png
diff --git a/documentation/5-building-blocks-status-arc42-org.drawio b/documentation/5-building-blocks-status-arc42-org.drawio
diff --git a/documentation/5-building-blocks-status-arc42-org.drawio.png b/documentation/5-building-blocks-status-arc42-org.drawio.png
diff --git a/documentation/adrs/0011-rate-limiter-with-persistently-stored-last-query-time.md b/documentation/adrs/0011-rate-limiter-with-persistently-stored-last-query-time.md
@@ -1,10 +1,11 @@
 # 11. rate-limiter with persistently stored last-query-time
 
 Date: 2023-12-29
+modified: 2024-01-25
 
 ## Status
 
-Accepted
+Accepted.
 
 ## Context
 
@@ -23,10 +24,10 @@ If our current server is a "fresh instance", we have to ignore the last-time-cal
 
 ## Decision
 
-1. Create a table that keeps the invocation times of our API
+1. Create a table that keeps the invocation times of **our** API
 2. Create tables that keep the last-query time for both Plausible and GitHub requests.
-2. Call the Plausible.io API only once every `Plausible_Rate_Limit_Minutes` (defaulting to 20)
-3. Call the GitHub API only once every `GitHub_Rate_Limit_Minutes` (defaulting to 3)
+2. Call the Plausible.io API only once every `plausible_Rate_Limit_Minutes` (defaulting to 20)
+3. Call the GitHub API only once every `gitHub_Rate_Limit_Minutes` (defaulting to 3)
 
 The tables shall have the following format:
 

diff --git a/documentation/arc42/chapters/03_system_scope_and_context.adoc b/documentation/arc42/chapters/03_system_scope_and_context.adoc
@@ -9,28 +9,34 @@ ifndef::imagesdir[:imagesdir: ../../images]
 :toc:
 
 
-
 [[section-system-scope-and-context]]
 == System Scope and Context
 
+image::03-context-status-arc42-org4canvas.drawio.png[]
 
+[cols="1,3,1"]
+|===
+| Element | Responsibility | Code
 
+| Plausible.io
+| Web analytics SAAS platform, counts viewers and pageviews
+|`/internal/plausible`
 
+| Fly.io
+| Cloud (hyperscaler) platform, where the status.arc42.org application is deployed and executed. Besides deploying the golang application on their servers, We ask fly.io for the current server region, utilizing their https://fly.io/docs/reference/[API].
+|`/internal/fly`
 
-=== Business Context
-
-
-
-**<Diagram or Table>**
-
-**<optionally: Explanation of external domain interfaces>**
-
-=== Technical Context
-
+| GitHub
+| Hosting all repositories for arc42 sites and subdomains. API returns bug and issue count plus other info related to these repositories.
+| `/internal/github`
 
+| Slack
+| Notification service: When users request certain actions in the status app, notifications are sent to a specific Slack channel/app.
+| currently in planning
 
-**<Diagram or Table>**
+| Turso Database
+| A cloud-hosted, multi-instance database storing several types of runtime data we need to persist. For example: startup times of the app.
+| ìnternal/database`
 
-**<optionally: Explanation of technical interfaces>**
+|===
 
-**<Mapping Input/Output to Channels>**
diff --git a/documentation/arc42/chapters/04_solution_strategy.adoc b/documentation/arc42/chapters/04_solution_strategy.adoc
@@ -13,5 +13,13 @@ ifndef::imagesdir[:imagesdir: ../../images]
 [[section-solution-strategy]]
 == Solution Strategy
 
+* implement in Golang (facilitates cloud deployment)
+* use https://plausible.io[plausible.io] to collect usage data (commercial service with excellent data privacy, no cookies)
+* static site generator (Jekyll, Github pages) for main site
+* use https://fly.io[fly.io] for cloud deployment
+* use https://turso.tech[turso.tech] for cloud data storage
+
+Fully open-source, source hosted on https://github.com/arc42/status.arc42.org-site[public GitHub repository].
+
 
 
diff --git a/documentation/arc42/chapters/05_building_block_view.adoc b/documentation/arc42/chapters/05_building_block_view.adoc
@@ -19,22 +19,42 @@ ifndef::imagesdir[:imagesdir: ../../images]
 
 === Whitebox Overall System
 
+image::05-building-blocks-status-arc42-org.drawio.png[]
 
 
-_**<Overview Diagram>**_
+[cols="1,3,1"]
+|===
+| Element | Responsibility | Code
 
-Motivation::
+| main
+| Golang requires a "main" func as entry point in the application.
+|
 
-_<text explanation>_
+| api gateway
+| an http server with several predefined routes. This server is called from the public website, dispatches to domain and returns either plain html to the static site (or other formats when called via the upcoming technical API)
+|`internal/api`
 
+| domain
+| core functionality, coordination of various subsystems, results collection
+|
 
-Contained Building Blocks::
-_<Description of contained building block (black boxes)>_
+| types
+| a few data types and -structures used by other parts. Extracted from the domain to avoid circular dependencies.
+|
 
-Important Interfaces::
-_<Description of important interfaces>_
+| github
+| wrapper for the public GitHub (graphql) API. We query several repository infos (e.g. nr of open issues, bugs and pull-requests).
+|
 
+|database
+|wrapper for https://turso.tech[Turso], an SQLite clone running in the cloud. We store some usage and operations data, and wanted to avoid hosting our own DB.
+|
 
+|fly.io
+|wrapper for the https://fly.io[fly.io API], which we use to find out the server region where our application is deployed. Not to be confused with our _hosting and deployment_ concept.
+|
+
+|===
 
 
 ==== <Name black box 1>
@@ -54,68 +74,3 @@ _<(Optional) Fulfilled Requirements>_
 _<(optional) Open Issues/Problems/Risks>_
 
 
-
-
-==== <Name black box 2>
-
-_<black box template>_
-
-==== <Name black box n>
-
-_<black box template>_
-
-
-==== <Name interface 1>
-
-...
-
-==== <Name interface m>
-
-
-
-=== Level 2
-
-
-
-==== White Box _<building block 1>_
-
-
-
-_<white box template>_
-
-==== White Box _<building block 2>_
-
-
-_<white box template>_
-
-...
-
-==== White Box _<building block m>_
-
-
-_<white box template>_
-
-
-
-=== Level 3
-
-
-
-
-==== White Box <_building block x.1_>
-
-
-
-
-_<white box template>_
-
-
-==== White Box <_building block x.2_>
-
-_<white box template>_
-
-
-
-==== White Box <_building block y.1_>
-
-_<white box template>_
diff --git a/documentation/concurrent-API-access.puml b/documentation/concurrent-API-access.puml
diff --git a/documentation/images/03-context-status-arc42-org4canvas.drawio.png b/documentation/images/03-context-status-arc42-org4canvas.drawio.png
diff --git a/documentation/images/05-building-blocks-status-arc42-org.drawio.png b/documentation/images/05-building-blocks-status-arc42-org.drawio.png
diff --git a/documentation/images/06-api-call-with-cors.png b/documentation/images/06-api-call-with-cors.png
diff --git a/documentation/images/06-api-call-with-cors.puml b/documentation/images/06-api-call-with-cors.puml
@@ -0,0 +1,29 @@
+@startuml
+'https://plantuml.com/sequence-diagram
+
+autonumber
+
+
+actor client as "Client"
+participant site as "status.arc42.org\nstatic website"
+participant htmx as "htmx\nJavaScript lib"
+participant server as "api Gateway"
+
+client -> site : "https://status.arc42.org"
+site -> htmx: replace table
+activate htmx
+
+htmx -> server: OPTIONS statsTable
+activate server
+server -> server : Set CORS headers
+server --> htmx : return 200 OK
+
+htmx -> server : GET /statsTable
+server -> server : Set CORS headers
+server -> server : Perform additional processing
+server --> htmx : return 200 OK with data
+deactivate server
+htmx --> site: html table\ncontaining results
+deactivate htmx
+site --> client: display table
+@enduml
diff --git a/documentation/images/06-concurrent-API-access.puml b/documentation/images/06-concurrent-API-access.puml
@@ -0,0 +1,34 @@
+@startuml
+'https://plantuml.com/sequence-diagram
+
+autonumber
+
+participant domain
+participant "domain.\nsiteStats" as siteStats
+participant "domain.\nrepoStats" as repoStats
+group parallel [domain.go]
+    domain -> domain : init logger
+    loop for site in allSites
+      par
+        domain -> siteStats: get SiteStatistics(site)
+        activate siteStats
+        par
+            siteStats -> plausible: get 7D( site )
+            else
+            siteStats -> plausible: get 30D( site )
+            else
+            siteStats -> plausible: get 12M( site )
+        end
+        deactivate siteStats
+        else
+        domain -> repoStats: getRepoStatistics(site)
+        activate repoStats
+            repoStats -> github: getRepoStatistics
+            activate github
+            deactivate github
+        deactivate repoStats
+    end
+    domain -> domain: aggregateResults( site )
+
+end
+@enduml
diff --git a/go-app/go.mod b/go-app/go.mod
@@ -7,7 +7,7 @@ require (
 	github.com/rs/zerolog v1.31.0
 	github.com/shurcooL/githubv4 v0.0.0-20230704064427-599ae7bbf278
 	github.com/tursodatabase/libsql-client-go v0.0.0-20231216154754-8383a53d618f
-	golang.org/x/net v0.18.0
+	golang.org/x/net v0.20.0
 	golang.org/x/oauth2 v0.14.0
 	golang.org/x/text v0.14.0
 )
@@ -20,11 +20,14 @@ require (
 	github.com/libsql/sqlite-antlr4-parser v0.0.0-20230802215326-5cb5bb604475 // indirect
 	github.com/mattn/go-colorable v0.1.13 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
+	github.com/mattn/go-sqlite3 v1.14.19 // indirect
 	github.com/shurcooL/graphql v0.0.0-20230722043721-ed46e5a46466 // indirect
 	github.com/valyala/bytebufferpool v1.0.0 // indirect
 	github.com/valyala/fasthttp v1.50.0 // indirect
 	golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e // indirect
-	golang.org/x/sys v0.14.0 // indirect
+	golang.org/x/mod v0.14.0 // indirect
+	golang.org/x/sys v0.16.0 // indirect
+	golang.org/x/tools v0.17.0 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
 	google.golang.org/protobuf v1.31.0 // indirect
 	nhooyr.io/websocket v1.8.7 // indirect

diff --git a/go-app/go.sum b/go-app/go.sum
@@ -60,6 +60,8 @@ github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/
 github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
 github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
 github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
+github.com/mattn/go-sqlite3 v1.14.19 h1:fhGleo2h1p8tVChob4I9HpmVFIAkKGpiukdrgQbWfGI=
+github.com/mattn/go-sqlite3 v1.14.19/go.mod h1:2eHXhiwb8IkHr+BDWZGa96P6+rkvnG63S2DGjv9HUNg=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
 github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -93,11 +95,15 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk
 golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8=
 golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e h1:+WEEuIdZHnUeJJmEUjyYC2gfUMj69yZXw17EnHg/otA=
 golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e/go.mod h1:Kr81I6Kryrl9sr8s2FK3vxD90NdsKWRuOIl2O4CvYbA=
+golang.org/x/mod v0.14.0 h1:dGoOF9QVLYng8IHTm7BAyWqCqSheQ5pYWGhzW00YJr0=
+golang.org/x/mod v0.14.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
 golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20210510120150-4163338589ed/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.18.0 h1:mIYleuAkSbHh0tCv7RvjL3F6ZVbLjq4+R7zbOn3Kokg=
 golang.org/x/net v0.18.0/go.mod h1:/czyP5RqHAH4odGYxBJ1qz0+CE5WZ+2j1YgoEo8F2jQ=
+golang.org/x/net v0.20.0 h1:aCL9BSgETF1k+blQaYUBx9hJ9LOGP3gAVemcZlf1Kpo=
+golang.org/x/net v0.20.0/go.mod h1:z8BVo6PvndSri0LbOE3hAn0apkU+1YvI6E70E9jsnvY=
 golang.org/x/oauth2 v0.14.0 h1:P0Vrf/2538nmC0H+pEQ3MNFRRnVR7RlqyVw+bvm26z0=
 golang.org/x/oauth2 v0.14.0/go.mod h1:lAtNWgaWfL4cm7j2OV8TxGi9Qb7ECORx8DktCY74OwM=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -110,6 +116,8 @@ golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.14.0 h1:Vz7Qs629MkJkGyHxUlRHizWJRG2j8fbQKjELVSNhy7Q=
 golang.org/x/sys v0.14.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/sys v0.16.0 h1:xWw16ngr6ZMtmxDyKyIgsE93KNKz5HKmMa3b8ALHidU=
+golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
@@ -119,6 +127,8 @@ golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
 golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
 golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.17.0 h1:FvmRgNOcs3kOa+T20R1uhfP9F6HgG2mfxDv1vrx1Htc=
+golang.org/x/tools v0.17.0/go.mod h1:xsh6VxdV005rRVaS6SSAf9oiAqljS7UZUacMZ8Bnsps=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 google.golang.org/appengine v1.6.7 h1:FZR1q0exgwxzPzp/aF+VccGrSfxfPpkBqjIIEq3ru6c=
 google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=