github, improve asset classification when release.json not present (#437

) * github-api, handle case where one unclassified asset can be classified as retail. * review feedback * updated constants for The War Within * updated constants for the release date of previous expansion * CHANGELOG
ogri-la · Sep 5, 2024 · 677836d · 677836d
1 parent 8c4ab7e
commit 677836d
Show file tree

Hide file tree

Showing 6 changed files with 276 additions and 17 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,12 +9,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 
 * error handling for a few cases where uncaught exceptions were being swallowed.
+* added another heuristic for guessing which asset in a Github release supports retail.
+    - in this case, if there is a single remaining unclassified asset, and there are other classified assets and nothing has been classified as retail yet, classify that asset as retail.
+        - yes, this is guessing, and it won't always be correct, but it's probably true most of the time.
 
 ### Changed
 
 * installing an addon by file now overwrites/updates ignored addons and unpins pinned addons.
     - this accommodates a use case where Strongbox is used to install a select few addons manually from unsupported sources (like Curseforge) but then the new addons match against older versions in the catalogue from wowi.
     - please open a feature request if you want this behaviour adjustable.
+* updated internal constants for The War Within.
+* updated internal constant for the date of the previous expansion.
+    - affects how much the emergency 'short' catalogue gets truncated from the 'full' catalogue.
+        - emergency catalogue is used when Github hosted catalogue is unavailable, which is rare.
 
 ### Fixed
 

diff --git a/TODO.md b/TODO.md
@@ -6,8 +6,6 @@ see CHANGELOG.md for a more formal list of changes by release
 
 ## done
 
-## todo
-
 * issue 433: preserve ignore and pin flags when addon updated manually via strongbox
     - usecase: 
         - old addon exists on wowi
@@ -17,11 +15,21 @@ see CHANGELOG.md for a more formal list of changes by release
     - problems:
         - if the addon was ignored, it shouldn't have been possible to overwrite it to begin with
         - it seems reasonable to update addons in this way
+    - done
 
 bug:
     - attempting to overwrite ignored addon results in "refusing to delete ignored addon: /path/to/addon/dir" with no addon name
     - it looks like it installed it anyway
-
+    - done
+
+* import addon from github 
+    - getting a 'no retail release' when there is a clear retail rfelease available
+    - done
+
+* update constants for war within
+    - done
+
+## todo
 
 * Desktop Entry Name should not include "(Flatpak)" 
     - https://github.com/flathub/la.ogri.strongbox/issues/7

diff --git a/src/strongbox/constants.clj b/src/strongbox/constants.clj
@@ -2,11 +2,10 @@
 
 (def mascot "ᕙ[°▿°]ᕗ")
 
-;; 
 (def release-of-previous-expansion
-  "'Shadowlands', released November 23rd 2020. Used to shorten the 'full' catalogue.
+  "'Dragonflight', released November 28th/29th 2022. Used to shorten the 'full' catalogue.
   https://warcraft.wiki.gg/wiki/Expansion#World_of_Warcraft"
-  "2020-10-23T00:00:00Z")
+  "2022-11-28T00:00:00Z")
 
 (def release-of-wow-classic
   "the date wow classic went live (addon development may have started before that). Used to guess possible game tracks when it's ambiguous.
@@ -17,7 +16,7 @@
 ;; used as a placeholder for an addon's supported version when we're forced to guess.
 ;; don't fret too much about patch versions. These values don't affect much.
 ;; https://warcraft.wiki.gg/wiki/Public_client_builds
-(def latest-retail-game-version "10.0.0")
+(def latest-retail-game-version "11.0.0")
 (def latest-classic-game-version "1.14.3")
 (def latest-classic-tbc-game-version "2.5.4")
 (def latest-classic-wotlk-game-version "3.4.0")
@@ -26,7 +25,7 @@
 ;; interface version to use if .toc file is missing one.
 ;; assume addon is compatible with the most recent version of retail (see above).
 ;; these values need to match the latest-* values above.
-(def default-interface-version 100000)
+(def default-interface-version 110000)
 (def default-interface-version-classic 11400)
 
 ;; take all of the game tracks to the right of your position

diff --git a/src/strongbox/github_api.clj b/src/strongbox/github_api.clj
@@ -101,15 +101,41 @@
                           unclassified-assets (->> asset-list (map :game-track) (filter nil?))
                           classified-assets (->> asset-list (map :game-track) (remove nil?) set)
                           diff (clojure.set/difference sp/game-tracks classified-assets)] ;; #{:classic :classic-bc :retail} #{:classic :classic-bc} => #{:retail}
-                      (if (and (= (count unclassified-assets) 1)
-                               (= (count diff) 1))
-                        (->> asset-list
-                             (mapv #(if (nil? (:game-track %))
-                                      (assoc % :game-track (first diff))
-                                      %)))
-                        asset-list)))
-
-        ;; finally, try downloading the release.json file, if it exists, otherwise just use 'all' game tracks originally detected.
+
+                      (if (or (= (count diff) 0) ;; addon covers all game tracks!
+                              (> (count unclassified-assets) 1)) ;; too many unclassified for this logic
+                        asset-list
+
+                        (cond
+                          ;; best case: 1 unclassified asset and exactly 1 available game track.
+                          ;; 2024-09-01: lots of addons still support the full set of classic versions,
+                          ;; but I've noticed many addons dropping tbc in favour of wotlk, then dropping wotlk in favour of cata.
+                          ;; this cond wouldn't fit those.
+                          (= (count diff) 1)
+                          (->> asset-list
+                               (mapv #(if (nil? (:game-track %))
+                                        (assoc % :game-track (first diff))
+                                        %)))
+
+                          ;; next case: 1 unclassified asset, multiple available game tracks and 
+                          ;; *if* we have no retail asset thus far and *if* we have 1 or more assets classified as classic,
+                          ;; assume addon only supports some classic game tracks and asset is retail.
+                          ;; it is a common case and not a huge assumption but it *is* possible that
+                          ;; the addon *doesn't* support retail and only supports some classic game tracks and
+                          ;; we failed to guess a game track. would love to see a real world example here.
+                          (and (> (count diff) 1) ;; many possible game tracks available...
+                               (utils/in? :retail diff) ;; ... and retail is among them.
+                               (>= (count classified-assets) 1)) ;; we've already classified at least one other asset as not-retail.
+                          (->> asset-list
+                               (mapv #(if (nil? (:game-track %))
+                                        (assoc % :game-track :retail)
+                                        %)))
+
+                          ;; we tried but no luck. return what we were given.
+                          :else asset-list))))
+
+        ;; finally, try downloading the release.json file if it exists.
+        ;; this is not an api call but it is 1 of N http requests for N releases.
         classify3 (fn [asset]
                     (if (:game-track asset)
                       [asset]
@@ -158,8 +184,13 @@
   returns the list of releases appropriate for the given `game-track`."
   [release-list vector?, addon :addon/expandable, game-track ::sp/game-track]
   (let [game-track-list (cond
+                          ;; did the catalogue specify a list of supported game tracks?
                           (not (empty? (:game-track-list addon))) (:game-track-list addon)
+
+                          ;; if not, was the addon installed with a specific game track?
                           (:installed-game-track addon) [(:installed-game-track addon)]
+
+                          ;; if not, ...
                           :else [])]
     (->> release-list
          (remove :prerelease)

diff --git a/test/fixtures/github-repo-releases--auctioneer.json b/test/fixtures/github-repo-releases--auctioneer.json
@@ -0,0 +1,178 @@
+[
+  {
+    "url": "https://api.github.com/repos/curseforge-mirror/auctioneer/releases/172913175",
+    "assets_url": "https://api.github.com/repos/curseforge-mirror/auctioneer/releases/172913175/assets",
+    "upload_url": "https://uploads.github.com/repos/curseforge-mirror/auctioneer/releases/172913175/assets{?name,label}",
+    "html_url": "https://github.com/curseforge-mirror/auctioneer/releases/tag/v2024.09.01.00.28",
+    "id": 172913175,
+    "author": {
+      "login": "github-actions[bot]",
+      "id": 41898282,
+      "node_id": "MDM6Qm90NDE4OTgyODI=",
+      "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
+      "gravatar_id": "",
+      "url": "https://api.github.com/users/github-actions%5Bbot%5D",
+      "html_url": "https://github.com/apps/github-actions",
+      "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
+      "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
+      "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
+      "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
+      "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
+      "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
+      "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
+      "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
+      "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
+      "type": "Bot",
+      "site_admin": false
+    },
+    "node_id": "RE_kwDOHyh6Ls4KTnIX",
+    "tag_name": "v2024.09.01.00.28",
+    "target_commitish": "master",
+    "name": "v2024.09.01.00.28",
+    "draft": false,
+    "prerelease": false,
+    "created_at": "2023-11-05T01:13:20Z",
+    "published_at": "2024-09-01T00:28:14Z",
+    "assets": [
+      {
+        "url": "https://api.github.com/repos/curseforge-mirror/auctioneer/releases/assets/189629546",
+        "id": 189629546,
+        "node_id": "RA_kwDOHyh6Ls4LTYRq",
+        "name": "Auctioneer.11.x.BETA.4.zip",
+        "label": "",
+        "uploader": {
+          "login": "github-actions[bot]",
+          "id": 41898282,
+          "node_id": "MDM6Qm90NDE4OTgyODI=",
+          "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
+          "gravatar_id": "",
+          "url": "https://api.github.com/users/github-actions%5Bbot%5D",
+          "html_url": "https://github.com/apps/github-actions",
+          "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
+          "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
+          "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
+          "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
+          "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
+          "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
+          "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
+          "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
+          "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
+          "type": "Bot",
+          "site_admin": false
+        },
+        "content_type": "application/zip",
+        "state": "uploaded",
+        "size": 2440507,
+        "download_count": 0,
+        "created_at": "2024-09-01T00:28:16Z",
+        "updated_at": "2024-09-01T00:28:16Z",
+        "browser_download_url": "https://github.com/curseforge-mirror/auctioneer/releases/download/v2024.09.01.00.28/Auctioneer.11.x.BETA.4.zip"
+      },
+      {
+        "url": "https://api.github.com/repos/curseforge-mirror/auctioneer/releases/assets/189629544",
+        "id": 189629544,
+        "node_id": "RA_kwDOHyh6Ls4LTYRo",
+        "name": "AuctioneerSuite-2.5.6774-bc.zip",
+        "label": "",
+        "uploader": {
+          "login": "github-actions[bot]",
+          "id": 41898282,
+          "node_id": "MDM6Qm90NDE4OTgyODI=",
+          "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
+          "gravatar_id": "",
+          "url": "https://api.github.com/users/github-actions%5Bbot%5D",
+          "html_url": "https://github.com/apps/github-actions",
+          "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
+          "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
+          "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
+          "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
+          "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
+          "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
+          "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
+          "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
+          "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
+          "type": "Bot",
+          "site_admin": false
+        },
+        "content_type": "application/zip",
+        "state": "uploaded",
+        "size": 3787292,
+        "download_count": 0,
+        "created_at": "2024-09-01T00:28:15Z",
+        "updated_at": "2024-09-01T00:28:16Z",
+        "browser_download_url": "https://github.com/curseforge-mirror/auctioneer/releases/download/v2024.09.01.00.28/AuctioneerSuite-2.5.6774-bc.zip"
+      },
+      {
+        "url": "https://api.github.com/repos/curseforge-mirror/auctioneer/releases/assets/189629541",
+        "id": 189629541,
+        "node_id": "RA_kwDOHyh6Ls4LTYRl",
+        "name": "AuctioneerSuite-3.4.6988-wrath.zip",
+        "label": "",
+        "uploader": {
+          "login": "github-actions[bot]",
+          "id": 41898282,
+          "node_id": "MDM6Qm90NDE4OTgyODI=",
+          "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
+          "gravatar_id": "",
+          "url": "https://api.github.com/users/github-actions%5Bbot%5D",
+          "html_url": "https://github.com/apps/github-actions",
+          "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
+          "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
+          "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
+          "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
+          "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
+          "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
+          "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
+          "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
+          "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
+          "type": "Bot",
+          "site_admin": false
+        },
+        "content_type": "application/zip",
+        "state": "uploaded",
+        "size": 3867469,
+        "download_count": 0,
+        "created_at": "2024-09-01T00:28:15Z",
+        "updated_at": "2024-09-01T00:28:15Z",
+        "browser_download_url": "https://github.com/curseforge-mirror/auctioneer/releases/download/v2024.09.01.00.28/AuctioneerSuite-3.4.6988-wrath.zip"
+      },
+      {
+        "url": "https://api.github.com/repos/curseforge-mirror/auctioneer/releases/assets/189629540",
+        "id": 189629540,
+        "node_id": "RA_kwDOHyh6Ls4LTYRk",
+        "name": "AuctioneerSuite-4.4.6991-classic.zip",
+        "label": "",
+        "uploader": {
+          "login": "github-actions[bot]",
+          "id": 41898282,
+          "node_id": "MDM6Qm90NDE4OTgyODI=",
+          "avatar_url": "https://avatars.githubusercontent.com/in/15368?v=4",
+          "gravatar_id": "",
+          "url": "https://api.github.com/users/github-actions%5Bbot%5D",
+          "html_url": "https://github.com/apps/github-actions",
+          "followers_url": "https://api.github.com/users/github-actions%5Bbot%5D/followers",
+          "following_url": "https://api.github.com/users/github-actions%5Bbot%5D/following{/other_user}",
+          "gists_url": "https://api.github.com/users/github-actions%5Bbot%5D/gists{/gist_id}",
+          "starred_url": "https://api.github.com/users/github-actions%5Bbot%5D/starred{/owner}{/repo}",
+          "subscriptions_url": "https://api.github.com/users/github-actions%5Bbot%5D/subscriptions",
+          "organizations_url": "https://api.github.com/users/github-actions%5Bbot%5D/orgs",
+          "repos_url": "https://api.github.com/users/github-actions%5Bbot%5D/repos",
+          "events_url": "https://api.github.com/users/github-actions%5Bbot%5D/events{/privacy}",
+          "received_events_url": "https://api.github.com/users/github-actions%5Bbot%5D/received_events",
+          "type": "Bot",
+          "site_admin": false
+        },
+        "content_type": "application/zip",
+        "state": "uploaded",
+        "size": 3882086,
+        "download_count": 0,
+        "created_at": "2024-09-01T00:28:14Z",
+        "updated_at": "2024-09-01T00:28:15Z",
+        "browser_download_url": "https://github.com/curseforge-mirror/auctioneer/releases/download/v2024.09.01.00.28/AuctioneerSuite-4.4.6991-classic.zip"
+      }
+    ],
+    "tarball_url": "https://api.github.com/repos/curseforge-mirror/auctioneer/tarball/v2024.09.01.00.28",
+    "zipball_url": "https://api.github.com/repos/curseforge-mirror/auctioneer/zipball/v2024.09.01.00.28",
+    "body": "# Auctioneer Mirror\n\nThis is a mirror of MentalPower's Auctioneer\n\n- [Curseforge URL](https://www.curseforge.com/wow/addons/auctioneer)\n\n----\n\nTo open a ticket related to this repository, please do so on [this repository](https://github.com/curseforge-mirror/.github)"
+  }
+]
diff --git a/test/strongbox/github_api_test.clj b/test/strongbox/github_api_test.clj
@@ -605,3 +605,39 @@
                [{:source-id "foo/bar"} "https://github.com/foo/bar"]]]
     (doseq [[given expected] cases]
       (is (= expected (github-api/make-url given))))))
+
+;; ---
+
+(deftest auctioneer
+  (let [addon-summary
+        {:url "https://github.com/curseforge-mirror/auctioneer"
+         :updated-date "2019-10-09T17:40:04Z"
+         :source "github"
+         :source-id "curseforge-mirror/auctioneer"
+         :label "Auctioneer"
+         :name "auctioneer"
+         :download-count 30946
+         :game-track-list [] ;; 'no game tracks'
+         :tag-list []}
+
+        fixture (slurp-fixture "github-repo-releases--auctioneer.json")
+        cases {:retail [{:download-url "https://github.com/curseforge-mirror/auctioneer/releases/download/v2024.09.01.00.28/Auctioneer.11.x.BETA.4.zip",
+                         :game-track :retail
+                         :version "v2024.09.01.00.28"}]
+
+               :classic [{:download-url "https://github.com/curseforge-mirror/auctioneer/releases/download/v2024.09.01.00.28/AuctioneerSuite-4.4.6991-classic.zip",
+                          :game-track :classic
+                          :version "v2024.09.01.00.28"}]
+
+               :classic-tbc [{:download-url "https://github.com/curseforge-mirror/auctioneer/releases/download/v2024.09.01.00.28/AuctioneerSuite-2.5.6774-bc.zip",
+                              :game-track :classic-tbc
+                              :version "v2024.09.01.00.28"}]
+
+               :classic-wotlk [{:download-url "https://github.com/curseforge-mirror/auctioneer/releases/download/v2024.09.01.00.28/AuctioneerSuite-3.4.6988-wrath.zip",
+                                :game-track :classic-wotlk
+                                :version "v2024.09.01.00.28"}]
+               :classic-cata nil}]
+
+    (with-fake-routes-in-isolation {}
+      (doseq [[game-track expected] cases]
+        (is (= expected (github-api/parse-github-release-data fixture addon-summary game-track)))))))