Create a sitemap

[danielcompton]

It would be good to create a sitemap for Clojars to enumerate the many many pages on the site.
More details: https://support.google.com/webmasters/answer/156184?hl=en

[renatoalencar]

This could possibly be a good issue to build, but I'm in doubt on which solution would be better. I thought about three possible solutions.

Just querying the database and rendering a XML

The simplest solution I could think of, but I guess it could a viable vector for DOS attack, since it would query the entire database every time some hits the server.

Querying and rendering with cache

Caching the results on something like an atom or a file, but I don't know how memory it could need giving the size of the database, or how difficult it would be to handle and maintain internal mutable state.

Generating a sitemap.xml from time to time

Calling it from the command line, on a regular period of time. It could be just a simple cron job, but it wouldn't be that much real time as the other alternatives.

Tell me what you think about the solutions above and how could we engage on building this.

[danielcompton]

The last option of generating a sitemap.xml file seems good to me and matches what we do currently for generating data feeds:

clojars-web/src/clojars/tools/generate_feeds.clj

Lines 72 to 88 in 6c7564a

	(defn generate-feeds [dest db s3-bucket]
	(let [feed-file (str dest "/feed.clj.gz")]
	(apply put-files
	s3-bucket
	(write-to-file (full-feed db) feed-file :gzip)
	(write-sums feed-file)))
	(let [poms (pom-list s3-bucket)
	pom-file (str dest "/all-poms.txt")
	gz-file (str pom-file ".gz")]
	(apply put-files
	s3-bucket
	(write-to-file poms pom-file nil println)
	(write-to-file poms gz-file :gzip println)
	(concat
	(write-sums pom-file)
	(write-sums gz-file))))

There's no need for this data to be particularly fresh, every 24 hours seems plenty to me.

[renatoalencar]

There are any pages that shouldn't go on the sitemap? Or there is any nontrivial detail about this, for example, on #482 you argued about using /:key for groups instead of users, since users also have their /user/:key as profile page. I mean, which would be the actual canonical URL for this?

[danielcompton]

I think user/:key makes more sense here and should be easier to generate.

[renatoalencar]

Great, I guess I'll just build what I have in mind first and get back to you later.

[opoku]

@danielcompton @tobias I've put up a PR to address this issue. Let me know if you have any feedback on it whenever you are able to take a look.

[tobias]

Thanks @opoku! I'll take a look once things settle down a bit.

[tobias]

Thanks for the PR @opoku! I merged it, but ran in a test failure locally, and noticed a couple of issues:

• the test looks for sitemap-index.xml, but the file is called sitemap.xml
• the .gz variant of the sitemap has locs for the non-gz files
• we are storing the sitemaps under /repo, but, according to https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-sitemaps?hl=en:

Sitemaps that are referenced in the sitemap index file must be in the same directory as the sitemap index file, or lower in the site hierarchy. For example, if the sitemap index file is at https://example.com/public/sitemap_index.xml, it can only contain sitemaps that are in the same or deeper directory, like https://example.com/public/shared/....

So that's a bigger issue. We may need to generate/store these in a different way. I'll think about it, and would be happy to chat about it tomorrow if you have a few minutes.

[opoku]

Thanks for the feedback. I'll put up some thing to address the test failure. Happy to chat some more tomorrow.

[tobias]

Thanks for the rework @opoku! I merged it and cleaned it up a bit. I then added a location block to our nginx config to serve these. I tested the sitemap with google, and it complained that our robots.txt file disallowed accessing it, but without any other info. I reworked robots.txt a bit, and we'll give it some time to see if it start working.

[tobias]

Proxying the sitemap instead of redirecting to it fixed the issue. This is now working. Thanks @opoku!

[opoku]

@tobias woohoo. Thanks for cleaning this up.