Feature/fetch sync

[KyleMit]

Script to call sessionize and cache json data files as commits to GitHub files
Script is exposed via Netlify Functions via this url
Additionally, add support for Zapier integration which calls the data cache on a daily chron job
Any updates to the data files will trigger a commit, which is automatically picked up by Netlify and published for a build

Alternative multi-step approach which closes #1

[zekefarwell]

@KyleMit, apologies for letting this PR sit for so long. I promise I will review it this week and then we can migrate this same strategy over to the 2019 site to create the speakers and sessions pages.

[KyleMit]

No big timeline on this. Definitely wherever we land here can help inform 2019. I've been meaning to better document both approaches and their tradeoffs. Because this PR definitely provides greater resilience, but at the cost of complexity and infrastructure.

[zekefarwell]

Ok I've read through the code and I think I mostly understand how this works even if I don't completely get all the details. I like this approach a lot! You are right about the extra complexity though.

One thing I would still like is a way to run the fetch data from Sessionize process manually on a local machine without triggering a commit to the repo. That would be pretty useful for testing and debugging. However, it looks like the script fetches data and then commits changed files directly to github without writing to the filesystem. I'm not super familiar with serverless functions yet, but if they don't really have a filesystem then that would make sense. So yeah, differing execution environments makes it a bit tricky. Anyway, before I look into that more I'm curious to hear how you feel about the trade offs of this approach. If the complexity and infrastructure feels worth it to you or if you'd prefer another approach.

Some additional thoughts:

1. Two months from now code camp is over and there's no need to update the 2019 site with new data any more
2. How often is Sessionize data going to change? If it's not that often, manually running the fetch data process and committing changed files as needed might do the job just fine? Or maybe it would be every day and get annoying...
3. If we want to run the fetch data as part of the build process we could make it fault tolerant. We could check if the Sessionize API responds with an error or blank result and skip writing the data files in that case. That way the build would fall back to the latest committed data files and we'd never randomly have blank speakers and sessions pages on the website.

[julielerman]

Sessionize data shouldn't change that much. Even when we are working on the schedule, y ou have to perform an explicit save and I wouldn't want it posting until we were ready with it anyway. After that first time publishing it however, if we are tweaking the schedule (or bios or abstracts) it would be nice for those to auto-publish.

[KyleMit]

I outlined the two high level approaches:

One interesting thing about the JAMStack (JavaScript, API, Markup) stack is that the JavaScript + API seems to refer to client side JS API calls. Which kinda makes sense as JAM isn't opinionated about your build stack, so while eleventy is JS based, there are other SSG engines that use Ruby or Python or Go. Which means there's not a well established best practice for consuming API data during the build process and baking it into the HTML.

I'm leaning toward the second approach though because I think it's too big a hit to have local builds need to hit a fresh API call every time you run npm run serve, and I'd rather own the data that's mission critical for builds in case sessionize ever goes down overnight and we need to push builds.

I'll outline some of the data caching workflows in another comment

[KyleMit]

Here's the various workflows depending on which method we pick

• If we run the local caching process only, we are just manually responsible for pushing sessionize updates.
• If we run the prod caching process only, it'll commit changes immediately against github, so it's tricky to fiddle / debug around with locally.
• We can always have two parallel processes that updates Prod (Github) via a commit via Octokit and also a separate call for updating our local files via node-based file IO.

[zekefarwell]

Thanks for the visuals. Very helpful for thinking about the process. Here's what I think we should do. Define these four npm scripts:

build

• Run npx eleventy (build site from json data files)

serve

• Run npx eleventy --serve

update-data

• fetch Data from Sessionize
• if Sessionize response is valid
  • overwrite json data files with data from sessionizeResponse
• if Sessionize responds with an error or empty arrays
  • leave json data files alone

update-build

• Run update-data
• Run build

In development, we can do npm run build or npm run serve without hitting the Sessionize api. If we know there is fresher data in Sessionize or we just want to check, we can do npm run update-data to pull down the latest, then do a commit if there are any changes.

In production (Netlify), we change the build command from eleventy to npm run update-build. So every production deploy pulls the freshest data from Sessionize, but leaves the json data files alone if the API returns an error, empty response, etc. Then the Zapier job triggers a Netlify deploy on a schedule. So this is essentially a slight modification of the Live Data Access Workflow in your chart.

Once code camp is over, all we have to do to winterize is make sure the committed json data files are up to date and then change the Netlify build command to npm run build so it won't hit the Sessionize API any more.

To me this seems a bit less complex than the serverless function and Octokit method while still meeting our needs, but I think either way will work well for us. What do you think and what have I overlooked? 😃