Index 2016 tech architecture discussion

[roll]

Static site

Being a static site is a questionable solution for project with this amount of business logic. But It doesn't look like it's strongly necessary (cost, time) to rewrite it to be a dynamic site based on flask or expressjs for example.

Datapackage as a database

For building the site uses something like datapackage (csv files) as a database. Also this csv files can be downloaded. Here is a biggest area for improvements.

Static site (logic in templates) and csv files as a database are not flexible enough to play together. I suppose datapacakge can be used as a database for a dynamic site but not for static because there is an additional work (logic in templates) to get data from csv files.

Proposed solution is to have more normalized json files as a database:

1. Census expose a rich API (for now the API is almost ready for this purpose with exclusion support)
2. odi download saves json files with all information received from Census:
   • entries.2013.json
   • entries.2014.json
   • places.json
   • etc
3. odi prepare generates json files with all additional data structures needed to be used
   in templates (for example for optimization purposes).
4. odi populate generates templates
5. odi build builds the site (including csv files for downloading)

Logic in templates

After switching database to json files amount of logic in templates has to be sharply reduced. For examples there is a recommendation to remove _scope.html at all as one point of failure, non debuggable code (because of bugs with the trace reporting in Pelican) and a bottle neck in sake of build time.

If additional logic is still be needed it can be written as template filters. But I suppose there has to be a recommendation for future developers to avoid logic in templates as much as possible because it was the most time consuming place (understading, debugging) of Index 2015 work.

template logic -> odi prepare / filters

[roll]

@pwalsh wrote:

I have been thinking about this a bit too, and I came to the following thoughts:

1. The static site idea was great last year, as, the census code itself was at the point of breaking for the use cases it had to meet
2. Since then, the Census site received a ​_major_​ upgrade, and, while still not perfect, it is in pretty good shape to build on
3. Following from (2) - all the logic the static site requires now is in the Census site anyway. And, we have nice static site generator options in Node.js (as you know with your own Mr. Smith). With some planning, we could build the static site directly out of the census codebase.

[roll]

@pwalsh

I also think after some arch redesign it will be able to be a good maintainable static site with fast build time.

Good thing about node.js static site generators - https://www.staticgen.com/ (filter by JavaScript) - for many of them nunjucks (jinja2 clone) can be used.

[pwalsh]

@roll you mean still keeping it a distinct codebase though, right? nunjucks would be good and make porting almost trivial. I'm positive that the build times will be better with node, but also probably as you point out, removing that _scope.html trickery I did last year which is really hard to debug.

[roll]

@pwalsh

you mean still keeping it a distinct codebase though, right?

Yes, of course.

[roll]

My main point about improvements (speed, complexity) is to change data structures passed to templates (there is no difference with pelican or nodejs).

Now we have main things like

data
    places
        name
        value
        value_2014

leads for complex code like this - https://github.com/okfn/opendataindex/blob/master/themes/odi/templates/_scope.html#L17-L47

But having something like this

data
    places
        2015
            name
            value
        2014        
            name
            value

leads to template code just like:

data['places'][prevYear]['value']

etc

So after changing it everything else will be much more easier.

[pwalsh]

@roll +1. Great, thanks for the insights.