Scaling stencil: support fetching templates from an object storage

[gmile]

Is your feature request related to a problem? Please describe.

We're looking at potentially scaling stencil deployment. It so happens, that right now stencil reads .docx templates from a disk on the file system. In pursue of scaling stencil deployment (e.g. have multiple copies of stencil running), we need to find a way to make sure all instances of stencil "see" the same set of .docx templates.

While I am able to think of a handful of workarounds, each with different trade-offs, it would be great for stencil to natively support fetching templates from an S3-compatible object storage.

Describe the solution you'd like

Ideally, we would like to be able to configure stencil to read templates from a Google Cloud Storage bucket. To sum up:

• we would be able to configure a bucket name,
• stencil application would find and read service account credentials (largely following Google Cloud guidelines),
• finally, stencil would make an API call to (recursively) fetch all templates provided in a bucket.

While our deployment is in Google Cloud, and we would very much see support of Google Cloud Storage (a largely S3-compatible storage) and IF this proposal will gain interest, it may be a good idea to consider a code design, such that allows for other S3-compatible storages too (AWS S3, Minio, OpenStack Swift, etc.).

Describe alternatives you've considered

As a workaround, we've used an object bucket in conjunction with an utility to mount bucket as a disk volume, for stencil to consume. Unfortunately, this solution, after working fine for quite some time, proved to be limited and not scale too well with the growing amount of templates.

Another idea that comes to mind: stencil supporting template content being part of the POST request to render a template. This way templates don't have to be present on disk at all for stencil to consume. This approach hasn't been thought through, e.g.

• would it make sense for stencil to cache the template it received in POST request's payload? if yes, when the cache should be invalidated?,
• what would be the shape & content type of POST request's payload should be (e.g. JSON with file contents being base64-ed, a form content-type, etc.),
• this may have network load implications, but this is surely out of scope for stencil to consider.

Additional context

None.