Syndication endpoint not working

[sentience]

Describe the bug

The default syndication endpoint does not appear to be working.

To reproduce

Steps to reproduce the behaviour:

1. Set up Indiekit with this configuration.
2. Log into the web admin.
3. Visit https://indiekit.kevinyank.com/syndicate (Alternatively, visit /syndicate?token=XXXX with a valid token.)
4. Get a 404 error.

Expected behaviour

Indiekit should respond by syndicating outstanding posts.

Environment

• Indiekit version: v1.0.0-beta.1
• Content store URL: https://github.com/sentience/kevinyank.com
• Publication URL: https://kevinyank.com

[sentience]

Figured it out from the source code: the endpoint only supports POST.

Not sure why my Netlify web hook didn't work initially, but when I replaced the token with a new one that I verified manually, it all seems to be working now!

[paulrobertlloyd]

Glad you got it working. Netlify’s webhook has stopped working for me recently, even after updating the token. As far as I can tell it’s a Netlify thing as making a POST request from another app works as expected (as do the syndication buttons in the UI). Do let me know if you come up against the same issue, as it has me mystified!

[sentience]

Actually it's not working for me now either. Netlify's UI is no longer complaining of an erroring deploy hook, but my post didn't get auto-syndicated, so I don't know if Netlify actually hit the endpoint.

[sentience]

Some relevant findings on #569 as I seek to understand why requests sent from my RapidAPI desktop app cause syndication to occur, but Netlify-sent requests do not.

[paulrobertlloyd]

I think I’ve worked out why this is not working.

Originally, prior to launching the first beta, the syndication endpoint only accepted url and token values from a provided query string (response.query). When I added support for syndicating via the interface, I allowed url and token properties to be provided in a form body (response.body).

However, the ping from Netlify includes a body, and one of its values is url. This is the URL of the site on Netlify (e.g. http://indiekit-sandbox.netlify.app), not the URL to be syndicated. As you can either syndicate the last most recently un-syndicated post, OR syndicate a particular URL, this url value takes precedence.

Two options for resolving:

1. Have the endpoint only accept parameters via a query string
2. Rename the url property (and possibly the token property) to something more unique.

(There’s also a third option, workout how to can create a JWS secret token to be sent with the request, which Netlify webhooks can provide. This way the properties would be encoded within a secret token, not query or body parameters. This may be a bit too Netlify-specific however, reducing the usefulness of this endpoint.)

Think I’m tempted to go for option 1, especially as this is the method that worked for a good many months previous.

[paulrobertlloyd]

Actually…

Option 2. Being able to submit via form is a useful feature, and using query strings either requires more workarounds than ideal, or is just plain icky. Instead, look either for request.query.url or request.body.syndication.url.

Also… I think it might be a case of just adding the X-Webhook-Signature header to the list of places we look for an access token. That would mean for that a Netlify webhook, you would add the syndication endpoint URL for ‘URL to notify’, and your server’s access token for ‘JWS secret token (optional)’, which makes much more sense.

[sentience]

Could it be possible to move the endpoint for the form submission to a different path, and have the documented /syndicate endpoint for the “deploy hook” functionality remain “dumb”, only taking an auth token?

…

On 4 Jan 2023 at 08:47 +1100, Paul Robert Lloyd ***@***.***>, wrote: Actually… Option 2. Being able to submit via form is a useful feature, and requires more workarounds than ideal. Instead, look either for request.query.url or request.body.syndication.url. Also… I think it might be a case of just adding the X-Webhook-Signature header to the list of places we look for an access token. That would mean for that a Netlify webhook, you would add the syndication endpoint URL for ‘URL to notify’, and your server’s access token for ‘JWS secret token (optional)’, which makes much more sense. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[sentience]

I like the idea of putting the auth token in a header instead (so it doesn't end up in as many logs), but maybe you should continue to support both for people who may need to integrate with a service that doesn't support setting request headers?

[paulrobertlloyd]

I like the idea of putting the auth token in a header instead (so it doesn't end up in as many logs), but maybe you should continue to support both for people who may need to integrate with a service that doesn't support setting request headers?

Take a look at a61ae7f; I’ve added X-Webhook-Signature to the list of places we look for an access token. Just going to test this now to see if this works as intended.

[paulrobertlloyd]

Could it be possible to move the endpoint for the form submission to a different path, and have the documented /syndicate endpoint for the “deploy hook” functionality remain “dumb”, only taking an auth token?

I think its fine for this endpoint to accept values via query string or form body (after all, this is how the specified Micropub endpoint works, so in that sense its consistent).

Given the syndication endpoint requires authorisation before it can be accessed, and authorised requests are interrogated for an access token in (now) 5 different locations, the documentation for this endpoint should list the places a token can be provided, rather than only mention the query string option. UPDATE: turns out the access token discovered during the authorisation flow isn’t used to authorise the subsequent Micropub request, which is silly.

[paulrobertlloyd]

So far, I’ve fixed the following:

• Updated the syndication endpoint such that form body values don’t collide with values sent in Netlify’s webhook POST request body
• Updated the authorisation flow to follow the same pattern used elsewhere (that is, given the access token discovered during the authorisation flow is saved to the session, use this stored value in the subsequent Micropub request rather than attempt discovery a second time)
• Provide additional support for Netlify by updating Indiekit’s token search to check if the X-Webhook-Signature header contains a token

And yet still, the syndication endpoint is failing (not helped by me tweaking too many different variables here). I get a notice that the webhook failed after 6 attempts due to a 400/500 error, so I’m guessing authorisation is still failing somewhere (maybe the session gets reset). I’ll need to log the request chain to see what’s happening - perhaps the server’s access token should be stored in the database, rather than memory.

I’ll return to this again tomorrow. 🤪

[paulrobertlloyd]

(Narrator: X-Webhook-Signature didn’t work like Paul thought it did. See #571)

[paulrobertlloyd]

Think this should be largely fixed, but with the proviso that #575 could cause potential issues for any existing posts in your database.

Fixing #575 would delay releasing the next beta, so think I’ll release the latest changes in Beta 2, and come back to the issue with data falling out of sync later.

[sentience]

Looking forward to giving Beta 2 a spin with these changes! I'll work around #575 for now by fixing the out-of-sync data in MongoDB manually.