You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hey I'm using colly for a personal project of mine, and I'd like to not spam the web page I'm scraping.
I saw there was a CacheDir function to colly, but what happens if the webpage changes? Maybe it has some new data.
How can I set a timeout that a cached page is stale after a day or so, and colly would automatically fetch the current one?
The text was updated successfully, but these errors were encountered:
Hey I'm using colly for a personal project of mine, and I'd like to not spam the web page I'm scraping.
I saw there was a
CacheDirfunction to colly, but what happens if the webpage changes? Maybe it has some new data.How can I set a timeout that a cached page is stale after a day or so, and colly would automatically fetch the current one?
The text was updated successfully, but these errors were encountered: