duplicates every run

[qrenz]

Hello,

I work at a school and have been using this script to sync some internal calendars for our security team. Recently it seems that every time the calendar syncs it creates duplicates of all of the events in them. I read that this could be caused by not giving enough time in between syncs so i enabled future only, updated our script, and set our frequency to 20 minutes. I even redeployed the entire script and used a fresh calendar. I've attached a screenshot of it after just the initial installation and waiting 20 minutes. Any help with this would be greatly appreciated.

I have one failed run, the log looks like this at the end

Jan 3, 2022, 2:26:19 PM Info Adding new event MLS-il27-1aa49c77-47de-4835-a7d6-242919f27bc5
Jan 3, 2022, 2:26:19 PM Info Timezone Z unsupported!
Jan 3, 2022, 2:26:19 PM Info Using Timezone Etc/GMT!
Jan 3, 2022, 2:26:19 PM Info Adding new event MLS-il27-7a10dadc-07d0-438e-a104-30d5fe2a319d
Jan 3, 2022, 2:26:19 PM Info Error, Retrying... [GoogleJsonResponseException: API call to calendar.events.insert failed with error: Rate Limit Exceeded]
Jan 3, 2022, 2:26:43 PM Error We're sorry, a server error occurred. Please wait a bit and try again.
Jan 3, 2022, 2:26:54 PM Info Timezone Z unsupported!

The timezone error comes up after every event. After this when it eventually runs again it starts then creates duplicate events for everything and eventually gets to

Jan 3, 2022, 2:56:30 PM Info Error, Retrying... [GoogleJsonResponseException: API call to calendar.events.insert failed with error: Rate Limit Exceeded]
Jan 3, 2022, 2:56:31 PM Info Error, Retrying... [GoogleJsonResponseException: API call to calendar.events.insert failed with error: Rate Limit Exceeded]
Jan 3, 2022, 2:56:31 PM Info Error, Retrying... [GoogleJsonResponseException: API call to calendar.events.insert failed with error: Rate Limit Exceeded]
Jan 3, 2022, 2:56:32 PM Info Error, Retrying... [GoogleJsonResponseException: API call to calendar.events.insert failed with error: Rate Limit Exceeded]
Jan 3, 2022, 2:56:34 PM Info Error, Retrying... [GoogleJsonResponseException: API call to calendar.events.insert failed with error: Rate Limit Exceeded]
Jan 3, 2022, 2:56:38 PM Info Error, Retrying... [GoogleJsonResponseException: API call to calendar.events.insert failed with error: Rate Limit Exceeded]
Jan 3, 2022, 2:56:44 PM Info Error, Retrying... [GoogleJsonResponseException: API call to calendar.events.insert failed with error: Rate Limit Exceeded]
Jan 3, 2022, 2:56:50 PM Error Exceeded maximum execution time

[qrenz]

Update, ran on a fresh calendar today after uninstalling for the rest of the day yesterday to hopefully skirt rate limitations. First run ran beautifully, finishes in 919 seconds. Second run (after 30 minutes this time) runs much longer and starts getting rate limit errors while creating duplicates.

[jonas0b1011001]

I suspect your calendar is either missing event-ids completely or changing them each time the script is accessing it. This would explain why the script can not match events in gcal with events from the ics, therefore creating new ones on each run. As it times out on the second run it never gets to the end where it would delete the old versions of the events.

Could you please share the content of the ics you are syncing
(you can hide personal information if you wish using https://derekantrican.github.io/icssanitizer.html)

[qrenz]

Weirdly enough, the link wont even run through the sanitizer.

Here's the link we're using (generated by a software called MLSchedules)
https://il27.mlschedules.com/iCalendar.ashx?subscriptionId=c4a00900-6a11-43a1-bfcc-6772fbbec1e7

[jonas0b1011001]

As expected the event ids are not persistent. In combination with the huge amount of events (2000+) this leads to the rate limiting, script timing out and duplicates not getting removed.

We can't do anything about the rate limiting, therefore the only solution i could offer right now is to further reduce the amount of events getting synced. This could be done by processing only events between today - 1 and today + 30 (for example), would that be suitable for your use case?

[qrenz]

I think that would work! How would I go about doing that?

Would another solution be to ask them to use persistent event id's? I appreciate your replies.

[jonas0b1011001]

Would another solution be to ask them to use persistent event id's?

That would be the best solution for sure. However i don't know if they will be able to change that/how much time they need to do that.

I think that would work! How would I go about doing that?

Try this version of the script.
You actually have two new options in the settings part.

var ignoreEventIDs
Will ignore the original ids and replace them with persistent ones. This should allow the script to properly match existing/new events and skip unchanged ones.
Try running the script with only this setting true for some time.
If you run into rate limiting again, set limitSyncRange to true, this will filter events older than 1 day or 14 days in the future, reducing the overall amount of events that need to be processed (the timeframe can be adjusted in code.gs line 143+144).

[qrenz]

This resolved our issues without even using limitSyncRange. We might turn that on just incase though. Thank you!

However, i've been still using onlyFutureEvents. If that is set to false I will get an error that says

Jan 7, 2022, 11:33:51 AM Error GoogleJsonResponseException: API call to calendar.events.insert failed with error: The specified time range is empty.
at unknown function
at callWithBackoff(Helpers:932:16)
at processEvent(Helpers:254:20)
at unknown function
at startSync(Code:217:15)

That's not too big of an issue because it happens on a past event, but somewhat worried about that happening with a new one.

[jonas0b1011001]

The following event is causing the error

BEGIN:VEVENT
DTSTART:20211210T192500Z
DTEND:20211210T090500Z
DTSTAMP:20211123T171700Z
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:xxx
LOCATION:xxx
DESCRIPTION:xxx
UID:xxx
CLASS:PUBLIC
TRANSP:OPAQUE
END:VEVENT

It is supposed to end roughly 10hrs before it starts. Google API understandably does not accept this.

I updated the script from above to recognize and correct this error by switching start/end times. The link above is still valid.

[qrenz]

The following event is causing the error

BEGIN:VEVENT
DTSTART:20211210T192500Z
DTEND:20211210T090500Z
DTSTAMP:20211123T171700Z
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:xxx
LOCATION:xxx
DESCRIPTION:xxx
UID:xxx
CLASS:PUBLIC
TRANSP:OPAQUE
END:VEVENT

It is supposed to end roughly 10hrs before it starts. Google API understandably does not accept this.

I updated the script from above to recognize and correct this error by switching start/end times. The link above is still valid.

This works thank you so much!

[Lonestarjeepin]

Duplicate of #283