Postpone Duplicate Submissions Incident - Apr 27, 2023

TL;DR

(We encourage you to read the entire post!)

• A Reddit bug cascaded into Postpone and caused posts to be submitted multiple times.

• We turned off submissions from 23:40 UTC-10 on April 26, 2023 to 5:20 UTC-10 on April 27, 2023 to work on a fix.

• Posts during that window were rescheduled to 10 years from now so that we don’t submit them all together.

• We released an update to ensure Postpone won't submit duplicate posts despite this Reddit bug. Submissions are currently re-enabled.

What happened?

For around 5 hours starting at 23:40 UTC-10 on April 26, 2023, Postpone submitted Imgur link posts to Reddit multiple times (typically 2-3 times) instead of only once.

At 5:20 UTC-10 on April 27, 2023, we turned off all submissions while we worked to figure out what was going wrong.

We discovered that Reddit was returning a “500 error response” for submissions that actually succeeded. A 500 error response means that Reddit’s servers are down and cannot handle the request. Despite receiving this error response, the post actually did successfully submit.

This issue only happened for posts that submitted Imgur links. The issue would happen in both SFW and NSFW subreddits, and was also happening on Reddit.com. For that reason, we think this is a bug in Reddit and that it is not related to Imgur’s recent changes to their terms.

Postpone uses a third party library called PRAW to submit posts. Unfortunately, this library automatically retries any request to Reddit that returns a 500 error response, since a 500 error means that the request entirely failed. This lead to the same post being submitted multiple times.

The Fix

We released a fix around 8:00 UTC-10 on April 27, 2023. The fix disables all retries when we receive a 500 error response, since those can no longer be trusted. Submissions were re-enabled at 8:25 UTC-10 after follow-up testing.

As of that time, Reddit is still returning 500 errors for submissions that include Imgur links. The submissions are succeeding, however. Postpone will no longer attempt to retry these posts. Check r/help and r/bugs for updates on this Reddit issue.

Out of an abundance of caution, we moved all posts that were scheduled during the time that submissions were disabled. We did this to ensure that all scheduled posts would not be submitted at the same time when submissions were re-enabled.

The posts were rescheduled to submit to 10 years later than they were originally scheduled. We did this to avoid deleting the posts, so that you may reschedule them if you’d like. You may delete these posts, or click the Edit button to reschedule them to a date/time in the future.

Why did it take so long to fix?

The timing of this bug is very unfortunate, as I’m currently in Hawaii with my wife on our “babymoon.” I am in a different timezone, 5 hours earlier than my usual timezone, otherwise I could have caught this quickly. I apologize for the unfortunate timing.

Next steps

We sincerely apologize that this Reddit bug cascaded to Postpone, as well. Postpone should be able to trust that when Reddit says “this post failed to submit” that it really did fail to submit, but unfortunately that seems to no longer be the case.

If your account was banned from a subreddit then please reach out to subreddit moderators requesting that they reverse the ban. We’ve heard that many subreddit moderators are aware of this issue and are reversing automatic bans because of it.

We are reaching out to Reddit to better understand what happened. We are also putting additional guardrails in place to ensure that this can’t happen again, and working on improving error alerting so that we can respond to major incidents like these more quickly.

Please reach out via Postpone chat or email if you have any follow up questions we can help address.

Cheers,

Grant, Creator of Postpone