Earlier today, one of our valued clients, Rainmakers, sent out a Shout Campaign to their list of members.  Unfortunately, the shout went out 10 times to the list.

In our old system, it was impossible to blast a shout out 10 times because you have to pay each time before it processes.  If an error happens, the shout doesn’t go out. In our new credit system, its possible to repeatedly send a shout out to a list of numbers if your credit balance is high enough.

Today, our client sent out their first campaign, which started processing.  Due to the length of the message and number of phone numbers, the processing screen took longer than normal, appearing to be frozen.  We failed today by not providing a stronger user interface indicating that the Shout was in progress of being sent.  Because the system appeared to hang, the shout was resent multiple times, and it got delivered multiple times.

What are we doing to fix this from happening again?

1. Provide a stronger User Interface after the campaign has been sent to show progress as the shout campaign is delivered
2. Disable rapid sending of the same message to the same list within 5 minute intervals
3. Ability to cancel a shout as its being delivered

We sincerely apologize for our epic fail today.

Dan, Joel, Darby and Graham

Update- June 28 6:13 pm EST

We found the bug today.  Apparently, our server configuration is setup to sort of ‘autorefresh’ a request after 60 secs.  Essentially, if a request takes too long to process, it sends it to another processor.  The campaign in question took way longer than 60 secs, resulting in multiple posts of the same data.  Here is an excerpt from an awesome post that helped us figure it out (http://www.opensourcery.co.za/2008/05/15/nginx-reading-the-fine-print/):

• nginx sends the request to any one of the mongrels specified in upstream. It will wait for the mongrel to complete and return a response that will be sent back to the client. However, it will not wait forever, it will wait for proxy_read_timeout which defaults to 60 seconds. What happens after 60 seconds. You might be inclined to think that the timeout will be communicated to the client in the form of a nginx error… I did. It appears not.

We have several upgraded fixes planned to prevent this happening again…its going to be a long night of code!