Fun week at work.

On Wednesday, Product sends out an email to 8k people it wasn’t supposed to after they removed one key filter on who should get that email. This incident causes over 400 new support cases in a 24-hour period as a result. People were confused, and it caused some users to respond and to ask questions completely unrelated. Thanks, Product!

Then this afternoon, my coworkers notice something seems to be wrong with our system and one of our products. We voice weird cases we are seeing to each other. I had one, two other coworkers had one each. Our tingly-Spider sense starts to go off that something is very wrong by these three cases.

I start testing and realize it seems to only be an issue for new users whose program starts Sunday. There are nearly 2k of them, so that’s a big fucking deal to have a product be broken on their first day and not linked correctly.

My coworker and I sit down together and dig deeper in our investigation. We confirm our worst fears that it IS an issue for that segment.

We report it to engineering. They appear to be confused because the product is working for everyone else, and they seem to have trouble seeing how it’s not working for the group of people we see are having this issue.

Sometimes I think engineers need to get their heads out of the codes. If they would just look at the UI support sees, it then becomes very apparent something is wrong. Very, very wrong. I then check email logs because we send an email confirming devices are linked when activated. We had only 8 so far for today. EIGHT. Out of 2k. I would have expected hundreds at a minimum.

Fuck.

Our team rallies, we brainstorm, we get an action plan going. We ask Product for help because, provided they prioritize this incident, they can save us a lot of work and make a better user experience by having shit work as it’s supposed to, or at least send these people a mass email for us. I mean, product has set that expectation for them that shit will work by a certain day, you know?

Product’s response: Fuck you, just ride it out. It’s only 2k people.

So now I and a coworker have to be on call this weekend to deal with this chaos if it’s not resolved on time.

The funny thing is we just had a workshop on non-violent communication yesterday, but all of us really wanted to flip some tables with Product’s response.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s