track down policy/code change causing cards to fail

[wrought]

We have had many instances of a user who gives a weekly gift to to the @sudoroom account, but at various points of late, their transactions have stopped successful completion. In some of these circumstances, users have reported that they needed to update their card / account information. For instance, I believe one user needed to provide an additional value, which had previously not been required.

For any card that is denied after successful transactions are made, the denial should trigger a notification to that user to inform them, perhaps repeatedly, until the correct and complete information is provided.

[Changaco]

We've recently added an on-site notification (#2640), but we still don't have email notifications. Closing in favor of #1124.

[wrought]

Maybe I should re-phrase. Lay aside all the other tickets for a second.

Does the following sound like a good user experience?

1. I start giving money to my new favorite hackerspace via gittip.
2. I stop watching my donation, which I know is recurring weekly.
3. Later, it turns out that my provided credit card information became insufficient for transactions, by which changes in gittip stop processing payments.
4. I do not receive any notification about this change.
5. I do not realize that I am no longer donating weekly to my favorite hackerspace via gittip, until perhaps I check at some later time.

To me, this is a very bad experience, which is completely avoidable.

It should be easy to tell which transactions were working, but stopped after a change in policy/code affecting card / bank transactions.

Sending receipts to users in the future still allows for this error to become silent. I'm suggesting pro-active notification is not only courteous, but essential to a positive experience if such issues have arisen and may arise again.

[chadwhitacre]

@wrought Sorry for the very bad user experience. :-(

Let's make sure we're on the same page about what exactly is happening here.

Notify User if unable to charge Credit Card / Debit Card / Bank Account

First, we aren't yet able to charge bank accounts. That's ticketed as "payin via ACH debit" (#777). We are only able to charge credit cards and debit cards.

For instance, I believe one user needed to provide an additional value, which had previously not been required.

If this is true then that's unintentional and it's a bug that we need to track down and fix. It may be a requirement that changed upstream with @balanced, our payments provider.

Later, it turns out that my provided credit card information became insufficient for transactions, by which changes in gittip stop processing payments.
[...] but stopped after a change in policy/code affecting card / bank transactions.

Right: We have not intentionally made any changes in Gittip's policy or code that would render previously working cards insufficient for transactions. Can you track down more information for us on the user that needed to provide an additional value that had previously not been required?

[chadwhitacre]

@wrought While this is the first report we've had of a card failing because of a policy/code change on our end, we definitely have many credit cards that fail due to reasons outside our control. Here are the four reasons we see most often, with their remedies:

1. insufficient funds—the user needs more money :-(
2. card expires or is cancelled—the user needs to update their credit card information on file with us
3. Balanced flags the user as fraud—the user needs to contact [email protected] and we will contact Balanced
4. User's bank flags Gittip as fraud—the user needs to contact their bank and have Gittip whitelisted

As @Changaco mentions, we added on-site notifications for credit card failures in #2640, but we still don't have email notifications for card failures (the problem runs deep: we don't have email notifications for anything, because we don't verify (#2312) let alone require emails yet). That's something we first ticketed back in #583. It looks like @Changaco closed that during a recent ticket purge in favor of on-site notifications (#1746), but on-site notifications are not sufficient: we need email notifications of credit card failures. I'm fine expanding the scope of #1124 to include email notifications for failed as well as successful card charges; I've modified the title over there to reflect this expanded scope.

So let's move the discussion of email notifications to #1124, and let's use this ticket to track down the policy/code change that caused previously working cards to fail.

[wrought]

@whit537
I see, perhaps the issues are upstream with balanced. On the other hand, these balanced error codes seem fairly expressive, so there seems to be good content to reveal to the user when there is some issue, rather than failing silently.

In any case, I'm trying to pull in more information about what users have reported and our experience has been over the last couple of months:

1. Jun 11 - One of our members noticed a significant change in our weekly donations
   • "Last week, our Gittip dropped by over $100/week. It seems like they
     were purging expired cards, or made some other changes to their
     system? Either way, please log into Gittip and confirm that you are
     actually still giving the amount you originally signed up for. And
     thanks for funding Sudo!"
2. Jul 11 - 29 - Try to publicize realization that no notice is given for declined cards.
   • "Please check your Gittip accounts as we've seen a sharp decline in the past week! Gittip doesn't notify users when their cards are declined."
3. Aug 5 - One member verified that no notice had been given to a rejected card that had previously worked fine.
   • "Gittip, our primary funding mechanism, doesn't notify you when your credit card fails to process. This has been happening with increasing regularity over the past two months (we're now at $338.40/wk, and not long ago we had tipped $500!). I signed in today and my card had been rejected, so I had to reenter my info."

Finally, this may be easiest to look up on your end since it seems failures are logged (#364) and theoretically it may be possible to match on intended recipient @sudoroom for payments between June and August 2014.

[wrought]

Also, it seems the member in item "3." above may have been required to re-enter information if the card was rejected once (thought I don't know what the actual issue was). Is gittip robust to multiple attempts of a card that had a previous error?

Simple use case for this is "empty account" say you run out of money before your next paycheck is cashed. If your transaction fails once while empty, even if the account will be replenished within 24 hours by your delayed paycheck, will the card be used again to attempt another transaction at a later time, or even just at next week's transaction? Or will the failure imply the card is faulty and therefore require the data to be "touched" or re-entered to proceed? Is the card data purged or hidden?

[chadwhitacre]

I am finding 82 Gittip users with tips set up to @sudoroom. Of these:

• 50 have a working credit card on file.
• 15 have a failing credit card on file.
• 17 have no credit card on file.

One of the users with no credit card on file is receiving money from elsewhere on Gittip, and regifts to sudoroom, explaining the 51 for number of patrons on sudoroom's profile page chart:

Here is the SQL I ran to get the above numbers:

=> select username, last_bill_result into sudoroom_backers from participants where username in (select tipper from current_tips where tippee='sudoroom' and amount > 0);
SELECT 82
=> select count(*) from sudoroom_backers;
┌───────┐
│ count │
├───────┤
│    82 │
└───────┘
(1 row)

=> select count(*) from sudoroom_backers where last_bill_result = '';
┌───────┐
│ count │
├───────┤
│    50 │
└───────┘
(1 row)

=> select count(*) from sudoroom_backers where last_bill_result > '';
┌───────┐
│ count │
├───────┤
│    15 │
└───────┘
(1 row)

=> select count(*) from sudoroom_backers where last_bill_result is null;                  
┌───────┐
│ count │
├───────┤
│    17 │
└───────┘
(1 row)

[chadwhitacre]

Here are the error messages we have on file for the 15 failing credit cards:

=> select last_bill_result from sudoroom_backers where last_bill_result > '' order by last_bill_result;
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                                        last_bill_result                                                                         │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 402 Payment Required, 127: This transaction was declined by the card issuer. Customer please call bank. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. │
│ 402 Payment Required, 301: Invalid Account Number Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.                                                       │
│ 402 Payment Required, 307: This transaction was declined by the card issuer. Customer please call bank. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. │
│ 402 Payment Required, 307: This transaction was declined by the card issuer. Customer please call bank. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. │
│ 402 Payment Required, 349: This transaction was declined by the card issuer. Customer please call bank. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. │
│ 402 Payment Required, 349: This transaction was declined by the card issuer. Customer please call bank. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. │
│ 402 Payment Required, 349: This transaction was declined by the card issuer. Customer please call bank. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. │
│ 402 Payment Required, 349: This transaction was declined by the card issuer. Customer please call bank. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. │
│ 402 Payment Required, 349: This transaction was declined by the card issuer. Customer please call bank. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. │
│ 402 Payment Required, 349: This transaction was declined by the card issuer. Customer please call bank. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. │
│ 402 Payment Required, 349: This transaction was declined by the card issuer. Customer please call bank. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx. │
│ 402 Payment Required, 900: This card has been invalidated. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.                                              │
│ 402 Payment Required, 900: This card has been invalidated. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.                                              │
│ 402 Payment Required, 900: This card has been invalidated. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.                                              │
│ 402 Payment Required, 900: This card has been invalidated. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.                                              │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
(15 rows)

[chadwhitacre]

For these 15 users with failing credit cards, we show them a red notification on the website with a link to fix their card (as discussed, email notifications have yet to be implemented; that's #1124):

[chadwhitacre]

We present the reason for the failure to the user on the credit card page itself:

[chadwhitacre]

Also, it seems the member in item "3." above may have been required to re-enter information if the card was rejected once (thought I don't know what the actual issue was). Is gittip robust to multiple attempts of a card that had a previous error?

Simple use case for this is "empty account" say you run out of money before your next paycheck is cashed. If your transaction fails once while empty, even if the account will be replenished within 24 hours by your delayed paycheck, will the card be used again to attempt another transaction at a later time, or even just at next week's transaction? Or will the failure imply the card is faulty and therefore require the data to be "touched" our re-entered to proceed? Is the card data purged or hidden?

We retry failing cards every week until the user removes or updates their credit card info on file with us. We've discussed more complex retry schemes (#1637), but I think email notifications are a higher priority.

Here are the three failure messages we're seeing above for sudoroom backers:

• (10) This transaction was declined by the card issuer. Customer please call bank.
• (4) This card has been invalidated.
• (1) Invalid Account Number

In terms of the failure scenarios outlined above, the first of these indicates that a user's bank has flagged Gittip as fraud, and the fix is for the user to call their bank. The card invalidation messages mean we have an expired or cancelled card, and the user should re-enter their details.

For what it's worth, we're in the same boat together: the Gittip team has 19 backers with failing credit cards, and I personally have 64.

Bottom line: I'm sorry we don't yet have email notifications for card failures. :-(

I think the best I can offer at this point is to work together on #1124.

[chadwhitacre]

@wrought Though I'm open to suggestions, and happy to answer further questions ...

[chadwhitacre]

@wrought You might also be interested in #2534 and #2603. Feel free to add +1s over there.

[wrought]

Thanks, I have a few questions about the information you've produced:

1. Of those 15 with failed transactions, had any of them made successful transactions (especially with sudo room) in the past, perhaps with the same card?
2. Of the 17 with no credit card on file, did they ever have a credit card on file, and thus did it work or fail at any point?
3. Further, of this 17 with no card on file, for the 16 non-receiving users (one receives money) ever receive enough income from tips in order to make successful transactions (especially with sudo room) in the past?

[chadwhitacre]

1. Of those 15 with failed transactions, had any of them made successful transactions (especially with sudo room) in the past, perhaps with the same card?

Yes, all but one of them. Here's the number of successful transactions for each of the 14 who made successful transactions before their cards started failing:

=> select count from
(select distinct on (tipper) count(*) from transfers where tipper in
  (select username from sudoroom_backers where last_bill_result > '')
 and tippee = 'sudoroom' group by tipper)
as foo order by count;
┌───────┐
│ count │
├───────┤
│     7 │
│     7 │
│     8 │
│     9 │
│     9 │
│    11 │
│    12 │
│    12 │
│    12 │
│    15 │
│    15 │
│    22 │
│    25 │
│    30 │
└───────┘
(14 rows)

Unfortunately, it's non-trivial for me to determine how many different cards each of them has used.

[chadwhitacre]

(2) Of the 17 with no credit card on file, did they ever have a credit card on file, and thus did it work or fail at any point?

One of the 17 had a card that worked for one week, and failed for 18 weeks before the user removed the card. The remainder have never had a card on file with us.

[chadwhitacre]

(3) Further, of this 17 with no card on file, for the 16 non-receiving users (one receives money) ever receive enough income from tips in order to make successful transactions (especially with sudo room) in the past?

Turns out there are three backers who do not have a card attached right now that are making successful payments to sudoroom. One of these is the one that had a card at one point but does not now; they make payments but not every week, just as their incoming money allows. The second is paying consistently out of their incoming tips. The third funded their Gittip account via bitcoin a while ago and has been working that down.

=> select count from
-> (select distinct on (tipper) count(*) from transfers where tipper in
(>   (select username from sudoroom_backers where last_bill_result is null)
(>  and tippee = 'sudoroom' group by tipper)
-> as foo order by count;
┌───────┐
│ count │
├───────┤
│     9 │
│    15 │
│    64 │
└───────┘
(3 rows)

I'm going to clarify the last_bill_result situation of this week's 51 backers ...

[chadwhitacre]

Only 49 of this week's sudoroom backers have a working credit card on file. The other two have no credit card on file, and used incoming money to fund their tip to sudoroom. But don't we have 50 backers with a working credit card on file? What happened to that last one?

=> select last_bill_result, count from 
(select distinct on (last_bill_result) last_bill_result, count(*)
 from transfers t join sudoroom_backers sb on t.tipper = sb.username
 where tippee='sudoroom' and timestamp > (select ts_start from paydays where ts_start::date = '2014-08-21')
 group by last_bill_result)
as foo;
┌──────────────────┬───────┐
│ last_bill_result │ count │
├──────────────────┼───────┤
│                  │    49 │
│ ¤                │     2 │
└──────────────────┴───────┘
(2 rows)

[chadwhitacre]

Ah! The extra person is marked suspicious in our database, and we didn't filter on that in our queries above. If you would like to discuss further whether this person is indeed suspicious, let's drop to private email for that: [email protected].

[chadwhitacre]

P.S. Here's the SQL I used to determine who the extra user is:

=> select tipper into sudoroom_cc from transfers t join sudoroom_backers sb on t.tipper = sb.username where tippee='sudoroom' and timestamp > (select ts_start from paydays where ts_start::date = '2014-08-21') and last_bill_result = '' order by tipper;
SELECT 49
=> select * from sudoroom_cc full join sudoroom_backers on tipper=username where last_bill_result='' and tipper is null;

[chadwhitacre]

What else can I answer for you, @wrought? :-)

[wrought]

I think those are sufficient answers, thanks. I sent an email to follow up on the suspicious activity.

[chadwhitacre]

@wrought Cool.

Re: suspicious activity: since sudoroom was on the homepage leaderboard for a while, it's possible that a suspicious person set up a tip to you at random, someone unassociated with the group.

[chadwhitacre]

https://gittip.freshdesk.com/helpdesk/tickets/739