refactor for double-entry accounting

[chadwhitacre]

See #3616 for initial thinking on this.

• Implement business logic as triggers, with good Python test coverage.
• Update payday.
• Write a migration script that converts transfers, payments, and exchanges into journal entries.

[chadwhitacre]

🐘 🍴

[chadwhitacre]

Hrm ... time to do a little spelunking.

What writes to and reads from the exchanges, payments, and transfers tables?

[chadwhitacre]

record_exchange and record_exchange_result.

sync_with_balanced looks like it can go away.

Related tickets: #2443 #3316 #2442 #2779 .

[chadwhitacre]

exchanges is referenced in notify_participants and update_stats during payday.

[chadwhitacre]

payday.sql builds a payday_payments table, and the Python side references payments as well.

[chadwhitacre]

I think we start with payments and then do exchanges.

[chadwhitacre]

After dropping the payments table in test:

22 failed, 442 passed, 75 xfailed in 47.60 seconds

[chadwhitacre]

I'm working on test_process_subscriptions. What does the journal look like for a subscription fulfillment during payday?

id	amount	debit	credit	parent	payday	route
3	10.00	cash	~user	null	1	1
4	10.00	~user	Team	null	1	null

(Let's keep parent just for state changes with asynchronous payments, refunds, and reversals.)

And how about payroll/owner distribution?

id	amount	debit	credit	payday	route
5	10.00	Team	~hannibal	1	null

See #3616 (comment) for payins, and #3616 (comment) for payouts.

[chadwhitacre]

Alright, I made it through one test. :-)

I actually feel pretty good about this approach. It took me a while to start getting comfortable with this table:

	Debit	Credit
Asset	Increase	Decrease
Liability	Decrease	Increase
Income/Revenue	Decrease	Increase
Expense	Increase	Decrease
Equity/Capital	Decrease	Increase

But now that it's starting to sink in, I feel like this is really good for us. We're going to have real double-entry accounting! :D That feels like Doing It Right™ to me.

[chadwhitacre]

20 failed, 457 passed, 75 xfailed

[chadwhitacre]

We should have all of our actual escrow accounts in the chart of system accounts. Today that's New Alliance and PayPal. It takes a few days to move money between the two, so we should also have accounts for pending transfers.

[chadwhitacre]

56 failed, 423 passed, 75 xfailed

[chadwhitacre]

@kaguillera and I went around the horn on this here at Blue Canary. He has a bit of a background in accounting and finance (@chrisdev could say even more, I'm sure ;-).

Regarding the two different schemas for representing journal entries, @kaguillera's input is that accountants will expect the former (expanded) format, and will find the second (compact) format idiosyncratic and wrong. He recommends using the compact format for our database schema since it makes more sense there (it's a simpler implementation of the accounting invariant that every debit has a corresponding credit), but to display our journal in the proper accounting format. Perhaps it would help to change the debit and credit columns to debiting and crediting to make clear that the journal table is slightly different than the resulting display will be (it includes extra fields, as well).

We also went through the chart of accounts together. Rather than a cash account and a single processing fee expenses account, we should have separate accounts for each place we're actually holding funds (along with pending transfer accounts for each, as suggested above), and also separate expense accounts for each one.

The journal needs to record everything that happens to Gratipay's escrow. As part of deploying this PR, we should reconstruct the journal back to the very beginning of Gratipay. Some of what's happened to our escrow is recorded in the three tables in our database, exchanges, transfers, and payments. The rest is recorded outside of our database, in the history of the various accounts where we've held escrow, such as at Balanced, New Alliance, and PayPal.

One of the purposes of the journal is to make auditing possible. Auditing means reconciling internally (ensuring that assets equal liabilities), reconciling externally (ensuring that our records line up with those of New Alliance, PayPal, etc.), and making sure we're on the up and up (no "$500 pencils," in @kaguillera's evocative phrase 💰 ✏️).

We will want to ensure that the journal as we reconstruct it is fully reconciled before we start writing to it, and then we should have a process in place for periodically reconciling the journal going forward. We should probably have an annual independent audit, and should probably level-set with a third-party audit as part of landing this PR. I've reticketed that as gratipay/inside.gratipay.com#285.

[chadwhitacre]

I'm trying to figure out how to get financial accounts elsewhere into our chart of accounts and journal, rather than being a separate class of entity. This is what I'm starting with:

id (~=relative timestamp)	account	debit	credit
1	cash	10.00	null
2	~user	null	10.00
3	~user	0.59	null
4	processing fee revenues	null	0.59

[chadwhitacre]

Actually, let's start with this (subbing New Alliance for cash, per recent comments).

id	account	debit	credit
1	New Alliance	10.00	null
2	~user	null	10.00
3	~user	0.59	null
4	processing fee revenues	null	0.59

[chadwhitacre]

We need some number of transactions/entry pairs preceding the New Alliance/~user pair.

Do we need to get money into New Alliance before we flow money out of it?

[chadwhitacre]

In order to fold exchanges into journal, we need to somehow fold exchange_routes into the chart of accounts.

https://en.wikipedia.org/wiki/Chart_of_accounts

[chadwhitacre]

To: a friend in accounting

Good to run into you the other day!

Can I buy you lunch at K&N some day this week or next? How about this Thursday (July 23) or next Monday (July 27)?

In addition to generally catching up, I'd love to pick your brain about setting up a chart of accounts for the escrow for the crowdfunding company I'm running (Gratipay).

[chadwhitacre]

My friend works with ACS (I actually used to work with him there, so I'm not unfamiliar with ACS myself). I picked up a few tidbits from poking around ACS with him:

• ACS uses the term "general ledger" for what I've been calling the journal.
• ACS has transactions which are composed of one or more debits and credits.
• Each transaction has a source which is separate from the accounts in the chart of accounts, as well as a reference number, a date, an explanation, a user, and a couple other things. Source is things like JRE (journal record entry) and CHK (check).
• Each debit and credit has its own description.
• Day to day activity takes place in "modules" like contributions, accounts payable, and accounts receivable.
• Actions performed in the modules have some moderately obfuscated effect on the general ledger.
• It seems that the reference number ties together the general ledger with information in the modules—the GL doesn't know about names of donors, for example, but a given $20 contribution will be available in both the GL and Contributions with a reference number tying them together.

The upshot is that we're building something roughly parallel to ACS. It'd be more helpful to talk to a designer or developer of (something like) ACS than a user of ACS.

[chadwhitacre]

I met with a CPA (gratipay/inside.gratipay.com#285). Tidbits from our initial conversation:

• We're on the right track here.
• Modern accounting uses "[general] ledger" to refer to the entire list of transactions, while a "journal" is a partial record ("a payroll journal"). This confirms what my friend said the other day, too. s/journal/ledger
• Transactions encompass one or more debits and credits.
• The payment method (exchange route: CC, PayPal) wouldn't touch the chart of accounts; that could just be a variable in the ledger ("source/destination of funds").
• PA has a statue against earning interest on escrow. It's aimed at lawyers.

Next steps:

• map out all of the things we need to account for and how we currently account for them
• refactor each thing in terms of double-entry accounting (chart of accounts + ledger)
• meet with our CPA again to review our design

[chadwhitacre]

Thanks for the meeting today, Peter. I'm excited to work with you to design a system for Gratipay that follows proper accounting procedures. As a next step, could we schedule a two-hour meeting for the week of August 9? Monday or Wednesday morning would work best for me that week. Do either of those work for you?

[chadwhitacre]

Great meeting you today and I look forward to our next meeting.

Let me get back to you by tomorrow morning on the dates you mentioned.

Thanks

[chadwhitacre]

Putting a plan together here ... October 9 is our deadline we're working against. Stepping back, it's going to make more sense for us to focus on other balls in the air during August, and hit the accounting system refactor in September. Can I get in touch with you in a few weeks to schedule our next meeting for some time in early- to mid-September?

[chadwhitacre]

(because gratipay/inside.gratipay.com#290 (comment))

[chadwhitacre]

Picking up in #3653.