Port to Beancount #35

chadwhitacre · 2016-12-21T21:24:06Z

Reticketing from #28 (comment). Beancount is a plain-text accounting program, derived from Ledger but better-designed, and written in Python 3. 💃 🐍 Beancount comes with a built-in web interface with standard accounting reports, with a third-party interface besides. Let's use Beancount! :D

Todo

chadwhitacre · 2016-12-21T21:32:15Z

Gosh this is so perfect! It's exactly what we were looking for! Great find, @nobodxbodon! 💃

!m @nobodxbodon

chadwhitacre · 2016-12-21T21:32:48Z

It shows us errors and source data, too!

chadwhitacre · 2016-12-21T22:17:18Z

@nobodxbodon Does the todo in the ticket description give you enough to go on here? Do you agree with this direction?

chadwhitacre · 2016-12-21T22:21:46Z

Basically I think we should port the existing master to Beancount—get it working with just the 2012-06.beancount file, where "working" means:

The file passes bean-check (we should add that check to our test suite for Travis).
The file passes our special-purpose tests in bin/test.py.
- We can do away with any tests in there (if any) that check things that Beancount already guarantees (perhaps test_{income,expenses}_balance?).
The file results in the proper reports via bean-web.

Once we are ported over, we can work on deploying a production instance of bean-web or fava in a separate PR. Sound like a plan, @nobodxbodon?

nobodxbodon · 2016-12-22T01:09:43Z

To make sure, shall I work on this newly created beancount branch?

Also, how about we not delete the original .dat file till .beancount file passes bean-check? Otherwise todo sounds good.

nobodxbodon · 2016-12-26T07:32:50Z

The file passes bean-check (we should add that check to our test suite for Travis).

Done

The file passes our special-purpose tests in bin/test.py.
We can do away with any tests in there (if any) that check things that Beancount already guarantees (perhaps test_{income,expenses}_balance?).

I'll add it after we validate the new report. It seems different from the original output from command line, like #28 (comment)

The original dat file is recovered to pass original test, for now.

The file results in the proper reports via bean-web.

Done.

nobodxbodon · 2016-12-28T06:51:35Z

According to Choosing an Account Type, the existing transactions seem to have some duplicates, like:

; Gittip Payday 0
2012-06-01 * "Users"
Assets:Escrow:Samurai 2.96 USD
Assets:Fee-Buffer:Samurai 0.34 USD
Income:Fee-Buffer:Samurai -0.34 USD
Income:Escrow:Samurai -2.96 USD
2012-06-01 * "Balance Sheet"
Income:Escrow:Samurai 2.96 USD
Income:Fee-Buffer:Samurai 0.34 USD
Liabilities:Fee-Buffer -0.34 USD
Liabilities:Escrow -2.96 USD

This leads to empty (all 0) incoming statement in beancount. Trying to merge them.

nobodxbodon · 2016-12-28T16:53:20Z

@whit537 @kaguillera May I check the meaning of the terms:

"Escrow": money received by receiver
"Fee-Buffer": money paid by giver
"Operations"?

And I suppose Samurai, New-Alliance, IHasAMoney, Stripe, VerificationAndTesting are all receivers?

What's Escrow:Cash?

chadwhitacre · 2016-12-28T19:31:13Z

@nobodxbodon Have you read through "How Our Books are Organized"? We tried to explain the terms there. If it's unclear let's figure out how to upgrade that doc.

chadwhitacre · 2016-12-28T19:50:24Z

FY2013/2012-06.beancount

@@ -3,6 +3,38 @@
 option "title" "Gratipay Finances"
 option "operating_currency" "USD" ; The main currencies you use

+** Accounts ; Open all the accounts you have


Since we want to use these accounts across all month files for the fiscal year, we should store them in a separate file. Under Ledger we were using FY2013.dat and declarations.dat. We will need to figure out a) what files it makes sense to use for Beancount, and b) how to make Beancount look at multiple files at once (simple cat?).

Seems here mentions how: https://bitbucket.org/blais/beancount/issues/84/include-doesnt-work-with-booking-method. Will try.

chadwhitacre · 2016-12-28T19:56:05Z

To make sure, shall I work on this newly created beancount branch?

Yes! :-)

Also, how about we not delete the original .dat file till .beancount file passes bean-check?

Suit yourself. I figured since it was still on master we didn't need it here. :-)

It seems different from the original output from command line, like #28 (comment)

Howso?

nobodxbodon · 2016-12-28T20:30:31Z

Have you read through "How Our Books are Organized"? We tried to explain the terms there. If it's unclear let's figure out how to upgrade that doc.

Ah that clears much. Sorry I overlooked. I'll need to check each transaction to understand better.

Howso?

The numbers don't match with the original report from command line. Like the Equity:Owners:Chad part.

------ Update -----
I just realize I was using 2012-06 instead of FY2013. After including 2012-06.beancount from FY2013.beancount, Assets total and equity from you are consistent with original report. Still need to fix liability part. Commits coming later.

chadwhitacre · 2016-12-28T22:39:10Z

According to Choosing an Account Type, the existing transactions seem to have some duplicates, like:

I don't believe those are true duplicates. We did it that way to get the income statement to come out properly under Ledger. Open to revising if need be.

nobodxbodon · 2016-12-29T07:41:32Z

I got the numbers agree with original reports. As I understand from the source code of original balance-sheet and income-statement, they both read only part of the ledger file, and the parts titled with 'balance sheet' seem to be part of corresponding transaction. So they can be merged to transaction, as in latest commit?

Still I feel uncertain about the accounts I created, like: Liabilities:Fee-Buffer:Samurai

nobodxbodon · 2016-12-29T17:15:51Z

Here's the first transaction in original dat. Could you check if I understand the items correctly?:

2012-06-01  Users
    Assets:Escrow:Samurai            $     2.96 <-- amount given to user Samuri
    Assets:Fee Buffer:Samurai        $     0.34 <-- processing fee of $2.96 above. Why isn't it negative?
    Income:Fee Buffer:Samurai       -$     0.34
    Income:Escrow:Samurai           -$     2.96
2012-06-01  Balance Sheet
    Income:Escrow:Samurai            $     2.96
    Income:Fee Buffer:Samurai        $     0.34 <-- above 4 lines sum up to 0, so I removed them all in beancount
    Liabilities:Fee Buffer          -$     0.34
    Liabilities:Escrow              -$     2.96

Besides, why is there a $31.35 fee buffer here? It seems much larger than normal fee:

2012-06-04  Samurai
    Expenses:Fee Buffer:Samurai      $    31.35
    Assets:Fee Buffer:New Alliance  -$    31.35

chadwhitacre · 2016-12-29T18:07:33Z

@nobodxbodon I've taken the liberty of reformatting your comment for better readability, using triple-backticks.

chadwhitacre · 2016-12-29T23:32:43Z

amount given to user Samurai

No, Samurai was our first payment processor. Individual users don't show up on our books here, only in the Gratipay database. The reconciliation dashboard is the intended bridge between the two. So the 2.96 here is the amount by which our escrow-held-at-Samurai increased during this transaction.

processing fee of $2.96 above. Why isn't it negative?

Different processor handle fees different ways. I will have to go back through to get it all straight, but basically some of them send us all the money first and then draw some of it back, whereas others only send us the net. Either way, though, the fee is coming out of the money that was charged (in aggregate here; we don't see individual charges as mentioned above). So basically there is an aggregate settlement into Samurai of $3.30 for this day, and $0.34 of that is fees. We account for that using the Fee Buffer for reasons described in the README.

above 4 lines sum up to 0, so I removed them all in beancount

Those lines were added so that the transaction would show up on the income statement. What is the effect of removing those lines on beancount's income statement? I haven't checked yet but that is the question I have.

Besides, why is there a $31.35 fee buffer here? It seems much larger than normal fee:

Yeah, Samurai was sloppy with their own accounting and made several errors. @kaguillera and I went through this with a fine-toothed comb vis-a-vis our bank statement when we initially produced the file; I'm sorry if we didn't comment it well enough. I will try to revisit ...

chadwhitacre · 2016-12-29T23:35:36Z

Off the top of my head:

Payment Processors: Samurai, Cash, Stripe, Balanced, PayPal, Coinbase, Braintree, TransferWise
Banks (Escrow): New Alliance, Ally, Citizens
Banks (Operations): New Alliance, Ally, Citizens, PNC

italics = current

nobodxbodon · 2016-12-30T05:50:19Z

No, Samurai was our first payment processor. Individual users don't show up on our books here, only in the Gratipay database.

Oh thanks for this and the explanation of fee buffer. Now it makes much more sense for me. After re-reading the README I think it already implicitly indicates those are processors, but I didn't realize.

What is the effect of removing those lines on beancount's income statement? I haven't checked yet but that is the question I have.

I don't think those sum up to 0 affect income statement. I got the same income statement after removing these lines in 2012-06.dat:

    Income:Fee Buffer:Samurai                                      -$     0.34
    Income:Escrow:Samurai                                          -$     2.96
2012-06-01  Balance Sheet
    Income:Escrow:Samurai                                           $     2.96
    Income:Fee Buffer:Samurai                                       $     0.34

Besides, as shown in the screenshot I posted, Beancount outputs the same income statement.

I will try to revisit ...

That will be ideal, but maybe we can start with the latest first, and backfill/revisit when there's bandwidth.

nobodxbodon · 2016-12-30T07:06:19Z

How can we do the same as 'bean-check' programatically in test.py?

chadwhitacre · 2016-12-30T07:38:44Z

How can we do the same as 'bean-check' programatically in test.py?

@nobodxbodon Can we shell out to the bean-check CLI?

nobodxbodon · 2016-12-30T08:57:51Z

@whit537 you mean using subprocess? I suppose so, but would it work with Travis? Can Travis call bean-check?

BTW how early you get up!

chadwhitacre · 2016-12-30T09:35:43Z

Up earlier than usual this morning. I'll probably crash again in a bit. How late you stay up! :-)

We're already calling ledger via subprocess or equivalent, aren't we?

chadwhitacre · 2016-12-30T15:07:28Z

This is looking great, @nobodxbodon! I made a couple commits to clean up the file structure—include is a great find!—and bring us into full parity with the balance sheet and income statement on master. With 368346d I think we're there! 💃

Old

New

chadwhitacre · 2016-12-30T15:47:46Z

@nobodxbodon Did you get anywhere last night with calling bean-check from bin/test.py?

nobodxbodon · 2016-12-30T16:34:11Z

Did you get anywhere last night with calling bean-check from bin/test.py?

@whit537 pitifully not really. The question was my last activity :)

chadwhitacre · 2016-12-30T16:42:34Z

At least that means I didn't clobber any of your work by pushing commits overtop of yours. :)

nobodxbodon · 2016-12-30T23:58:30Z

About porting existing test suite, here are some tests covered already by bean-check. Maybe there are more:

test_income_balances
test_expenses_balance
test_net_income_reconciles_with_current_activity  // because current equity is calculated by income+expense already

The ones left:

test_escrow_balances
test_fee_buffer_balances
test_fee_buffer_income_reconciles_with_fee_buffer_liability

chadwhitacre · 2017-02-02T20:22:38Z

If it's an income/expense it should just be "processing fees," not "fee buffer." But ... what's the problem with tracking it as a fee buffer the whole way back to the beginning?

chadwhitacre · 2017-02-02T20:40:32Z

I guess that's just not accurate. How will we institute the fee buffer in FY2014, though? I guess we can cross that bridge when we come to it?

chadwhitacre · 2017-02-02T22:02:18Z

I made some commits, but 621eba0 is actually buggy. More work required ...

chadwhitacre · 2017-02-02T22:03:31Z

... but I'm out of time for now. I'll be back.

It wasn't implemented until June, 2013, as now noted in the README.

chadwhitacre · 2017-02-02T22:14:24Z

Alright, fee buffer removal redone in 150c3d0. You good with that, @nobodxbodon et al.?

chadwhitacre · 2017-02-03T09:33:12Z

I can see in the screenshot that Stripe held all together $25.29 before 2/28, and $20.97 was transfered to new alliance on 2/28, with $4.32 fee. But how can I tell that $20.67 in this $20.97 was escrow?

That info comes from the Gratipay database, per fee and amount in exchanges. I believe we had a detailed view for admins on the reconciliation dashboard that gave this, but maybe that is gone now? I'll have to check ...

BTW in the log, I don't even see the $25.29 in either New Alliance or Stripe statement. Do we have another source of statements?

Um ... the 25.29 is in that Stripe screenshot, no? They clear us the net (not the gross, as Samurai did), so we wouldn't expect to see 25.29 in New Alliance. You want to see 25.29 somewhere in the logs repo, is that it?

chadwhitacre · 2017-02-03T13:12:38Z

You want to see 25.29 somewhere in the logs repo, is that it?

Looks like the CSVs we got from Stripe show the net they paid, not the gross or fees, e.g.:

https://github.com/gratipay/logs/blob/master/statements/2012/06/stripe.csv

chadwhitacre · 2017-02-03T13:13:49Z

I'll have to check ...

Yeah, looks like that is gone. I don't think we should block this on that, though, since we're not fundamentally changing the numbers here.

chadwhitacre · 2017-02-03T13:14:12Z

Last thing from my pov here is to give the README and data file one more review.

chadwhitacre · 2017-02-03T16:04:26Z

README looks fine to me, taking a last pass through the beancount file ...

chadwhitacre · 2017-02-03T16:18:10Z

Let's go ahead and record a single transaction for the Stripe transfer (2b7320e), since that's how the info is coming to us in statements as at #35 (comment).

chadwhitacre · 2017-02-03T16:19:09Z

chadwhitacre · 2017-02-03T16:19:53Z

I've confirmed that the sum of Assets:{Escrow,Operations}:New-Alliance matches the balance in our bank statement for that month.

chadwhitacre · 2017-02-03T16:26:00Z

chadwhitacre · 2017-02-03T16:51:20Z

Okay, the balance sheet as of e6c9998 matches the balance sheet in #35 (comment), with the fee buffer (28.86) folded into current activity, and no final Stripe expense (14.63).

-9.79 + 28.86 - 14.63 = 4.44

@nobodxbodon I'm ready for a merge. How about you? :)

nobodxbodon · 2017-02-03T17:01:50Z

LGTM! Shall I merge?

mattbk · 2017-02-03T18:54:43Z

Meeeeeerge...

chadwhitacre · 2017-02-03T19:07:31Z

@nobodxbodon I'm ready for you to merge! :-)

chadwhitacre · 2017-02-04T12:38:08Z

Woo-hoo! Awesome work, everyone! Gratipay is now officially on Beancount! 💃 🌻

chadwhitacre · 2017-02-04T12:48:42Z

Announced: https://twitter.com/Gratipay/status/827861257223073792.

This was referenced Dec 21, 2016

start a Django app #28

Closed

#28 skeleton Django app with balance sheet page #34

Closed

chadwhitacre commented Dec 28, 2016

View reviewed changes

chadwhitacre force-pushed the beancount branch from c7323fe to 54ece88 Compare February 2, 2017 21:35

chadwhitacre force-pushed the beancount branch from 3d213ed to 47f1e2f Compare February 2, 2017 22:12

There *was* no fee buffer in 2012-06

150c3d0

It wasn't implemented until June, 2013, as now noted in the README.

chadwhitacre force-pushed the beancount branch from 47f1e2f to 150c3d0 Compare February 2, 2017 22:14

Reflect inclusion of fees on Stripe's statement

2b7320e

Fix bug with Samurai on balance sheet

e6c9998

nobodxbodon merged commit 50d526a into master Feb 3, 2017

nobodxbodon deleted the beancount branch February 3, 2017 19:43

chadwhitacre mentioned this pull request Oct 5, 2017

Respond to IRS letter gratipay/inside.gratipay.com#1164

Closed

nobodxbodon mentioned this pull request Dec 11, 2018

Use Liberapay for funding opencompany/www.opencompany.org#175

Closed

Port to Beancount #35

Port to Beancount #35

Conversation

chadwhitacre commented Dec 21, 2016 • edited Loading

Todo

chadwhitacre commented Dec 21, 2016 • edited Loading

chadwhitacre commented Dec 21, 2016

chadwhitacre commented Dec 21, 2016

chadwhitacre commented Dec 21, 2016 • edited Loading

nobodxbodon commented Dec 22, 2016

nobodxbodon commented Dec 26, 2016

nobodxbodon commented Dec 28, 2016

nobodxbodon commented Dec 28, 2016

chadwhitacre commented Dec 28, 2016

chadwhitacre Dec 28, 2016

Choose a reason for hiding this comment

nobodxbodon Dec 28, 2016

Choose a reason for hiding this comment

chadwhitacre commented Dec 28, 2016

nobodxbodon commented Dec 28, 2016 • edited Loading

chadwhitacre commented Dec 28, 2016

nobodxbodon commented Dec 29, 2016 • edited Loading

nobodxbodon commented Dec 29, 2016 • edited by chadwhitacre Loading

chadwhitacre commented Dec 29, 2016

chadwhitacre commented Dec 29, 2016

chadwhitacre commented Dec 29, 2016

nobodxbodon commented Dec 30, 2016

nobodxbodon commented Dec 30, 2016

chadwhitacre commented Dec 30, 2016

nobodxbodon commented Dec 30, 2016 • edited Loading

chadwhitacre commented Dec 30, 2016

chadwhitacre commented Dec 30, 2016

Old

New

chadwhitacre commented Dec 30, 2016 • edited Loading

nobodxbodon commented Dec 30, 2016

chadwhitacre commented Dec 30, 2016

nobodxbodon commented Dec 30, 2016

chadwhitacre commented Feb 2, 2017

chadwhitacre commented Feb 2, 2017 • edited Loading

chadwhitacre commented Feb 2, 2017

chadwhitacre commented Feb 2, 2017

chadwhitacre commented Feb 2, 2017

chadwhitacre commented Feb 3, 2017

chadwhitacre commented Feb 3, 2017

chadwhitacre commented Feb 3, 2017

chadwhitacre commented Feb 3, 2017

chadwhitacre commented Feb 3, 2017

chadwhitacre commented Feb 3, 2017

chadwhitacre commented Feb 3, 2017

chadwhitacre commented Feb 3, 2017

chadwhitacre commented Feb 3, 2017

chadwhitacre commented Feb 3, 2017

nobodxbodon commented Feb 3, 2017

mattbk commented Feb 3, 2017

chadwhitacre commented Feb 3, 2017

chadwhitacre commented Feb 4, 2017

chadwhitacre commented Feb 4, 2017

chadwhitacre commented Dec 21, 2016 •

edited

Loading

chadwhitacre commented Dec 21, 2016 •

edited

Loading

chadwhitacre commented Dec 21, 2016 •

edited

Loading

nobodxbodon commented Dec 28, 2016 •

edited

Loading

nobodxbodon commented Dec 29, 2016 •

edited

Loading

nobodxbodon commented Dec 29, 2016 •

edited by chadwhitacre

Loading

nobodxbodon commented Dec 30, 2016 •

edited

Loading

chadwhitacre commented Dec 30, 2016 •

edited

Loading

chadwhitacre commented Feb 2, 2017 •

edited

Loading