Crypto-Crypto Accounting

[the-nurk]

Curious whether I'm doing this correctly. An USD > ETH > BTC trade series should be in manually added as follows?

in_csv will have ETH bought?
in_csv will have BTC bought?
out_csv will have ETH sold?

[eprbell]

That sounds correct:

• USD -> ETH is a normal buy transaction
• ETH -> BTC is a conversion, which gets represented as a sell transaction (ETH) + a buy transaction (BTC). More on this at: https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-conversion-of-a-cryptocurrency-to-another

[the-nurk]

ok sweet. I hadn't made it to the rp2 docs yet. this is all quite confusing for me. the exchanges list their pairs differently and all their csvs have different column names that mean different things. this is really a pain.

[eprbell]

True. Over time more and more exchanges will be natively supported by DaLI via data loader plugins, which will make the process automatic. But if you have to use the manual CSV what you're doing is correct.

[the-nurk]

i do have another question which is: how does dali know what currency the crypto_fee is?
"crypto_fee (optional): transaction fee (if it was paid in crypto). This is mutually exclusive with fiat_fee;" from https://github.com/eprbell/dali-rp2/blob/main/docs/configuration_file.md#manual-section-csv

Binance charges fees in the coin you receive, coinbase in the coin you spend.

I could manually make the adjustments in when translating the binance/coinbase csv into dali in_csv, but this gets finicky when you do crypto-crypto in the same file as crypt-fiat
coinbase:
bought ETH on ETH/USDC
paid fee in USDC
in_csv has to represent this as buying ETH with crypto_fee = USDC fee

sold MATIC on MATIC/BTC
paid fee in BTC
in_csv has to represent this as buying BTC with crypto_fee = BTC fee

i would think the line in question is this:

dali-rp2/src/dali/transaction_resolver.py

Line 639 in 5252b3d

crypto_sent: str = str(RP2Decimal(out_transaction.crypto_out_no_fee) + RP2Decimal(out_transaction.crypto_fee))

but i'm not quite able to parse where crypto_fee on the in_csv gets consumed.

[eprbell]

A crytpo to crypto conversion (e.g. USDC->ETH) gets represented as:

• USDC sell out-transaction
• ETH buy in-transaction
  If the fee is in USDC, then model it with crypto_fee in the out-transaction. If it's in ETH, then model it with crypto_fee in the in-transaction.

So going back to your MATIC/BTC conversion example:

• MATIC sell out-transaction, with 0 fee
• BTC buy in-transaction, with non-zero crypto_fee

Here are two more relevant FAQs:

• https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-represent-fiat-vs-crypto-transaction-fees
• https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-handle-fee-only-defi-transactions

Every DaLI data loader plugin knows the details of how the exchange represents fees (and other things) and it's the data loader's job to translate that information into RP2 primitive patterns. E.g. this is a bit of logic in the Coinbase plugin handling conversions (aka swaps): https://github.com/eprbell/dali-rp2/blob/main/src/dali/plugin/input/rest/coinbase.py#L297

So for exchanges for which we have data loaders (e.g. Coinbase) you can just use the data loader and let it do this work. For unsupported exchanges the manual plugin can be used. Another option is to add a data loader plugin for the unsupported exchange (this work will become much easier once #80 is merged, which should happen fairly soon).

Hope this helps.

i do have another question which is: how does dali know what currency the crypto_fee is? "crypto_fee (optional): transaction fee (if it was paid in crypto). This is mutually exclusive with fiat_fee;" from https://github.com/eprbell/dali-rp2/blob/main/docs/configuration_file.md#manual-section-csv

Binance charges fees in the coin you receive, coinbase in the coin you spend.

I could manually make the adjustments in when translating the binance/coinbase csv into dali in_csv, but this gets finicky when you do crypto-crypto in the same file as crypt-fiat coinbase: bought ETH on ETH/USDC paid fee in USDC in_csv has to represent this as buying ETH with crypto_fee = USDC fee

sold MATIC on MATIC/BTC paid fee in BTC in_csv has to represent this as buying BTC with crypto_fee = BTC fee

i would think the line in question is this:

dali-rp2/src/dali/transaction_resolver.py

Line 639 in 5252b3d

crypto_sent: str = str(RP2Decimal(out_transaction.crypto_out_no_fee) + RP2Decimal(out_transaction.crypto_fee))

but i'm not quite able to parse where crypto_fee on the in_csv gets consumed.

[the-nurk]

thanks a bunch for your time answering this. sorry i required it. I found rp2, then started reading and realized i needed dali, so i went on reading and trying to use that without ever getting back to all of rp2's documentation. There is overlap, so maybe the dali README needs to reference some of the RP2 README?

your links helped of course, but i'll also link this for any future googlers.

https://github.com/eprbell/rp2/blob/4915daeefbe28421b1e67d916f0c43281a8258aa/docs/input_files.md?plain=1#L64

[the-nurk]

uh. what should crypto_fee be when it's not applicable? When I paid a fee in USD, i mark the crypto_fee as 0 and get:

Exception: Internal error: both 'crypto_fee' and 'fiat_fee' are defined, instead of only one: their values are 0 and 9 respectively

when i mark crypto_fee as '' (empty string) i get:

Exception: Internal error: crypto_fee is empty: 94647865-4360-5883-6847-859226820269,2018-12-16 19:38:38+00:00,USDC,Coinbase Pro,nick,SELL,1,150,,150,150,0,

[eprbell]

Can you provide the stack trace for both errors?

[the-nurk]

setting crypto_fee to 0:

2022-09-29 12:41:33,338/dali/INFO: Country: us
2022-09-29 12:41:33,339/dali/INFO: Initialized input plugin 'dali.plugin.input.csv.manual'
2022-09-29 12:41:33,340/dali/INFO: Initialized pair converter plugin 'dali.plugin.pair_converter.ccxt'
2022-09-29 12:41:33,342/dali/INFO: Reading crypto data using plugin 'dali.plugin.input.csv.manual'
2022-09-29 12:41:33,343/dali/ERROR: Fatal exception occurred:
Traceback (most recent call last):
  File "/home/vagrant/.local/lib/python3.10/site-packages/dali/dali_main.py", line 163, in _dali_main_internal
    result_list = pool.map(_input_plugin_helper, input_plugin_args_list)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/vagrant/.local/lib/python3.10/site-packages/dali/dali_main.py", line 199, in _input_plugin_helper
    plugin_transactions = input_plugin.load()
  File "/home/vagrant/.local/lib/python3.10/site-packages/dali/plugin/input/csv/manual.py", line 100, in load
    self._load_in_file(result)
  File "/home/vagrant/.local/lib/python3.10/site-packages/dali/plugin/input/csv/manual.py", line 135, in _load_in_file
    InTransaction(
  File "/home/vagrant/.local/lib/python3.10/site-packages/dali/in_transaction.py", line 80, in __init__
    raise Exception(
Exception: Internal error: both 'crypto_fee' and 'fiat_fee' are defined, instead of only one: their values are 0 and 9 respectively
2022-09-29 12:41:33,347/dali/INFO: Log file: ./log/rp2_2022_09_29_12_41_33_095669.log
2022-09-29 12:41:33,347/dali/INFO: Generated output directory: output
2022-09-29 12:41:33,348/dali/INFO: Done

setting crypto_fee to "":

2022-09-29 12:40:21,948/dali/INFO: Country: us
2022-09-29 12:40:21,950/dali/INFO: Initialized input plugin 'dali.plugin.input.csv.manual'
2022-09-29 12:40:21,951/dali/INFO: Initialized pair converter plugin 'dali.plugin.pair_converter.ccxt'
2022-09-29 12:40:21,953/dali/INFO: Reading crypto data using plugin 'dali.plugin.input.csv.manual'
2022-09-29 12:40:22,151/dali/ERROR: Fatal exception occurred:
Traceback (most recent call last):
  File "/home/vagrant/.local/lib/python3.10/site-packages/dali/dali_main.py", line 163, in _dali_main_internal
    result_list = pool.map(_input_plugin_helper, input_plugin_args_list)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/vagrant/.local/lib/python3.10/site-packages/dali/dali_main.py", line 199, in _input_plugin_helper
    plugin_transactions = input_plugin.load()
  File "/home/vagrant/.local/lib/python3.10/site-packages/dali/plugin/input/csv/manual.py", line 101, in load
    self._load_out_file(result)
  File "/home/vagrant/.local/lib/python3.10/site-packages/dali/plugin/input/csv/manual.py", line 184, in _load_out_file
    OutTransaction(
  File "/home/vagrant/.local/lib/python3.10/site-packages/dali/out_transaction.py", line 69, in __init__
    self.__crypto_fee: str = self._validate_numeric_field(Keyword.CRYPTO_FEE.value, crypto_fee, raw_data, disallow_empty=True, disallow_unknown=True)
  File "/home/vagrant/.local/lib/python3.10/site-packages/dali/abstract_transaction.py", line 56, in _validate_numeric_field
    value = cls._validate_string_field(name, value, raw_data, disallow_empty, disallow_unknown)
  File "/home/vagrant/.local/lib/python3.10/site-packages/dali/abstract_transaction.py", line 49, in _validate_string_field
    raise Exception(f"Internal error: {name} is empty: {raw_data}")
Exception: Internal error: crypto_fee is empty: 94647865-4360-5883-6847-859226820269,2018-12-16 19:38:38+00:00,USDC,Coinbase Pro,nick,SELL,1,150,,150,150,0,
2022-09-29 12:40:22,152/dali/INFO: Log file: ./log/rp2_2022_09_29_12_40_21_710513.log
2022-09-29 12:40:22,152/dali/INFO: Generated output directory: output
2022-09-29 12:40:22,152/dali/INFO: Done

[eprbell]

Thanks for the stack traces:

• the first one is in regards to an in-transaction and it's straightforward: in in-transactions only one fee type is allowed (as reported by the message), so you can't have both fiat and crypto fees;
• the second one is in regards to a different transaction, an out-transaction: here the issue is that you left crypto_fee empty: DaLI and RP2 are a bit pedantic and they want you to write 0 instead. This already came up in other discussions: the design principle is avoiding any possible ambiguity, even if that adds a bit of pedantic procedure. However, since you're not the first one to get stuck on this, let me make the error message a bit less obscure: I'll work on that.

[the-nurk]

I don't have crypto and fiat fees on the same line. each in-transaction was either

1. a SELL with the fee paid in the crypto I received
2. a BUY with fee paid in fiat
3. a BUY with crypto fee placed in the out-transations.

I assume you mean for the entire file, which I find to be an interesting design choice, but you're the expert so it's what it's, I guess.

I'll just move those crypto fees to the out-transactions, no biggie.

Thanks again, sorry for being annoying.

[eprbell]

No, I don't mean for the entire file: the message says you have an in-transaction with both crypto_fee and fiat_fee defined, as 0 and 9 respectively (the solution is to remove the 0 from crypto_fee). Now, I personally love tools that are pedantic and non-ambiguous (e.g. I always enable all warning in compilers and have zero tolerance for warnings), but I understand this can be a little frustrating to a user. BTW, no need to apologize: thanks for reporting your experience as a user (if this keeps happening I may even end up relaxing some of the pedantry :-D ).

[the-nurk]

uh ok. so for in-transactions, crypto_fee should be an empty string, but for out-transactions it should be 0? That seems to work for me, but it's confusing cause I was testing both with empty strings or both with 0s and that is why i was getting two different errors.

[eprbell]

Yes, I acknowledge this is a little confusing. It was discussed here: #75. Check that conversation for more details on this topic.

[eprbell]

Also, as mentioned in #75, I'm open to reviewing PRs modifying this, if somebody wants to explore it.

[the-nurk]

sweet. definitely going to try to contribute at some point. but i say that with every good project i find.