Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

V1 API /trades Missing Data #172

Open
matthewdowney opened this issue May 12, 2019 · 4 comments
Open

V1 API /trades Missing Data #172

matthewdowney opened this issue May 12, 2019 · 4 comments

Comments

@matthewdowney
Copy link

I'm not sure if this is the right place put API issues — let me know if I should move this somewhere more appropriate!

I'm gathering public trade data from the trades rest endpoint and it seems like there are situations in which the API omits trades during pagination.

Every few minutes I'm hitting the endpoint asking for trades at or after the most recent trade that I've seen and paginating forward in time until I see the page with the most recent trades, but there are times when the API excludes some block of trades directly after the trade I am querying from.

Missing Trades

For example, in a previous request, I had seen the trade

{"timestamp":1557677028,
 "tid":356105007,
 "price":"6957.6",
 "amount":"0.16",
 "exchange":"bitfinex",
 "type":"sell"}

I subsequently queried https://api.bitfinex.com/v1/trades/btcusd?limit_trades=25&timestamp=1557677028 and saw that the first trade was

{"timestamp":1557677283,"tid":356106715,"price":"7000.0","amount":"0.01","exchange":"bitfinex","type":"sell"}

Since I had already seen the above trade with tid 356105007, I am sure there is at least one trade at the timestamp 1557677028, but the second request starts at 1557677283, 255 seconds later. Trades are missing!

Attempts to Resolve

Once I discovered the issue, I tried several things to figure out what was going on.

My first thought was that maybe the API docs were mistaken when they say that the timestamp parameter prompts it to "Only show trades at or after this timestamp". Perhaps the timestamp was exclusive. So subtracting one from the previous timestamp, we'd request https://api.bitfinex.com/v1/trades/btcusd?limit_trades=25&timestamp=1557677027, which still does not include the original trade.

I tried with many different limit_trades values to see if it might be a page size issue, but I saw the same problem with sizes of 25, 250, 500, and 999.

Interestingly, I was only seeing this problem on the BTC/USD book, so it might have something to do with the amount of activity taking place on the exchange.

Pagination Broken?

As I'm writing this, I'm checking the response of https://api.bitfinex.com/v1/trades/btcusd?limit_trades=25&timestamp=1557677028 periodically. It changes every time.

Is the pagination on the endpoint just broken? Is it taking limit_trades starting now, rather than at timestamp?

@matthewdowney
Copy link
Author

I definitely think I'm seeing this issue more, now that the price is popping off! :)

I'm looking at switching to the V2 API but since it's still in beta I'm kind of reluctant, plus it doesn't report trade side which is really necessary for what I'm doing.

@tenninebt
Copy link

I'm facing it with the Websocket API (v1), a lot of missing trades, I'm using the api on a plain simple example, but subscribing to all of the margin symbols. Comparing to market trades directly on the web app, I can see a lot of discrepancies. Didn't try to switch to the V2 yet.

@matthewdowney
Copy link
Author

Wow that's really interesting that you're seeing it on the websocket as well! I assumed it was some sort of caching issue, but that doesn't seem as likely if it's also affecting the socket.

@tenninebt
Copy link

Just realised there was maybe an issue on my side, I need to do more tests to confirm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants