Reading from one database and writing to another is very slow

[samuelgozi]

I hope I will give enough information to clarify the issue, but if not, please let me know, I really need to solve this :)

I have a very big database, with almost 2 Billion entries. And in order to query it efficiently enough, I am manually creating indexes.
The way I create the indexes, is by taking database "A" and iterating over all of its values, and at the same time writing to database "B" with a modified key.

I do this process two times before I get to the final database "N" that is ordered, and parsed in the optimal way.

When I do this with LevelDB, the performance is good, but not great. I am able to average 150K writes per second when building the index. So again, building the index is Reading from DB "A", generating an new key and writing to "B".

Its important to note, that I am doing a multi step index building because in order for the process to perform as fast, each step only reads from one DB, and only writes to the other. Thats why with LevelDB this process is able to perform 150K writes per second.

For whatever reason, If I read from LevelDB and write to LMDB the process is extremely fast (600K writes per second), but if I read from a LMDB database, and write to another LMDB database, it goes down to ~3K writes pre second, which will take ages...
And yes, I am batching writes (it seems like in my machine the optimal batch size is 300K).

It seems to me that I am doing something wrong because if I compare LMDB to LevelDB in only reads, or only writes, LMDB performs much better. So I think that maybe both databases share the same process, or something like that.
What can I be doing wrong?

[kriszyp]

When you talk about database "A" and "B" are you talking about two different database stores in the same database file (opened via rootStore.openDB), or two different files (two different calls to open())? It is possible that read transactions (that span the whole database file) could be impacting write transactions since it can't clear older pages that may still be in use by open read transactions.

It might also be interesting to look at your stats (db.getStats()) as that has a lot information about number of I/O writes and time spent on transactions.

[samuelgozi]

Its two different database files (open). Should I be calling db.getStats() while performing the write operations?