Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Index structures #1

Merged
merged 49 commits into from
Feb 28, 2024
Merged

Index structures #1

merged 49 commits into from
Feb 28, 2024

Conversation

mrjleo
Copy link
Owner

@mrjleo mrjleo commented Feb 28, 2024

  • Implement OnDiskIndex (using HDF5)
  • Implement sharding for InMemoryIndex to support index growth

@mrjleo
create module for indexes
0cee25e
@mrjleo
move interpolate to ranking.py
e3aeb96
@mrjleo
implement chunked InMemoryIndex, update return type of _get_vectors
f16f2c3
@mrjleo
refactor InMemoryIndex._get_vectors
9732c0b
@mrjleo
fix InMemoryIndex._add when chunked and add test case
90b1c12
@mrjleo
Index._get_vectors: remove mode argument
a32258d
@mrjleo
move __version__
16a0c56
@mrjleo
initial implementation of OnDiskIndex
838e840
@mrjleo
OnDiskIndex: implement automatic resizing
8d7db96
@mrjleo
OnDiskIndex: add docstring and comments
00898a1
@mrjleo
OnDiskIndex: refactoring
3910d06
@mrjleo
InMemoryIndex: remove unused lists
ad71836
@mrjleo
InMemoryIndex: add new chung only if necessary
6933e0d
@mrjleo
InMemoryIndex: log error if passage ID already exists
593ecef
@mrjleo
InMemoryIndex: allow first chunk to be larger, implement consolidate …
22f1438 
…method
@mrjleo
add __len__ method to Index class
c8ee460
@mrjleo
OnDiskIndex: use empty shape for ID datasets (hdf5)
3a574a0
@mrjleo
InMemoryIndex: fix consolidate function
1a04bc3
@mrjleo
InMemoryIndex: rename chunks -> shards
cfba2b9
@mrjleo
add dim property to indexes, raise ValueError if dims don't match
f6157de
@mrjleo
OnDiskIndex: fix _get_doc_ids and _get_psg_ids for empty indexes
2c00b9a
@mrjleo
update unit tests for common index functions
7a35b67
@mrjleo
Index: add tests for len, dim, errors
4a3a233
@mrjleo
split index tests into multiple classes
85a2255
@mrjleo
TestOnDiskIndex: remove temp dir when done
a4b64b3
@mrjleo
add tests that resize (grow) indexes
6591419
@mrjleo
add test for adding and retrieving
313400f
@mrjleo
remove logging config for unit tests
acefe3b
@mrjleo
rename modules
99bbff5
@mrjleo
reduce precision for assert_almost_equal
fd532ca
@mrjleo
InMemoryIndex: fix cur_shard_size after consolidation
48b4bc9
@mrjleo
TestInMemoryIndex: add test case for consolidate method
9b0a55d
@mrjleo
InMemoryIndex: don't include unused space when consolidating
fb165ef
@mrjleo
InMemoryIndex: allow consolidation of indexes with a single shard
7b887a0
@mrjleo
add pytest config to pyproject.toml
738eff7
@mrjleo
remove extra tests
ec0ebd2
@mrjleo
implement OnDiskIndex.load
6e14497
@mrjleo
add h5py dependency
e6692e6
@mrjleo
InMemoryIndex: fix _get_vectors when result is empty
60c182f
@mrjleo
InMemoryIndex: add dtype argument
7b19581
@mrjleo
OnDiskIndex: change dataset structure, keep doc and passage ID mappin…
da4c74c 
…gs in memory
@mrjleo
Index: change type of IDs in _add, OnDiskIndex: fix loading with empt…
e0b781b 
…y IDs, update test case
@mrjleo
OnDiskIndex: implement to_memory
4081839
@mrjleo
tests_index: implement function to test vectors and IDs
1cf9bb1
@mrjleo
TestOnDiskIndex: add more test cases in test_load
66daef7
@mrjleo
OnDiskIndex: add max_id_length argument and raise an error when the l…
f09b8a9 
…ength is exceeded
@mrjleo
add test cases for indexes with partial IDs
91848f9
@mrjleo
update readme and documentation
c8ef2fb
@mrjleo
update .gitignore
028712f
@mrjleo mrjleo merged commit 18e1184 into main Feb 28, 2024
7 checks passed
@mrjleo mrjleo deleted the index_structures branch February 28, 2024 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@mrjleo