Skip to content
This repository has been archived by the owner on Jun 6, 2024. It is now read-only.

Use shared kernel and user buffers + use ReadFixed and WriteFixed opcodes #11

Open
connortsui20 opened this issue Apr 14, 2024 · 0 comments
Labels
bpm Buffer Pool Manager

Comments

@connortsui20
Copy link
Member

connortsui20 commented Apr 14, 2024

Right now, the DiskManagerHandle will use the Read and Write opcodes to read and write to files.

A normal file read or write will use the kernel's buffers to bring in files from disk, and only then will it memcpy the data into our buffer Frame. Since io_uring provides a way to register shared buffers between the user and the kernel, we can completely bypass this.


I have already done a first attempt at registering the buffers, but there is still a lot of work to be done. Right now io_uring only supports registering (I think) up to 2^16 buffers at a time, which might seem like a lot of memory but this is actually only 1 MB (2^20 bytes). So we will want to support multiple groups of shared buffers.

Note that io_uring supports something called group IDs, but we don't actually care about these because that is for automatic buffer selection, and we will be doing everything manually (also there can only be up to 2^16 bits of group IDs, which is not enough for what we will want to do because that only allows for 4 GB). We will still need to have a concept of a "group ID", probably with a hash map from "group ID" to a pointer to the base of the group of fixed buffers.

So with that being said, the Frame struct will likely need an ID field or some sort of way to figure out which group of registered/fixed buffers the frame belongs to. We can probably use 20 bits of a 64-bit integer for groups (24 bits of group ids means 2^20 bytes per group * 2^20 buffer groups = 2^40 bytes = 1024 GB), since a buffer pool is likely to not use more than 1024 GB of memory in the near future.

Section 8 of the io_uring article explains this in more detail.


In page/pagedef.rs, there is an implementation of PageId. Each of these should represent a logical page of the database (not a buffer in memory). We should leave some of the least significant bits for future software RAID 0 implementations, and in general we should think about what type of information this PageId needs to carry around (while still fitting into 64 bits).

Once the above is implemented, then we need to switch to using the ReadFixed and WriteFixed opcodes instead of the Read and Write opcodes for the io_uring crate. See the doc pages for the Rust io_uring bindings for more information.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bpm Buffer Pool Manager
Projects
None yet
Development

No branches or pull requests

1 participant