This repository has been archived by the owner on Jun 6, 2024. It is now read-only.
Use shared kernel and user buffers + use ReadFixed
and WriteFixed
opcodes
#11
Labels
bpm
Buffer Pool Manager
Right now, the
DiskManagerHandle
will use theRead
andWrite
opcodes to read and write to files.A normal file read or write will use the kernel's buffers to bring in files from disk, and only then will it
memcpy
the data into our bufferFrame
. Sinceio_uring
provides a way to register shared buffers between the user and the kernel, we can completely bypass this.I have already done a first attempt at registering the buffers, but there is still a lot of work to be done. Right now
io_uring
only supports registering (I think) up to 2^16 buffers at a time, which might seem like a lot of memory but this is actually only 1 MB (2^20 bytes). So we will want to support multiple groups of shared buffers.Note that
io_uring
supports something called group IDs, but we don't actually care about these because that is for automatic buffer selection, and we will be doing everything manually (also there can only be up to 2^16 bits of group IDs, which is not enough for what we will want to do because that only allows for 4 GB). We will still need to have a concept of a "group ID", probably with a hash map from "group ID" to a pointer to the base of the group of fixed buffers.So with that being said, the
Frame
struct will likely need an ID field or some sort of way to figure out which group of registered/fixed buffers the frame belongs to. We can probably use 20 bits of a 64-bit integer for groups (24 bits of group ids means 2^20 bytes per group * 2^20 buffer groups = 2^40 bytes = 1024 GB), since a buffer pool is likely to not use more than 1024 GB of memory in the near future.Section 8 of the
io_uring
article explains this in more detail.In
page/pagedef.rs
, there is an implementation ofPageId
. Each of these should represent a logical page of the database (not a buffer in memory). We should leave some of the least significant bits for future software RAID 0 implementations, and in general we should think about what type of information thisPageId
needs to carry around (while still fitting into 64 bits).Once the above is implemented, then we need to switch to using the
ReadFixed
andWriteFixed
opcodes instead of theRead
andWrite
opcodes for theio_uring
crate. See the doc pages for the Rust io_uring bindings for more information.The text was updated successfully, but these errors were encountered: