How to prevent fio from touching the page cache?

[evopen]

Hi. I have a question about fio readonly test and file page cache.

Below are four benchmark commands that I use.

> fio --filename=/data/somefile --name=read --ioengine=io_uring --iodepth=8 --readonly --rw=read --bs=4M --direct=1 --buffered=0 --numjobs=1 --invalidate=1 --nonvectored=1 --registerfiles=1

Even with the use of direct=1, the kernel still populate page cache for /data/somefile.

> fincore /data/somefile 
  RES    PAGES  SIZE FILE
44.8G 11752448 44.8G /data/somefile

How can I keep fio from touching the file page cache? Thanks.

[axboe]

Even if fio laid out the files with O_DIRECT, if the fs is instantiating new blocks, it'll resort to buffered IO regardless. Once the file has been laid out, fio will invalidate the page cache contents of it before starting the job, however, by default.

[axboe]

Forgot to mention, there are always filesystems (like btrfs) that are broken in how they deal with O_DIRECT async reads and just do buffered IO for that if it's larger than a page... This could be another explanation for why there's page cache contents for the inode afterwards. In any case, this isn't really something fio can control or manage - it just does O_DIRECT and if the kernel decides to do buffered IO for whatever reason, then fio doesn't even know about it.

[evopen]

I am indeed using btrfs. Btrfs is really good reading transparent zstd compressible file, almost double the speed of the underlying physical disk with 1:10 compression ratio. But this is only true if the page cache is empty before the read for larger than memory file.

Thanks.

[evopen]

from man 5 btrfs

COMPATIBILITY
Compression is done using the COW mechanism so it's incompatible with nodatacow. Direct IO works on com‐
pressed files but will fall back to buffered writes and leads to recompression. Currently nodatasum and
compression don't work together.

[evopen]

Direct io falling back to buffered io is implicit