Skip to content

Commit

Permalink
storage_simple: Implement asynchronous iteration and use it for the i…
Browse files Browse the repository at this point in the history
…terator

This commit implements the asynchronous iteration API defined and described in
previous commits for the simple storage layer and reimplements the synchronous
iterator with it.

This commit message does not provide background information, please refer to the
two previous commits.

Implementation

sml_ai_init() initializes the handle and choses either a simple or more
elaborate "boc" lease function depending on whether or not a streaming fetch is
ongoing (boc present).

sml_ai_lease_simple() is just that, dead simple. It iterates the storage segment
list and fills the struct vaiov array provided by the caller. It is a good
starting point into the implementation.

sml_ai_lease_boc() handles the busy case and is more elaborate due to the nature
of streaming fetches. It first calls ObjVAIGetExtend() to get the current
extent. If no data is available, it returns the appropriate value.

Other than that, is basically does the same things as sml_ai_lease_simple() with
these exceptions: It also needs to return partial extends ("fragments") and it
needs to handle the case where the last available segment is reached, in which
case there is no successor to store for the next invocation, but also the last
segment could get returned and thus freed before the next invocation.

sml_ai_return() is only used for the "boc" case. It removes returned full
segments from the list and then frees them outside the boc mtx.

sml_ai_fini() is straight forward and should not need explanation.

Implementation of sml_iterator() using the new API

To reimplement the existing synchronous iterator based on the new API, we first
need a little facility to block waiting for a notification. This is struct
sml_notify with the four sml_notify* functions. sml_notify() is the callback,
sml_notify_wait() blocks waiting for a notification to arrive.

Until it runs out of work, the iterator performs these steps:

ObjVAIlease() is called repeatedly until either the vaiov array is full or a
negative value is returned. This allows the rest of the code to react to the
next condition appropriately by sending an OBJ_ITER_FLUSH with the last lease
only.

Calling func() on each extent is trivial, the complications only come from
handling OBJ_ITER_FLUSH, "just in time" returns and error handling.
  • Loading branch information
nigoroll committed Nov 1, 2024
1 parent 7651240 commit 4ed8587
Show file tree
Hide file tree
Showing 4 changed files with 424 additions and 110 deletions.
1 change: 1 addition & 0 deletions bin/varnishd/storage/storage_persistent.c
Original file line number Diff line number Diff line change
Expand Up @@ -692,6 +692,7 @@ smp_init(void)
smp_oc_realmethods.objsetattr = SML_methods.objsetattr;
smp_oc_realmethods.objtouch = LRU_Touch;
smp_oc_realmethods.objfree = smp_oc_objfree;
smp_oc_realmethods.vai_init = SML_methods.vai_init;
}

/*--------------------------------------------------------------------
Expand Down
Loading

0 comments on commit 4ed8587

Please sign in to comment.