[FSTORE-1064] Improve docs for spine groups #1137

jimdowling · 2023-10-21T08:18:06Z

This PR adds/fixes/changes...

please summarize your changes to the code
and make sure to include all changes to user-facing APIs

JIRA Issue: -
https://hopsworks.atlassian.net/browse/FSTORE-1064

Priority for Review: -

Related PRs: -

How Has This Been Tested?

Unit Tests
Integration Tests
Manual Tests on VM

Checklist For The Assigned Reviewer:

- [ ] Checked if merge conflicts with master exist
- [ ] Checked if stylechecks for Java and Python pass
- [ ] Checked if all docstrings were added and/or updated appropriately
- [ ] Ran spellcheck on docstring
- [ ] Checked if guides & concepts need to be updated
- [ ] Checked if naming conventions for parameters and variables were followed
- [ ] Checked if private methods are properly declared and used
- [ ] Checked if hard-to-understand areas of code are commented
- [ ] Checked if tests are effective
- [ ] Built and deployed changes on dev VM and tested manually
- [x] (Checked if all type annotations were added and/or updated appropriately)

SirOibaf · 2023-10-22T12:46:27Z

@jimdowling - you need to format the changes with black (see readme) otherwise we can't merge them.

SirOibaf · 2023-10-22T12:50:15Z

python/hsfs/feature_store.py

-        select label from the left feature group, so that we don't need to provide a spine
-        for online serving.
+        !!! note
+            Spine Groups are not currently supported for online serving.


I think this is misleading. Spine feature groups have no meaning in serving as they represent labels and the online feature store only store the most recent version of the data.

SirOibaf · 2023-10-22T12:53:16Z

python/hsfs/feature_store.py

+        Instead of using a feature group to save a label/prediction target, you can use a spine together with a dataframe containing the labels and join keys for features in other feature groups.
+        A Spine is essentially a metadata object similar to a feature group, however, its data is not stored in the feature store.
+        The Spine stored in the feature store only contains the needed metadata such as the name, version, primary key column(s), and event time column. 
+        The Spine DataFrame is provided when you need to (1) create training data and (2) create batch inference data. The Spine DataFrame should also contain any join keys (primary keys to other feature groups) needed to join features included in a feature view containing the Spine group.If you don’t include the event_time in the Spine DataFrame (such as in batch inference), it will retrieve the latest feature value for that feature using the join key(s).


Are you sure about the last part? If you don’t include the event_time in the Spine DataFrame (such as in batch inference), it will retrieve the latest feature value for that feature using the join key(s).

I don't think that's the case. if you don't provide the event_time, I think Hopsworks falls back to non-pit join and it doesn't guarantee time ordering. You most likely end up with multiple rows for each pk in the spine feature group (one for each event in the joined feature group).

jimdowling added 2 commits October 21, 2023 10:14

Improve docs for spine groups

7e5561a

fix

e5b3afe

SirOibaf reviewed Oct 22, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FSTORE-1064] Improve docs for spine groups #1137

[FSTORE-1064] Improve docs for spine groups #1137

jimdowling commented Oct 21, 2023

SirOibaf commented Oct 22, 2023

SirOibaf Oct 22, 2023

SirOibaf Oct 22, 2023

[FSTORE-1064] Improve docs for spine groups #1137

Are you sure you want to change the base?

[FSTORE-1064] Improve docs for spine groups #1137

Conversation

jimdowling commented Oct 21, 2023

SirOibaf commented Oct 22, 2023

SirOibaf Oct 22, 2023

Choose a reason for hiding this comment

SirOibaf Oct 22, 2023

Choose a reason for hiding this comment