appcache: add new magic for Steam Client Beta (June 2024)

[sonic2kk]

Fix #462.

The latest Steam Client Beta changed the magic for appinfo.vdf. This PR adds the new magic to the file. I confirmed that this allows my appinfo.vdf to be parsed, but have not checked for backwards compatibility (although it should work given that we check based on a tuple of magics).

Unsure how the data_sha1 should look, and if the if check needs to now include the new magic too. Alternatively the if could be changed to if magic != b"'DV\x07", but have not tested this.

I am using this as a reference implementation, although my C# skills are close to zero 😅 SteamDatabase/SteamAppInfo@56b1fec

[sonic2kk]

Hm, this might need more work. Converting to draft.

[sonic2kk]

Getting closer, seems like data_sha1 is not being read properly and may be why binary_load is not working to load the data. But the other information seems correct for the first app:

{
    'appid': 5,
    'size': 85,
    'info_state': 1,
    'last_updated': 1680901338,
    'access_token': 0,
    'sha1': b'\x87\xfaCg\x85\x80\r\xb4\x90Im\xdc}\xb4\x81\xeeQ\x8b\x825',
    'change_number': 22534048,  # Matches SteamDB: https://steamdb.info/app/5/info/
    'data_sha1': b'~\xa7WO\xe35\xb5\x07~k>\xba|B\xc8\x90\x1e\xab\x91\xa9'  # Does not match 'sha1'
}

[sonic2kk]

Didn't really make much progress tonight. Remaining issues are:

• data_sha1 does not appear to be correct, though all other data is strangely?
• binary_load fails to parse the remaining bytes. Best case, fixing the data_sha1 to read correctly will fix this. Worst case, binary_vdf needs a separate patch.

If anyone has any pointers I'd love to hear them!

[Matoking]

EDIT: This is partly incorrect. Read the next comment.

---

Had a closer look at this since the Steam beta change broke Protontricks' own appinfo.vdf parser (each binary VDF was parsed using vdf, however). Keep that in mind, as I don't have any experience with the steam library. Slicing each binary VDF segment and parsing them using vdf.binary_loads() worked fine until now.

The new appinfo.vdf V29 seems to have changed in several ways. The file header is now 8 bytes longer and appears to be:

uint32   - MAGIC: ")DV\x07"
uint32   - UNIVERSE: 1
uint64   - FIELD MAP OFFSET: 13352136 (0xcbbcc8) NEW

Note that the first byte of MAGIC is the format version. ) corresponds to 29, and it was ( (28) previously.

---

The field map offset points to a new segment at the very end of the appinfo.vdf that contains an array of field names. Looking up data from that offset brings us this (truncated for brevity):

00cbbcc0: 0008 0808 0000 0000 443d 0000 6170 7069  ........D=..appi
00cbbcd0: 6e66 6f00 6170 7069 6400 7075 626c 6963  nfo.appid.public
00cbbce0: 5f6f 6e6c 7900 636f 6d6d 6f6e 006e 616d  _only.common.nam
00cbbcf0: 6500 7479 7065 0061 7373 6f63 6961 7469  e.type.associati
00cbbd00: 6f6e 7300 6761 6d65 6964 0065 7874 656e  ons.gameid.exten
00cbbd10: 6465 6400 6265 7461 5f6e 616d 655f 3100  ded.beta_name_1.
00cbbd20: 6265 7461 5f63 6f6e 7465 6e74 6c69 7374  beta_contentlist
00cbbd30: 5f31 0062 6574 615f 6e61 6d65 5f32 0062  _1.beta_name_2.b
00cbbd40: 6574 615f 636f 6e74 656e 746c 6973 745f  eta_contentlist_
00cbbd50: 3200 6265 7461 5f6e 616d 655f 3300 6265  2.beta_name_3.be
00cbbd60: 7461 5f63 6f6e 7465 6e74 6c69 7374 5f33  ta_contentlist_3
00cbbd70: 0062 6574 615f 6e61 6d65 5f34 0062 6574  .beta_name_4.bet
00cbbd80: 615f 636f 6e74 656e 746c 6973 745f 3400  a_contentlist_4.
00cbbd90: 636f 6e76 6172 5f75 6763 6364 6e72 6570  convar_ugccdnrep
00cbbda0: 6f72 7470 6374 0063 6f6e 7661 725f 616c  ortpct.convar_al

The struct for each app section seems unchanged, but the binary_vdf seems to have changed as vdf.binary_loads can no longer parse them. Each binary_vdf is preceded with a SHA1 checksum. Calculating it ourselves - once we adjust for the file header being 8 bytes longer - matches what's in the app section, so it doesn't seem to be due to wrong section of data being parsed. This is what the binary VDF segment for app 7 looks like now (truncated again for brevity):

00000000: 0000 0000 0002 0100 0000 0700 0000 0003  ................
00000010: 0000 0001 0400 0000 5374 6561 6d20 436c  ........Steam Cl
00000020: 6965 6e74 0001 0500 0000 436f 6e66 6967  ient......Config
00000030: 0000 0600 0000 0802 0700 0000 0700 0000  ................
00000040: 0800 0800 0000 0109 0000 0053 7465 616d  ...........Steam
00000050: 2042 6574 6120 5570 6461 7465 0001 0a00   Beta Update....
00000060: 0000 7374 6561 6d65 7865 2070 7562 6c69  ..steamexe publi
00000070: 6362 6574 6100 010b 0000 0053 7465 616d  cbeta......Steam
00000080: 2046 616d 696c 6965 7320 4265 7461 0001   Families Beta..
00000090: 0c00 0000 7374 6561 6d65 7865 2070 7562  ....steamexe pub
000000a0: 6c69 6362 6574 615f 6661 6d69 6c69 6573  licbeta_families
000000b0: 0001 0d00 0000 5374 6561 6d20 4465 636b  ......Steam Deck
000000c0: 2053 7461 626c 6500 010e 0000 0073 7465   Stable......ste
000000d0: 616d 6578 6520 7374 6561 6d64 6563 6b5f  amexe steamdeck_
000000e0: 7374 6162 6c65 0001 0f00 0000 5374 6561  stable......Stea
000000f0: 6d20 4465 636b 2042 6574 6100 0110 0000  m Deck Beta.....
00000100: 0073 7465 616d 6578 6520 7374 6561 6d64  .steamexe steamd
00000110: 6563 6b5f 7075 626c 6963 6265 7461 0002  eck_publicbeta..

The appinfo.appid field seems to have the index 02 01, common.name has 01 04, and common.type has 01 05.

So, looks like each field name is no longer written in full each time. Instead, the mapping at the end of the appinfo.vdf needs to be used by the parser.

My analysis might be off, so keep that in mind. :)

[Matoking]

Wrote the analysis before looking up the SteamAppInfo PR; they seem to have figured out the new format, and they call the field map "string pool" instead. Oh well, was fun figuring this one out. :)

I guess it would be possible to provide the array of field name strings (i.e. the string pool, table or whatchacallit) as a parameter to vdf.binary_load. It would prevent Protontricks from having to add steam as a dependency, which comes with a set of dependencies of its own, which is overkill since we only need to parse the appinfo.vdf file. That said, it's kind of an awkward solution since the binary VDF segments are no longer self-contained.

[Matoking]

Hacked together quick support in vdf here:

https://github.com/Matoking/vdf/tree/support_new_bvdf

and in steam here, based on the current PR (thanks!):

https://github.com/Matoking/steam/tree/appinfo_v29

It's somewhat ugly, but seems to work. @rossengeorgiev is probably best qualified to answer how this "binary VDF requires separate table to parse" stuff should be handled in the API.

[sonic2kk]

Legend! Thanks for this, I don't really know much about reading from binary files but I have certainly learned a lot from your explanation and branches!

[sonic2kk]

However we want to handle this going forward is fine with me; I can close this PR and @Matoking you can open a PR with your changes as you already have the changes ready to go for vdf as well. I'm just glad my hackings were able to be of some use 😄 We ultimately have the same goal of getting this fixed here so any approach is good with me :-)

[Matoking]

However we want to handle this going forward is fine with me; I can close this PR and @Matoking you can open a PR with your changes as you already have the changes ready to go for vdf as well. I'm just glad my hackings were able to be of some use 😄 We ultimately have the same goal of getting this fixed here so any approach is good with me :-)

I squashed the commits together, added you as a co-author and created the PR #464. Does this work for you?

[sonic2kk]

Yes, that works perfectly. Thank you and I appreciate being added as a co-author too 😄

I will close this PR in favour of #464.

[Matoking]

Great, thanks for the help! 👍