Please help! Unknown error when apfs-inspect

[zhanchengqian]

Hi,

My file system got damaged when I was trying to copy large trunk of files from the drive (apfs encrypted MacBook Pro ssd, I put it inside a third-party ssd-to-usb external case). When I was copying, suddenly it says the directory doesn’t exist, and soon I found out the folder I was copying disappeared.

I tried to do ls in terminal, but no result, it disappeared. Then that night, I downloaded disk drill. Disk drill shows the missing folder but after I recovered the files I found out 50% of the files are garbled, not opening. The other 50% is okay, but anything before November 2017 is non openable.

Then I ran across this GitHub page and I tried to use apfs-inspect and apfs-read, but no success. I think both of these returned ABORT unknown error. I don’t know what happened because the message offers little detail.

I mean these files are not something too important, but are memorable things that I kept during college. I really want to recover these data. The garbled portion is around 35GB. Please help me diagnose the issue. Is it possible to get more detail of why apfs-inspect failed?

Also just FYI, the recovered non-openable files are the same size as the good files. I also have very small portion of the garbled data on another drive (known to be good and can be opened), like 1GB out of the garbled 35GB. I compared the folders that are in common. for example, file A (from the other drive) is 35MB, and the garbled file A (what I recovered) is also 35MB. I compared the hex, the hex has nothing in common, totally garbled in the recovered file.

I was reading the drive on another Mac machine, High Sierra, and the ssd is connected thru ssd-usb3.0 external dongle. This MacBook ssd is extracted out of my MacBook, since it got damaged in a recent incident.

Please help me! I don’t know what caused the data damage, and I don’t understand why Disk Drill recovered some good files and some garbled files. Also, why the apfs-inspect and apfs-read didn’t work at all? I will provide screenshots if needed.

Let me know what I should do next.

Thanks, God bless you all

[zhanchengqian]

[zhanchengqian]

also, when I ran fsck it says Object map is invalid.

[jivanpal]

Output of diskutil list and diskutil apfs list would be helpful just to make sure we're not missing anything trivial.

There are a couple of possible (but exceedingly unlikely) error conditions for fopen() that I didn't handle, but have now added code for. Can you checkout the unstable branch, recompile, and try again? That may give us some info as to why you can't open /dev/disk4s1.

The actual "ABORT: Unknown error" message you are seeing appears when fopen() cannot open the file (/dev/disk4s1) for some unknown reason. I don't know what to do with that info (or rather, lack of info). My best guess is that your SATA-to-USB adapter (or something along the way, such as your USB controller/bus) is not working properly, and it may have caused the damage in the first place. Can you attach the drive directly to a machine's SATA port and try again?

I've been meaning to sort out this project for a long time now, but hey, life gets in the way. Hopefully I'll have a week or two in November to dedicate to working on this.

[zhanchengqian]

Thanks for the reply. I tried the unstable branch (on another mac), and still got the "ABORT unknown error". I assume that maybe it has something to do with the USB adapter. The thing is Apple uses its own proprietary hardware interface, so I have no way of directly attach the SSD to a machine's SATA port. It means I can only access it thru an adapter of some sort. I will try to get another adapter from another brand, and see if any thing different will appear.

Also, here are the outputs from diskutil: (shows as disk3s1 on this machine)

Thanks once again for your help, hope all the best!

[zhanchengqian]

I know it's kinda dumb but I ran the repair in disk utility and it gave me this. I don't know whether this would give some more info or not. But I wanna post anyway.

I also tried to unmount other volumes on the same SSD, and I also tried to do this in recovery mode. It always give me the same error message.

[jivanpal]

here are the outputs from diskutil: (shows as disk3s1 on this machine)

disk3s1 is not the APFS partition/container itself; it is the Data volume within your APFS container. The container itself is disk2s2 (see "Physical store" in the output of both commands). Try that first.

I ran the repair in disk utility and it gave me this.

This is telling you that a repair cannot be done whilst you are running macOS normally, since one of the volumes is mounted. I would clone the partition or whole disk to another disk (in case the repair actually worsens things) and then attempt repair from macOS Recovery. Hold CMD+R when booting, then open Terminal and find the physical store ID via diskutil list (e.g. as disk2s2 is currently). Unmount all mounted volumes belonging to the APFS container with diskutil unmountDisk diskXsY, and then run e.g fsck_apfs -y /dev/rdiskXsY, where X and Y are from the physical store ID. Note, that's rdisk in the latter command, not just disk.

[zhanchengqian]

Hi jivanpal, thanks for the reply.

Now I have a new adapter, and now I understand that it's actually drive disk2s2. Here is what happened when I ran diskutil verifyVolume:

~ diskutil verifyVolume /dev/disk2s2
Started file system verification on disk2s2
Verifying storage system
Performing fsck_apfs -n -x /dev/disk2s2
Checking the container superblock
Checking the EFI jumpstart record
Checking the space manager
Checking the space manager free queue trees
Checking the object map
error: (oid 0xd55eb) om: invalid o_xid (0x6a1c29)
error: (oid 0xd55eb) om: invalid o_type (0x40000003, expected 0x4000000b)
error: verification/reading of the omap object failed: Illegal byte sequence
The volume /dev/disk2s2 could not be verified completely
Storage system check exit code is 8
Error: -69716: Storage system verify or repair failed
Underlying error: 8

Then I ran these commands, for you reference. (I am a CS grad student, but please forgive me that I don't have a good understanding of what's going on in the Apple APFS system, please point out any problem here)

➜  bin sudo ./apfs-list /dev/disk2s2 0 /Users/Encholl
Opening file at `/dev/disk2s2` in read-only mode ... OK.
Simulating a mount of the APFS container.
Validating checksum of block 0x0 ... OK.
Locating the checkpoint descriptor area:
- Its length is 280 blocks.
- It is contiguous.
- The address of its first block is 0x10647c.
Loading the checkpoint descriptor area into memory ... OK.
Locating the most recent well-formed container superblock in the checkpoint descriptor area:
- It lies at index 201 within the checkpoint descriptor area.
- The corresponding checkpoint starts at index 200 within the checkpoint descriptor area, and spans 2 blocks.

Loading the corresponding checkpoint ... OK.
- There are 68 checkpoint-mappings in this checkpoint.

Reading the Ephemeral objects used by this checkpoint ... OK.
Validating the Ephemeral objects ... OK.
The container superblock states that the container object map has Physical OID 0xd6f38.
Loading the container object map ... OK.
Validating the container object map ... OK.
Reading the root node of the container object map B-tree ... OK.
Validating the root node of the container object map B-tree ... OK.
The container superblock lists 5 APFS volumes, whose superblocks have the following Virtual OIDs:
- 0x403
- 0x54961
- 0x54963
- 0x5ee18
- 0x3cc230

Reading the APFS volume superblocks ... OK.
Validating the APFS volume superblocks ... OK.

 Volume list
================
 0: Macintosh HD - 数据
 1: Preboot
 2: Recovery
 3: VM
 4: Macintosh HD
The volume object map has Physical OID 0xd6edc.
Reading the volume object map ... OK.
Validating the volume object map ... OK.
Reading the root node of the volume object map B-tree ... OK.
Validating the root node of the volume object map B-tree ... OK.
The file-system tree root for this volume has Virtual OID 0x404.
Looking up this Virtual OID in the volume object map ... corresponding block address is 0x6250.
Reading ... validating ... FAILED.
Going back to look at the previous checkpoint instead.
➜  bin

[zhanchengqian]

The output when running apfs-inspect:

inspect-output.txt

Also I tried to read somewhere around 0xa4bf0 ~ 0xa4bf2, which is the address I found some of the missing files via Disk Drill. And I got these two results, and it seems something wrong happened around these addresses, I assume?

➜  bin sudo ./apfs-read /dev/disk2s2 0xa4bf0

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xa4bf0 ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0xa4bf0:
--------------------------------------------------------------------------------
Stored checksum:    0xbbfcbad22880ea15
OID:                0x5a217bf9d6cd0011
XID:                0x71f860f74621dcab
Storage type:       Ephemeral
Type flags:         Non-persistent (should never appear on disk --- if it does, file a bug against the APFS implementation that created this object)
Type:               Unknown type (0x%08x) --- perhaps this type was introduced in a later version of APFS than that published on 2019-02-27.
Subtype:            Unknown subtype (0x00003c3f) --- perhaps this subtype was introduced in a later version of APFS than that published on 2019-02-27.

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.
➜  bin sudo ./apfs-read /dev/disk2s2 0xa4bf1

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xa4bf1 ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0xa4bf1:
--------------------------------------------------------------------------------
Stored checksum:    0x0000000000000000
OID:                0x0
XID:                0x0
Storage type:       Virtual
Type flags:         (none)
Type:               (none/invalid)
Subtype:            (none/invalid)

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.
➜  bin

[zhanchengqian]

Running the fsck gives me this:

fsck_apfs -y -x /dev/rdisk3s1
Checking the container superblock.
Checking the EFI jumpstart record.
Checking the space manager.
Checking the space manager free queue trees.
Checking the object map.
error: (oid 0xd55eb) om: invalid o_xid (0x6a1c29)
error: (oid 0xd55eb) om: invalid o_type (0x40000003, expected 0x4000000b)
error: verification/reading of the omap object failed: Illegal byte sequence
The volume /dev/rdisk3s1 could not be verified completely.

In all, I know there is /Users/Encholl folder somewhere near 0xa4bf2, (I got that address from Disk Drill), and what kind of "next step" should I take? I hope I've provided enough info for you to find what the problem is. I will provide more info as you request. I lack some knowledge about this kind of file system. I will be very grateful if you could give me some more insight into how I could recover some of the files near that address in the drive.

Thanks for your help again, I think I cannot find another knowledgeable guy like you on the Internet.
Hope you have a great day!

[jivanpal]

Your volume's object map block is at address 0xd6edc. This block only stores metadata about the object map; it isn't the object map B-tree itself. However, it tells you where the B-tree is. Reading that block with apfs-read should reveal the address of the object map B-tree's root node. Substitute the address you find for <omap root node address> in the commands below.

---

We want to use apfs-recover to grab your files from /Users/Encholl. Unfortunately, apfs-recover always traverses from / down the tree, but the address that virtual OID 0x404 (the root of the volume) resolves to is an invalid block, so apfs-recover will currently encounter an error because it cannot find /.

We can either:

(a) Check whether virtual OID 0xa4bf2 is really /Users/Encholl, after which we can modify apfs-recover to scan from there rather than from /; or

(b) Fix the blocks which represent the filesystem root, /, /Users, and /Users/Encholl, then use apfs-recover in its current form.

---

For (a), the way forward is:

1. Check block 0xa4bf2 with apfs-read. Please provide this output, as it seems to be missing from the output you pasted in your previous comment.
2. If it really is a filesystem B-tree node, check its contents with apfs-explore-fs-tree /dev/disk2s2 0xa4bf2 <omap root node address>.
3. If it really contains entries from /Users/Encholl, modify apfs-recover to fetch files using this info.
4. Use apfs-recover to grab your files.

---

For (b), the way forward would be:

1. Correct the block with virtual OID 0x404, which represents the root of your Data volume. This block is distinct from the root of your filesystem, /; in fact, this block lists the virtual OID of / (usually 0x2) and another entity called private-data (usually 0x1).
2. Correct the blocks which represent /, /Users, and /Users/Encholl, if necessary.
3. Use apfs-recover to grab your files from /Users/Encholl.

Ideally, steps (1) and (2) would be automated, but these tools are not sophisticated enough to do this (implementing this is the main thing I would like to work on right now), as this can be quite case-specific. The apfs-explore-* commands allow you to explore B-trees (object maps and filesystem records), and apfs-search allows you to scan the disk to look for blocks with certain properties (e.g. if you are hunting for a specific dentry or virtual OID) in order to find the relevant info to do these steps manually instead. Manual fixing of blocks is done with the aid of apfs-modify (use with extreme caution, it WILL irrevocably overwrite data on disk). This whole process can be extremely time-consuming.

The first thing to do would be to see whether a valid instance of the block with virtual OID 0x404 can be found. You can explore the object map B-tree in order to look for the physical address of blocks with a given virtual OID. Do this with apfs-explore-omap-tree /dev/disk2s2 <omap root node address>.

---

Best of luck!

[zhanchengqian]

Thanks for the reply, here is the 0xa4bf2 apfs-read result:

➜  bin sudo ./apfs-read /dev/disk2s2 0xa4bf2

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xa4bf2 ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0xa4bf2:
--------------------------------------------------------------------------------
Stored checksum:    0x0000000000000000
OID:                0x0
XID:                0x0
Storage type:       Virtual
Type flags:         (none)
Type:               (none/invalid)
Subtype:            (none/invalid)

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.

[zhanchengqian]

To be specific, I tried to read multiple addresses, and arrived at these results:

0xa4bf0 (checksum OID XID looks good)
0xa4bf1 (checksum OID XID All zeroes)
0xa4bf2 (checksum OID XID All zeroes)
...
...
0xd4155 (checksum OID XID All zeroes)
0xd4156 (checksum OID XID All zeroes)
0xd4157 (checksum OID XID looks good)

By "looks good" I mean they have some values instead of zeroes.

[zhanchengqian]

Here is the object map info:

➜  bin sudo ./apfs-read /dev/disk2s2 0xd6edc

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xd6edc ... validating ... OK.

Details of block 0xd6edc:
--------------------------------------------------------------------------------
Stored checksum:    0x5184bf83edf6462b
OID:                0xd6edc
XID:                0x6a1c6d
Storage type:       Physical
Type flags:         (none)
Type:               Object map
Subtype:            (none/invalid)
Flags:
- No flags are set.
Object mappings tree:
- Storage type:         Physical
- Type flags:           (none)
- Type:                 B-tree (root node)
- Object ID:            0xd6ef8
Snapshots tree:
- Storage type:         Physical
- Type flags:           (none)
- Type:                 B-tree (root node)
- Object ID:            0x0
- Number of snapshots:  0 snapshots
- Latest snapshot XID:  0x0
In-progress revert:
- Minimum XID:          0x0
- Maximum XID:          0x0
--------------------------------------------------------------------------------

END: All done.
➜  bin

[zhanchengqian]

So from the following result, I assume that 0xa4bf2 is not really the address I want?
It does not bring any useful result and thus it seems it's not the file-system tree root node.

➜  bin sudo ./apfs-explore-fs-tree /dev/disk2s2 0xa4bf2 0xd6ef8

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading the file-system tree root node (block 0xa4bf2) ... validating ... FAILED.

Reading the object map root node (block 0xd6ef8) ... validating ... OK.


Node details:
--------------------------------------------------------------------------------
Stored checksum:    0x0000000000000000
OID:                0x0
XID:                0x0
Storage type:       Virtual
Type flags:         (none)
Type:               (none/invalid)
Subtype:            (none/invalid)
Flags:                          (none)
Number of child levels:         0
Number of keys in this node:    0
Location of table of contents:
- Offset from start of node data area:  0x0 = 0
- Length (bytes):                       0x0 = 0
Location of key–value shared free space:
- Offset from start of keys area:       0x0 = 0
- Length (bytes):                       0x0 = 0
--------------------------------------------------------------------------------


Node has 0 entries, as follows:
Assertion failed: (node->btn_nkeys > 0), function main, file src/apfs-explore-fs-tree.c, line 160.
[1]    20545 abort      sudo ./apfs-explore-fs-tree /dev/disk2s2 0xa4bf2 0xd6ef8

[zhanchengqian]

output relates to 0x404:

➜  bin sudo ./apfs-read /dev/disk2s2 0x404
Password:

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0x404 ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0x404:
--------------------------------------------------------------------------------
Stored checksum:    0x3a177b0ee6eef00c
OID:                0x44cd78635880773e
XID:                0x3657f74039de0215
Storage type:       Ephemeral
Type flags:         (none)
Type:               Unknown type (0x%08x) --- perhaps this type was introduced in a later version of APFS than that published on 2019-02-27.
Subtype:            Unknown subtype (0x00000cf8) --- perhaps this subtype was introduced in a later version of APFS than that published on 2019-02-27.

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.
➜  bin sudo ./apfs-explore-omap-tree /dev/disk2s2 0x404

Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0x404 ... validating ... FAILED.


Object map trees don't have variable size keys and values ... do they?
➜  bin

[zhanchengqian]

Oh one more thing here, I have a small portion of files that I have copied years before the incident, and after comparing the two files (one I stored elsewhere, and the other one is recovered using DiskDrill), I found the compositions of the files are wildly different. Here are two files (one good and one bad) for your inspection: two files.zip

Maybe somehow the extraction process got mislead and got offset by a few blocks? I don't know.

Here is the full output of apfs-inspect: (I notice that last time I didn't post the full inspect result, because my terminal caps at 1000 lines) apfs-inspect-full.txt

[jivanpal]

I was just reading back through the thread, and noticed that I previously said:

We can either:

(a) Check whether virtual OID 0xa4bf2 is really /Users/Encholl, ...

---

For (a), the way forward is:

1. Check block 0xa4bf2 with apfs-read. ...

The instructions I gave were incorrect, apologies for the mistake! I said to check Virtual OID 0xa4bf2, but incorrectly gave instructions for what to do if the object with Physical OID 0xa4bf2 was an FS B-tree node containing records for /Users/Encholl. There may be something we can do here. You said you obtained that ID from Disk Drill, but didn't say how; I assumed it was a Virtual OID, but it could be something else, like a Physical OID or FS OID. Can you clarify what sort of ID it is — a screenshot of the Disk Drill output may help. It's probably a Virtual OID, but let's make sure.

[zhanchengqian]

Hi jivanpal,

This would be something interesting: DDLogExcerpt.txt

(I cut this part out of the entire log which is quite long, and it shows where it found to be problematic) I will send an email to you containing two full logs that I got from DiskDrill. (I don't know if they contain any privacy data so I am sending these through email rather than posting all of them here)

Here is a screenshot of DiskDrill with the 0xa4bf2 part highlighted. (I am not sure if it's really 0xa4bf2. I was assuming it to be address 0xa4bf2 since it says 丢失00000000000a4bf2 (which means lost00000000000a4bf2) on the screen)

[jivanpal]

Thanks for providing the logs, I received your email 🙂 Unfortunately, there doesn't seem to be much there that I can use — there is also no mention of a4bf2 anywhere in either log.

It certainly will not be block address / Physical OID 0xa4bf2, as we have already checked that. I can only assume that it is Virtual OID 0xa4bf2. We should see what Virtual OID 0xa4bf2 is mapped to, if anything. We know your volume's Omap has Physical OID 0xd6edc, and that Omap's B-tree root node has Physical OID 0xd6ef8. Do the following:

apfs-explore-omap-tree /dev/disk2s2 0xd6ef8

Then you will be presented with a list of the keys in the Omap B-tree's root node, which should be in ascending order of OID. Find the highest listed OID that is less than or equal to 0xa4bf2, then type that entry's index and press ENTER. You will descend that node and be presented with another list of keys in that child node. Repeat this until you are presented with some KEY/VALUE info and the program exits. Please provide all of the output.

[zhanchengqian]

Thanks for the reply,

I am getting these: explore-omap-output.txt
It seems that there is no exact virtual OID 0xa4bf2, the closest I am getting here is :

Choose an entry [0-69]: 59
KEY:
  - OID:                            0xa4b15
  - XID:                            0x60c1d

VALUE:
  - Object size:                    4096 bytes
  - Object address in container:    0x3706

I guess 0xa4bf2 might be misleading us here?

[jivanpal]

It certainly seems like it... let's ignore that, then, and focus on searching for specific dentries. I'll be working on search so that this should be easier for you. For now, let's see what's wrong with the filesystem root tree, which has Virtual OID 0x404. Search for that in the same way as you did with 0xa4bf2, using the same command. That is, apfs-explore-omap-tree /dev/disk2s2 0xd6ef8. As before, please provide all of the the output.

---

You may have noticed that I have renamed the tool to Drat and released v0.1.0. If you are using the new code, there is now only a single drat command, and you will instead need to run drat explore-omap-tree /dev/disk2s2 0xd6ef8.

In order to pull the new code, you will need to update your Git remote and change branch to main. It is probably simpler to just delete your current copy and clone the current codebase to a new directory using the new repo URL (git clone https://github.com/jivanpal/drat.git). Alternatively, you can avoid compiling and just download the binary from the releases page. I would be grateful to know if using that pre-compiled binary causes you any issues.

[zhanchengqian]

Hi Jivan,

Running the code gives me this output: 0x404-output.txt
I was using the new drat binary and didn't cause any issue so far.

It arrives at:

Choose an entry [0-59]: 0
KEY:
  - OID:                            0x404
  - XID:                            0x6a1c6d

VALUE:
  - Object size:                    4096 bytes
  - Object address in container:    0x6250

In the log that I provided you yesterday, there were things like Could not load 21-th checkpoint at position 0x1064a3000. Omap btree descriptor is broken. Could not load 72-th checkpoint at position 0x106500000. No NXSB magic signature. Cyclic tree reference detected for block 0x80010 in omap (store). Would these help us in any way?

Thanks for the support!

[jivanpal]

The detailed output you've provided shows that block 0x6250, which the Omap maps Virtual OID 0x404 to, doesn't actually contain data for a block with Virtual OID 0x404 (Target's actual OID = ... (   NO)). This seems to be the case for basically all of the entries in the Omap which we have seen so far; it seems the the Omap is completely unreliable. We will have to resort to finding blocks with certain data we are interested in by hand, which will require scanning the whole partition each time, though we can narrow down the search space with a first pass. My planned work on search should make this easy to do.

I was using the new drat binary and didn't cause any issue so far.

Lovely, good to know!

In the log that I provided you yesterday, there were things like ...

These are remarks about things which are not as they should be, but they are not really useful to us. In particular, your inspect output shows that the latest checkpoint is completely valid, so we are not really interested in older checkpoints. When we scan the whole partition with search, we will see objects with all manner of XIDs anyway; we will not be using the checkpoints or omaps in any way.

[zhanchengqian]

Oh gosh, so the omap is completely useless in this case? How come it got completely damaged? I don't quite understand, uh. I will certainly be waiting for your further development on the search feature, and hopefully the data that I'm interested in is still there for me to recover. Oh btw, do you think that search development would take several weeks of time or maybe several months? Just want to know a rough estimation, no rush though.

Thanks for your reply again!

[jivanpal]

At least the most recent Omap is; we can check the older ones, too. I will probably add such checks to inspect so that we can see which Omaps have the most correct entries. I will be working regularly on this, focusing on bugfixes, refactoring, and adding proper file size and block size support first, which will probably take 2–4 weeks to finish. Doing search will probably take another week after that.

You can keep an eye on my Trello board if you wish.

[zhanchengqian]

Great! I will be following your Trello board. Hopefully everything goes well with your upcoming developments.

I agree that it would a good idea to include checking older omaps with inspect. I will check back at this project from time to time. If there's any other things I can do with my current disk, feel free to let me know.

Thanks and wish you have a great day.

[jivanpal]

Hello, unrelated to your issue, but I have released v0.1.1 and would like to check a bugfix. Here, you ran the command sudo ./apfs-read /dev/disk2s2 0xa4bf0, which has a bug in the output (Unknown type (0x%08x)). Can you run the command again with v0.1.1, please? That is, sudo drat read /dev/disk2s2 0xa4bf0. The output should show an actual hex number instead of %08x.

[zhanchengqian]

Hi Jivan, here is the output:

➜  drat-new sudo ./drat-0.1.1-darwin-amd64 read /dev/disk2s2 0xa4bf0
Password:
Opening file at `/dev/disk2s2` in read-only mode ... OK.

Reading block 0xa4bf0 ... validating ... FAILED.
The specified block may contain file-system data or be free space.

Details of block 0xa4bf0:
--------------------------------------------------------------------------------
Stored checksum:    0xbbfcbad22880ea15
OID:                0x5a217bf9d6cd0011
XID:                0x71f860f74621dcab
Storage type:       Ephemeral
Type flags:         Non-persistent (should never appear on disk --- if it does, file a bug against the APFS implementation that created this object)
Type:               Unknown type (0x00009e2c) --- perhaps this type was introduced in a later version of APFS than that published on 2019-02-27.
Subtype:            Unknown subtype (0x00003c3f) --- perhaps this subtype was introduced in a later version of APFS than that published on 2019-02-27.

--- This tool cannot currently display more details about this type of block ---
--------------------------------------------------------------------------------

END: All done.

[jivanpal]

Lovely, thanks!

[zhanchengqian]

Hi Jivan,

Just wondering if everything goes well? Is there a new ETA on development of search / modify ?

Wish you have a great day!

[jivanpal]

Hi @zhanchengqian, I believe we have discussed this before. Since your filesystem tree is corrupted, you are missing records for files, and if those files are fragmented (that is, their file extents are not contiguous on disk), then Disk Drill can only identify blocks of data by scanning the disk. If it happens to find separate chunks of data which belong to the same file, it has no way of knowing this, so cannot combine them, leaving you with file fragments rather than whole files.

[zhanchengqian]

Ah, got it. Would your tool be able to deal with this kind of fragmentation problem? Or maybe I can use your tool and somehow manually stitch fragments together? Just wondering... 🤔

[jivanpal]

Not magically by just looking at file fragments and deciding whether they go together.

The only way to do this is to reconstruct an object mapping tree and filesystem tree for a particular transaction ID. If the data on disk is sufficiently malformed, then there is no hope of performing such a reconstruction. However, this does not mean that a sufficiently intact tree needs to still exist on disk; in the worst case, even if no sufficiently intact object mapping trees or filesystem trees exist on disk, one can scan the entire disk for all sufficiently intact filesystem tree leaf nodes in order to find all intact filesystem records, and then construct new trees or otherwise explore the filesystem based on the info available that way.

I do not know if Disk Drill has proper support for past APFS transactions/checkpoints.

[zhanchengqian]

Thanks for the reply.

I think DiskDrill only has basic recovery methods, currently there is really no support for past transactions. Would you consider adding support for searching old omap trees? My understanding would be that my case is quite tough to deal with, since it appears to be that great number of files are fragmented and also the omap is in bad state. In this case, your search feature will also not be able to fetch a file in one piece, right?

[jivanpal]

The plan is to include scanning of past checkpoints/transactions in inspect, and have it summarise diagnostic info and suggestions on how to resolve the problem or otherwise recover the data, e.g. in your case, No good object maps found. Index the disk now to try and reconstruct an object map? (Estimated completion time: 16 hours) [y, n]: _. In the worst case, where no Omap tree or FS tree can be reconstructed, it will offer to construct ones from the available FS tree leaf nodes, which contain the records which list the file extents. The file extents are what specify the locations of each file's fragments (extents), so unless these are corrupted, your files will be recoverable.

The search command will be able to search for any type of block (or data contained therein) as specified by the user, e.g. drat search --volume-superblocks --volume-name 'macOS - Data', drat search --fs-records --fso-parent-id 0x123, etc. This searching will be sped up by first indexing the disk if not already done, else each search would take e.g. 16 hours every time.

I am currently working on specifying this desired functionality/behaviour in the docs for v0.2 (see latest draft here, content is in the docs-for-v0.2 branch), most of which will hopefully be implemented in v0.2.0; if not, it will be included in point releases.

[jivanpal]

Hi @zhanchengqian, apologies for my absence in recent months. I am dedicating time to working on Drat again, and hope to finish work on v0.2 before the end of the year. If you would still like assistance with this issue, please reply and I will keep you updated on when tree reconstruction is implemented. If not, I will lock this thread soon.

[jivanpal]

Locking thread now. Please free free to start a new thread in the future.