Thread with 10 posts
jump to expanded post
I am trying to do a backup of my iPhone to my MacBook. it's around 90GB of data. there is not enough space on my MacBook's SSD, so I've made it go to an external, USB3 hard drive instead. copying the data to the PC took several hours, but the final step seems to take forever…
I was worried it had gotten stuck, so I used the sampling option in Activity Monitor to see what it was doing. a rename() call? okay, maybe one of those got stuck, I thought.
I wanted to be sure that wasn't it, so I used dtruss to look at the syscalls.
It's not stuck.
it has thousands, maybe millions, of files. they're named like
68/68a3df34dff1713f0307bbca1832901f623293a3
the directory name always matches the first two letters of the filename. they're encrypted blobs.
it is moving the files from Backup/…/Snapshot/ to Backup/…/
since this is a completely fresh backup, I think it's moving every single file in this way.
the most efficient way to do this would be to move one directory at a time. 256 rename() calls at worst.
that is not what it is doing.
it is calling rename() for each individual file. in seemingly random order. synchronously.
and my poor external hard drive is managing about one file rename per second.
😭
OK so I did some testing and found that moving a whole directory's files at once, even individually, is way faster. it's jumping between directories that's slow.
I looked at the plist files and the how libimobiledevice (OSS) deals with them and.… I'm daring to take a shortcut 🤞
ADVANCED HIGH-PERFORMANCE ALGORITHM:
for i in Snapshot/*/; do pushd $i; LIST=ls; echo $LIST; mv $LIST ../../basename $i/; popd; done
it finished! I did mv Snapshot/Manifest.db ., mv Snapshot/Manifest.plist . and plutil -replace SnapshotState -string finished Status.plist, and now Apple's backups manager sees the backup, but it doesn't associate it with my device yet. a challenge for tomorrow I guess.
…okay, I don't know how to solve that challenge. I couldn't figure out where the relevant state was stored :(
I decided to just do some desperate shuffling of files between drives and do the backup to the SSD, moving to the HDD only later. wow, it got to 25% complete in minutes
I think the copying is slowed down by the same thing that renaming was. it doesn't copy the files in a HDD-friendly order. the problem is that the naming of files on the device is essentially randomised relative to the host filesystem, since all the names get hashed.