My iCloud Sync Failure Saga

Published

It was mid 2019 and because my 2016 13” MacBook Pro was a secondary computer (to a 2018 Mac mini), I lasted only a few days past WWDC before installing the first developer beta of macOS Catalina. It was rough, to say the least, and many things didn’t work or only worked intermittently. Several betas later I noticed that my iCloud documents were not syncing from the laptop but, well, that’s what you get with betas, right?

It was after I had installed the final version of Catalina that I noticed my chosen writing app, Ulysses, did not seem to be syncing properly via iCloud. I got in touch with Ulysses support who worked diligently with me to identify the problem was iCloud. Recalling the earlier issues, I looked more closely and discovered that original problem had remained, and I set about figuring out exactly what was… and wasn’t… happening.

The situation was this: Any file that was created or modified on the MacBook Pro would never make it to the cloud, and thus other devices. Directories were fine, deletions of anything were fine, and any action taken on any other device were fine. To put it another way, iCloud was syncing every which way as it should, except any content originating from the MacBook Pro. This did not affect all iCloud content, however. Photos, calendars, notes and more would sync just fine. It was only files in iCloud Drive, and some apps’ documents, including Ulysses but not KeepIt. The difference between these apps is in how they use iCloud — the old way (Ulysses) or the new way (KeepIt).

On 26 November 2019 I reported the problem to Apple. I quickly made it to the first of several Senior Advisors who would take me through a series of troubleshooting steps. This basically consisted of an endless series of questions and instructions on gathering copious logs and recreating the problem with time-stamped notes and screen captures, all to be sent to “Engineering”. After spending from 15-60 minutes on the phone with a Senior Advisor, I’d be lucky to get a call back several weeks later asking me to do it all again but slightly differently. If I was unlucky I’d get no response and I’d have to call them back after 3 weeks. At which point I’d likely get a new Senior Advisor. The penultimate such advisor promised to call me back but instead left the company and failed to hand over my case to anyone! On the 27 March 2020 — four months after I reported the issue — I got several messages saying Apple tried to call but I had not answered my phone. Well, that’s because it never rang1. When my attempts to follow up on this met with silence, I gave up. I wiped the laptop, sold it, and bought an iPad Pro to replace it.

It is important to note here that among the troubleshooting steps I performed were three complete wipes of the laptop followed by internet installs of macOS — both Catalina (twice) and the laptop’s original OS, High Sierra — and the problem persisted. This is part of why I gave in and got rid of the laptop. Like a cockroach, the issue survived even the nuclear option.

From May 2020 I hugely enjoyed my 11” iPad Pro with Magic Keyboard. It was fantastic for writing (Ulysses worked very well on it), superb for TV watching (I watched the entire run of Long Way Up on it — stunning visuals!), and pretty decent for most other things. But… I couldn’t fiddle with Xcode. I couldn’t use my preferred photo processing software. I often hit the “iPadness” of iPadOS that just makes some tasks that much harder. For instance, I could never get the hang of the Affinity Photo interface on iPad. I’m a Mac person at heart.

So… when the Apple Silicon Macs were announced in November 2020, I decided it was time to ditch the iPad Pro lifestyle and return to the fold with a new 13” MacBook Pro. This laptop is everything I hoped it would be. It’s incredible. I charge it like I charged my iPad. Overnight. It’s fast. The keyboard is superb. The screen is gorgeous. My photos really pop, and I’ve watched the entire first season of For All Mankind on it.

So… imagine my horror when I realised a couple of weeks after making it my primary Mac (with the 2018 Mac mini likely to be sold as the M1 runs rings around it), that files which are created or modified on my brand new M1 MacBook Pro, running Big Sur, do not make it to iCloud. It is exactly the same problem. Different OS, different laptop, same problem! I actually felt slightly sick on the realisation.

A little over 13 months after originally reporting the problem, I reported it again on 30 December 2020. I’ve had a few different Senior Advisors again, but all of them have been great (no-one has left the company yet, that I am aware of!) and they have all listened to my pleas not to go through ALL of the troubleshooting done the first time around because it must surely be a problem with the iCloud servers. I have collected logs and screen captures once and I have been called back within reasonable timeframes. Every time I spoke to a Senior Advisor I pleaded my case to get someone to look at the servers. I think it wasn’t really going anywhere again2 until I got frustrated and decided to do some of my own digging.

From searching Google I had learned some details of iCloud processes that run on macOS so I opened the Console app and started looking to see what the logs would reveal. I couldn’t make head nor tail of most of it, but decided on an experiment. I set a filter to only show the one iCloud process and waited on it to stop spewing messages, then cleared the log window. At a precise time I created a new file by means of a screen capture which was saved on my Desktop (being part of iCloud Drive). More messages spewed forth and when they stopped arriving, I copied all of them out to a text editor and started to study them.

I was able to locate messages relating to the syncing of my new file being attempted, and of the sync attempt not failing but being re-queued for later, then nothing. I repeated the experiment and found exactly the same pattern. On 7 March 2021 I sent a detailed email to Apple Support with what I’d done and what I’d found. I got a call from my Senior Advisor who made sure he understood what I had sent and then forwarded it to Engineering. Two weeks later, my Senior Advisor called unexpectedly. He had news from Engineering. I steeled myself for disappointment but inwardly I was hopeful.

“Engineering have identified the bug. 😯 They have said there will be a fix. 🤭 But they cannot say when that fix will be released. 😶”

I had a bit of a fruitless conversation with my Senior Advisor about what this meant. In short, every time a new macOS release is published, I will be hoping it fixes my problem. All I can tell you so far is it was not in 11.3 nor 11.3.1.

But there is a twist to this story. Since December, I have been using my Mac mini to effect iCloud uploads stuck on the MacBook Pro. This involved setting up a folder on the Mac mini which I then shared on the network. Stuck files are easy to identify on the MacBook Pro as they have an iCloud symbol next to them and usually an arrow indicating they are currently uploading. They aren’t, of course. Carefully, directory by directory, I would move these files out of iCloud on the MacBook Pro and into the network share on the Mac mini, and then on the Mac mini I would move them into the correct iCloud folder again. Moments later, the files would re-download on the MacBook Pro. A fiddly process, but it worked. Until I decided there must be a way to automate this.

It turned out to be quite simple, if a little complex on syntax. I built a Quick Action on the MacBook Pro using Automator, which I titled “iCloud Fix”. This action moves the file to the network share and renames it so the file name includes the entire original path. On the Mac mini, Hazel watches this folder and decodes the file name back to the path and the original file name and moves it back to the correct location in iCloud. The net effect of this is right-clicking a stuck file and choosing “iCloud Fix” causes the file to temporarily disappear and then reappear, correctly synced to iCloud, and thus all my other devices. It can be done on one or many files and in any iCloud directory, it’s relatively fast, and it’s not fiddly at all.

Screen capture of the iCloud Fix Quick Action in operation.

On a final note, the hugely repetitive gathering of logs and other details that I had to do with Apple the first time around, with no real payback, made me think of a passage from my favourite book, The Hitchhiker’s Guide to the Galaxy. In reference to the Vogon race, The Guide had this to say.

They wouldn’t even lift a finger to save their own Grandmothers from the Ravenous Bugblatter Beast of Traal without orders signed in triplicate, sent in, sent back, queried, lost, found, subjected to public enquiry, lost again, and finally buried in soft peat for three months and recycled as firelighters.


The Automator Quick Action script (Workflow receives files or folders. Pass input as arguments):

#!/bin/zsh
for f in "$@"
do
  new_name="${f//\//%}"
  mv $f /Volumes/macminiusername/tempicloud/$new_name
done

The Hazel script for watching tempicloud (match on file starts with %):

#!/bin/zsh
input_leaf="$(basename "$1")"
old_name="${input_leaf//\%//}"
mv "$1" "$old_name"

  1. On a call for the second reporting of the issue, an Advisor told me there was a known issue calling from the US to other countries that resulted in this exact scenario. It seems like a known issue should be… known.
  2. I was offered the option of “resetting” my iCloud account, but I was not keen on that option because it’s REALLY hard to figure out all the iCloud data that is NOT in iCloud Drive, which would all be lost. How do you back up that which you cannot identify?

Header photo by Josep Castells on Unsplash.