Catalina kernel panics, spontaneous reboots to unmounted HD

[nk_e]

Hi all.

This is a long set up for what I hope are simple questions....

ISSUE

My set up is quite complex as I'm running a robust home recording set up with all kinds of hardware and multiple monitors connected to an iMacPro 10 core 64gb /4TB machine running OS X 10.15.7.

My relatively new iMac Pro randomly reboots. (Added backstory and detail are at the end of the email for those interested.)

I'd learned to live with it, but now it's taken a different twist that is untenable: When it randomly restarts, the HD is always unmounted. I have to go into recovery mode to remount the HD, but the remount doesn't "stick". Even after mounting it and restarting, the machine restarts/reboots again to an unmounted HD. (By that I mean that the login screen comes up and upon logging in, I get an Apple icon for a few seconds before the "circle with a line through it" icon comes up. It then goes to a black screen.)

The ONLY way I can get the machine to reboot properly is to reinstall the system each time this happens. Then the machine boots up (and fortunately my preferences and other info are preserved). But the next time the Mac is rebooted - either intentionally or through a watchdog panic (described at the end) - the HD is unmounted.

I have tried resetting PRAM, SMC, etc. I've booted into recovery mode and run disk first aid and everything checks out fine. I finally rebooted the machine in recovery mode, wiped the HD, restored from a recent TM back up and ... everything was fine and normal for 2 days.

Then last night I installed the new MOTU audio drivers for AVB/Pro Audio and I am back to square one. The machine won't boot to a mounted HD. I have to go into recovery mode and reinstall the system to get the machine to boot. I should emphasize that everything works "properly" from that point. I can complete projects, run my software, access connected peripherals, etc. I just dare not turn the machine off or reboot it.

THE QUESTION

So, I don't expect my problem to be diagnosed / solved here, but I want to ask: 1) Does this seem like a hardware issue at this point? Is there a physical reason why the HD won't behave properly? 2) Is there some test or other thing I should do to try to address this issue?

The machine is still under Apple warranty, but sending it away is highly problematic as it is my only production machine and I have projects scheduled through July.

More history and detail is below if you are interested. Any insights are appreciated. Thank you.

ADDITIONAL DETAILS

Upon a spontaneous restart there is always the same message:

panic(cpu 10 caller 0xffffff7fa03a1ad5): userspace watchdog timeout: no successful checkins from com.apple.WindowServer in 120 seconds

There's more to the message, but an important bit here is the reference to "watchdog" as there is a raft of online discussion about this issue. (Do a google search using the entire phrase above.)

For me, the problems didn't start UNTIL I installed a Catalina Security update for 10.15.7 back in December 2020. The crashes were very sporadic at first, then they occurred daily, then the crashes were coming like clockwork every 760 seconds or so.

My set up is quite complex as I'm running a robust home recording set up with all kinds of hardware and multiple monitors connected to an iMacPro 10 core 64gb /4TB machine running OS X 10.15.7. Because of that, I do not expect Apple Support to be any help whatsoever as there are too many other things to point to. BUT the system was stable and operating near perfectly before that update.

Over the holidays I set out to try to diagnose the cause of the problem. A lot of online discussion centered around external attachments, so I started disconnecting peripherals and experimenting with different configurations of attachments. Through A LOT of trial and error, I isolated the stability issue to a UAD Octo 8 card in an approved Sonnet Echo Express SE1 Thunderbolt 3 Expansion Chassis. With that box disconnected from the system, the panics became extremely rare (though they did not go away completely). I leave this box unconnected and have lost the use of my UAD plugins. Since then the only peripheral I have added to the system is an USB C 8TB external LaCie d2 Professional HD for my TimeMachine backups.

The consistent booting to an unmounted HD is a relatively new development and one that has me completely stumped...

[nk_e]

Just a quick bit of additional info…

I disconnected everything from the iMac to test whether or not a peripheral was causing the unmounted HD reboot issue. The machine would not boot to a mounted HD.

I then did the “fix” as described above and booted into recovery mode, reinstalled the system, successfully booted to a fully functional machine. I then tried the experiment again. I shutdown the system and attempted to start the machine up. Again, the same result:

Turn on
Apple logo
Catalina login screen after a few seconds
After entering password it almost immediately goes to the Apple logo
After a few seconds the circle with the bar through it appears
About 30 seconds after that the screen goes black

What I did notice this time is that after 1 or 2 minutes, the Apple logo appears again and then the machine will reboot of its own accord and end up back at the Catalina screen. If I enter the password, the behavior is as above. This loop will persist as long as I’m willing to play.

I’ve work to do, so I’ve reinstalled the system again to finish up a couple of projects. I’m beginning to believe that my only option is to do a wipe of the hard drive, install a fresh copy of Catalina, then either a) manually reinstall ALL of my software fresh *shudder* or b) use migration assistant and my TM backup to try to port as many of my settings over.

My four questions at this point are:

1 - Does this unmounted HD behavior sound like a physical problem of some sort? If the underlying cause is a faulty HD, I’d hate to go through this process only to end up in the same place. (I suppose after installing a fresh system I can try rebooting several times and using it for a few days to get some assurance that it works, but it worked for a couple of days during one of my previous fixes too…)

2 - Will migration assistant port applications, my plugins, and their preferences? Or does it just port things like account settings, mail settings, bookmarks, passwords, and such?

3 - Is there a chance that in using migration assistant I might re-import this issue? Is there anyway a gummed up preference or .plist or something might be gumming things up?

4 - Come to think of it - and this is a total WAG - Is there some sort of preference file related to my hardware config that I can eliminate that would force OSX to create a new one? Perhaps the issue is there?

Thank you for any insights.

G

[nk_e]

As a CODA to this "symphony of sorrow", here is what it took to fix this. I leave the info here in case anyone else runs across a similar issue.
==============================
So, over the last 10 days I went through many more variants and iterations of kernel panics, system reinstalls, PRAM and SMC resetting, etc. Too many to count to be honest.

I updated to Tech Tool Pro 14 to run various diagnostics. (No issues reported.) After a final TM backup, I booted into recovery mode and wiped the drive clean and installed a fresh OS. I then (re)created a fresh user account and did not migrate data or restore from an old backup. I shut down and restarted many times without problems, ran diagnostics, etc. Everything was solid and behaved normally.

After a couple of days of letting the machine run (largely to see if there would be any spontaneous reboots or "watchdog" panics), the machine was stable. I decided it was safe to use migration assistant to copy over my old user data, settings, and apps. After doing that, I restarted the machine without incident. I went through and cleaned up login items and security settings. I then restarted / rebooted the machine many times with no issues.

Now, I am fixing those things that migration assistant didn't get quite right. For example, I had to reinstall my MOTU drivers and give them security permissions again before my 1248 and 16A were recognized by Audio MIDI set up. I had to completely reinstall the software for my Raven and associated drivers and grant security permissions. I'm sure that I will find plugins that need to be re-authorized etc.

But in the end, I seem to have a machine that is once more fully operational and rock solid. Although it's only been 48 hours since everything has been set back to the last state before the installation of the OS, there's been no unmounted hard drives upon restart or reboot, no "watchdog timeouts", no weirdness in general. It's acting like the machine I bought 10 months ago.

So, if anyone finds themselves in a similar situation, I hope something in here might spur a couple of ideas that are helpful. Cheers all.

G

PS

To answer question #2 that I posed in the previous entry: Migration Assistant is quite comprehensive and ports applications, their settings, plugins, etc.

[mikehalloran]

This is normally caused by something that is supposed to be sandboxed but isn't. If this reoccurs, have Apple log into your machine and figure out what it is — then remove it. An old version of Soundflower and NI GuitarRig 2 (installed with a very old version of Finale were the culprits in my case.

Once all is done, see if you can do a Safe Boot.

If so, great.

Lastly,

Reset Launch Services

1. Open Terminal and run these commands:
2. sudo /System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -kill -seed -lint -r -f -v -dump -domain local -domain system -domain user -domain network
3. (enter Admin password when prompted)
4. killall Dock
5. sudo mdutil -E /
6. Restart your Mac
By wiping your drive clean you've done this recently. Still, it's a good idea to do this after System/Security updates, etc. If DP starts getting slow after a plugin install, this should bring everything back to life.

There's more to the message, but an important bit here is the reference to "watchdog"

Nope. There are tons of threads with a lot of blind guessing by people who've never asked Apple about this. Not important at all.

[nk_e]

Thanks Mike. I always learn a ton from you.

I researched ‘reset launch services’ and found a couple of good articles. This one was particularly helpful:

https://eclecticlight.co/2017/08/10/rec ... filelists/

I’m just curious: Why do you advise doing this after every security update or if DP slows down after a plugin install? (Is the launch services database prone to getting out of whack at those times?)

Also, would doing this help with the original soul sucking problem I was having (rebooting to an unmounted system disk)?

Thank you.

[mikehalloran]

Yes, the Launch Services database does get out of whack and not remembering the right startup disk can be one of the symptoms (also can be a corrupt paths error). I first discovered this when, after a High Sierra security update in 2018, 3 out of 5 Macs in my house would no longer shut down without a Force-Quit.

When my iMac Pro became unusable after the March 2020 Catalina update, I called Apple, they logged in and found the offending .kxt files. Besides the two I mentioned, there was one from PACE designed to make iLOK work over Lion. Anyway, removing those three stopped the crashing but left me with a consistent 6 minute boot time. What the heck? While waiting for Apple to get back to me on our weekly support calls, I wondered about Launch Services, ran the commands and was back in business. When I spoke to Apple again, it was way down on their list of things to try.

Resetting Launch services doesn't just reset the Mac to default; it resets every installed app — you'll see a page flash for each one. DP and Office often slow down after a plugin install (DP) or update (both). This restores the launch times I expect. BTW, after an update, Office always re-verifies the app with MS and this has no effect—only if after that verification, Office takes more than 30 seconds to launch do I run it. Many think that Office takes 5–10 minutes or more to launch nowadays but it shouldn't

Likewise, software updates and installs can corrupt your System paths, a different issue — this is what a Safe Boot resets. I do that before resetting the Launch Services. Boot while depressing the Shift key; log in to your User account when prompted and wait for any icons to rebuild. Reboot. If you have any generic alias icons, this is a symptom of corrupted paths and they will be fixed. In the old, pre-OSX days, this was called Rebuilding the Desktop and was done the same way.

[mikehalloran]

New Catalina Security updates out today.

Oh goodie…

[nk_e]

Thank you for taking the time to write all of that. It's incredibly helpful.

I'm too gun shy to do a security update right now. It's been working flawlessly for the last few days. I had forgotten what that is like. I'd like to savor it a little longer...

[mikehalloran]

I ran these without incident this afternoon. DP took 35 seconds for a project to open — with the number of plugins I run, that's about right.

[nk_e]

This is normally caused by something that is supposed to be sandboxed but isn't. If this reoccurs, have Apple log into your machine and figure out what it is — then remove it. An old version of Soundflower and NI GuitarRig 2 (installed with a very old version of Finale were the culprits in my case.

Once all is done, see if you can do a Safe Boot.

If so, great.

Lastly,

Reset Launch Services

1. Open Terminal and run these commands:
2. sudo /System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -kill -seed -lint -r -f -v -dump -domain local -domain system -domain user -domain network
3. (enter Admin password when prompted)
4. killall Dock
5. sudo mdutil -E /
6. Restart your Mac
By wiping your drive clean you've done this recently. Still, it's a good idea to do this after System/Security updates, etc. If DP starts getting slow after a plugin install, this should bring everything back to life.

There's more to the message, but an important bit here is the reference to "watchdog"

Nope. There are tons of threads with a lot of blind guessing by people who've never asked Apple about this. Not important at all.

Hi Mike,

I’ve been sort of limping along since I last posted. The watchdog panics seem to come and go in waves…long periods (a couple of weeks) where things are fine, then a random reboot, a shorter period of stability, then a couple of random reboots, a few days of stability then several reboots in a single day. At that point, I run your launch services reset routine in terminal and I’m stable for a couple of weeks then the process begins again.

I’m ready to throw in the towel and call Apple Care. Believe it or not, in over almost 40 years of using macs, I’ve never once had to call them on a hardware issue. Not that I’m brilliant (obviously), but more a testament to their designs and engineering.

I have a quick question on something you wrote earlier:

If this reoccurs, have Apple log into your machine and figure out what it is — then remove it.

Is that a special appointment or request that I make, or is that something that evolves out of the process of working with a tech? I ask because sorting this through with support could involve a lot of blind paths that I’ve already travelled and I’d like to jump straight to this level of diagnostics without all of the “OK let’s reset your PRAM and SMC then call us back if it happens again…”

Thank you for your advice.

[mikehalloran]

Is that a special appointment or request that I make, or is that something that evolves out of the process of working with a tech?

You initiate on Apple's support pages and request a call back. I've never had it take less than 10 minutes when requested during normal hours — whatever those are. The tech can log in or email a support bot that lets him/her do so. They also have logging apps that can run in the background—you'll be given instructions where to email the report (they don't do so automatically due to security concerns). If the tech is unable to resolve the issue, ask for a follow up call—if a log report needs to be uploaded, the tech will likely suggest one. An appointment will be made and they're very good about keeping them.

Best case: A crash happens while the tech is logged in, you've uploaded the crash log and the tech sees the problem in real time. Can happen.

MacOS support is for three years and it's free. High Sierra is no longer officially supported unless, like mine, it was the base OS on a 2017 Mac such as all iMac Pros and still under AppleCare . Mine was bought less than a year ago with AppleCare.

How to save some time: Boot into the Recovery partition holding the Command r keys and reinstall the current MacOS which will install the latest version of your installed OS without the last security update. Run the security update when prompted (I did this a couple days ago to help Dropbox diagnose a problem on my iMP—didn't fix anything). The tech will sit on the phone with you if you haven't done it so you save time if you say, 'did it yesterday' Same with a Safe Boot, resetting the NV RAM and Resetting Launch Services. These are the four Hail Marys of Apple Support because, so often, they really do fix the problem. The tech may ask you to do one or more of these tasks again.

Reinstalling the MacOS doesn't touch your data (still, have a backup). This is different than Internet Recovery which wipes your Mac and installs the OS your machine came with—do not do that until it's time to sell your machine.

These issues are usually caused by old crap that hasn't caused a problem in years. If so, they'll walk you through removing the offending files—I stored mine in a folder on my desktop in case removal caused any apps to have problems (didn't). NI Guitar Rig II, Soundflower, PACE, Word 1998, Finale 2009 Help files, Bresink Temp Sensor, HP drivers from 2005 are among the troublemakers I've found over the years and none were newer than OS Lion.

[nk_e]

Hugely helpful. Thank you!

[nk_e]

Mike,

Just wanted to thank you again for the info above.

After 42 “watchdog” kernel panics since June 3rd, I finally threw in the hat and contacted Apple support. I followed your advice and prepped for the meeting by performing the four “Hail Mary”s the night before contacting them. Sure enough, it streamlined the whole thing.

The tech had me download an Apple app to gather diagnostic data, and also had me upload the crash reports from the 42 panics. (I had saved them.) He sent it all off to some engineers to look at, and we are scheduled to have a follow up call next week. All in all, it took about 90 minutes.

He started to go down the “it could be a peripheral” route. I told him that I need to know which it is then because this is a production machine used for working on audio and video and peripherals are necessary. I told him that I’m betting/hoping it’s some old KEXT file. Hopefully, all of the panic reports will show a pattern.

Anyway, I just want to thank you again for all of the extremely helpful and valuable information and advice you’ve given in this and other matters.

I’ll let you know what happens.

Cheers.

[nk_e]

Well, that was fun.

I got a call back from Mac Support. After a week of deliberation the engineers had this advice:

Upgrade to Big Sur.

When I said I can't do that, their next step was to remove all connected peripherals and wait for another panic to occur to capture the resultant crash report.

When I said I can't do that because I need this machine to deliver projects to clients and those projects require peripherals (like my MOTU interfaces), they had nothing else. Basically:

"Well you are just screwed sir."

FWIW, I did go through the peripheral exercise earlier in the year, and I did isolate one culprit. (A Sonnet Echo Express SE I housing my UAD OCTO card.) Disconnecting it and reconnecting other things one at a time seemed to cut out the attacks for a long while, but obviously it hasn't eliminated them.

He says the engineers did not have anything to say about deprecated or problematic KEXTs.

Do you think I should just try the process again and hope for a different tech support person? Ask for escalation? Insist on a new machine? Suck it up over the next couple of years until I buy an M2+ AS machine? Switch to PC? (Just kidding about that last one.)

Blessedly, I have not had a panic attack in over a week...something of a record of late.

Anyway, as I said before, thanks for your suggestions. They really have been a lifesaver.

Cheers,
George

[mikehalloran]

Did you remove everything they told you to? If not, you need to do so. I created a desktop folder and archived them there. If any of my apps had issues, I was prepared to put them back but none did so.

You need to eliminate those kernel panics. Period. Stop second-guessing it. Once they are gone, then can you deal with the aftermath. Normally, if an app doesn’t work anymore, installing the latest build of it fixes you up.

Mojave is still supported till a few weeks after Monterey is released. If the people you are talking to don’t understand this, escalate it elsewhere.


I’ve been having issues with Dropbox messing up my system — spinning beachball, not kernel panic, though. With Apple’s help and after many hours, I’ve isolated the problem as a conflict with Corel. Dropbox didn’t want to deal with it till I let them know I isolated the issue. I quoted them a consulting fee and haven’t heard back. I’m talking to Corel.