2009 Mac Pro frequent kernel panics - RESOLVED

The forum for petitions, theoretical discussion, gripes, or other off topic discussion.

Moderator: James Steele

Forum rules
The forum for petitions, theoretical discussion, gripes, or other matters outside deemed outside the scope of helping users make optimal use of MOTU hardware and software. Posts in other forums may be moved here at the moderators discretion. No politics or religion!!
Post Reply
BobK
Posts: 1529
Joined: Mon Oct 18, 2004 10:01 pm
Primary DAW OS: MacOS
Location: Oakland, CA
Contact:

2009 Mac Pro frequent kernel panics - RESOLVED

Post by BobK »

I've been getting frequent kernel panics the past few days on my 2009 Mac Pro 8 x 2.26 GHz "Nehalem".

I've done a bunch of troubleshooting, but am not yet sure if it's a hardware or software problem. It's happened while working in several different apps - nothing consistent.

Steps taken, not in chronological order: reinstalled El Capitan from Recovery Mode, used Disk Utility to perform first aid on System drive while in Recovery Mode, reset NVRAM, reset SMC, used Tech Tool Pro to repair permissions and check the RAM, disconnected all peripherals except the display and a USB hub for my keyboard and trackpad, checked the system drive with Onyx, booted in Safe Mode, removed all RAM except the original RAM, removed the original RAM and left in the newer RAM.

Based on some stuff I read online, I checked the panic logs, especially the section "Kernel Extensions in backtrace".

Some of them mention "com.apple.driver.AppleIntelCPUPowerManagement".

Others mention what I think has to do with the graphics card, "com.apple.nvidia.classic.NVDAResmanTesla". Except from what I've read online, a bad graphics card would probably show symptoms like a gray or black screen.

As I write this, I've been booted in Safe Mode for almost two hours, and haven't yet had a crash. So maybe it's a software issue.

The panics started soon after I installed the latest security update for El Capitan (along with a few other app updates, including Logic Pro, MainStage, and Postbox).

Anybody had a similar experience and figured it out?

EDIT: Here's the solution, to save you reading the whole thread. In between CPU A and CPU B is a chip called the Northbridge (which controls traffic to the CPUs). It has its own little heat sink with vertical metal fins. One of the two plastic push-pins which secures the heat sink to the chip had disintegrated. The Northbridge chip was overheating, causing the panics and shutdowns.

See more here: http://www.xlr8yourmac.com/archives/jul ... verheating

The heat sink for CPU A partially covers the Northbridge heat sink, including one of the push pins. So you need to remove the heat sink (and the processor, which will probably stick to the bottom of the heat sink) in order to check or replace it.

To remove the CPU heat sink, you need a long 3mm hex wrench to reach the captive bolts, like the Eklind 54930.

https://www.amazon.com/Eklind-54930-Cus ... B000X285AW

Push-pins are hard to find locally, and ones from eBay have to be altered to be used, but as mentioned in a Mac Rumors post, I used #4-40 machine screws with nuts.

Hope this helps.

Bob
Last edited by BobK on Wed Aug 09, 2017 4:42 pm, edited 4 times in total.
Bob

M1 Max Mac Studio - 64 GB RAM - macOS 14.1.2
MacBook Pro (15-inch, Mid 2012) - 2.6 GHz Intel Core i7 - 16 GB RAM - macOS 10.15.7
DP 11.23
Metric Halo ULN-8 mk4
User avatar
Gravity Jim
Posts: 2005
Joined: Wed Apr 30, 2008 2:55 am
Primary DAW OS: MacOS
Location: Santa Rosa, CA

Re: 2009 Mac Pro frequent kernel panics

Post by Gravity Jim »

That sounds like a hardware problem to me... bad RAM would be my first guess, but it sounds like you've done some thorough troubleshooting there.

Have you used Apple Hardware Test?

https://support.apple.com/en-us/HT201257
Jim Bordner

MacPro 5,1 (3.33Ghz 12-core), 32g RAM, OS X 10.14.6 • MOTU DP 10.11 • Logic Pro X 10.2.5 • Waves Platinum, UAD-2, Slate Digital, Komplete, Omnisphere 2, LASS, CineSamples, Chipsounds, V Collection 5[color]
User avatar
Shooshie
Posts: 19820
Joined: Sat Oct 16, 2004 10:01 pm
Primary DAW OS: MacOS
Location: Dallas
Contact:

Re: 2009 Mac Pro frequent kernel panics

Post by Shooshie »

I've been getting a lot of that kind of thing. Then, sometime in the past week, I noticed that my 40 GB RAM had changed to 32 GB. I checked the RAM, and one of my 8GB SIMMS or DIMMS or whatever they are had gone out. Swapping positions to verify that it wasn't the slot itself, I confirmed that it was the RAM. A new one is on the way.

In the meantime, I changed the configuration so that the empty slot is the last one, and the other slots are matched symmetrically between the two groups of 4.

Still getting a lot of kernel panics, even with the RAM card removed. Maybe others are about to go out. I'm amazed that it's able to reroute around a faulty card, even if it does crash a few times.

Shoosh
|l| OS X 10.12.6 |l| DP 10.0 |l| 2.4 GHz 12-Core MacPro Mid-2012 |l| 40GB RAM |l| Mach5.3 |l| Waves 9.x |l| Altiverb |l| Ivory 2 New York Steinway |l| Wallander WIVI 2.30 Winds, Brass, Saxes |l| Garritan Aria |l| VSL 5.3.1 and VSL Pro 2.3.1 |l| Yamaha WX-5 MIDI Wind Controller |l| Roland FC-300 |l|
User avatar
mikehalloran
Posts: 15132
Joined: Sun Jan 25, 2009 5:08 pm
Primary DAW OS: MacOS
Location: Sillie Con Valley

Re: 2009 Mac Pro frequent kernel panics

Post by mikehalloran »

How old is the NV RAM battery? A bad battery will cause problems that a missing battery cannot. Recommended interval is 5 years. A cheese grater is fine with the ubiquitous CR2032 battery — the BR2032 should be used in 2011 and later iMacs and 2013 and later Mac PROs because of the greater heat.

Are there crash reports?

Will Console.app show a log that includes the kernel panic?

Tech Tool Pro is better at finding RAM problems than the Apple RAM test. Atomic, also from Micromat, is even better but expect major down time while testing.
DP 11.31; 828mkII FW, micro lite, M4, MTP/AV USB Firmware 2.0.1
2023 Mac Studio M2 8TB, 192GB RAM, OS Sonoma 14.4, USB4 8TB external, M-Audio AIR 192|14, Mackie ProFxv3 6/10/12; 2012 MBPs Catalina, Mojave
IK-NI-Izotope-PSP-Garritan-Antares, LogicPro X, Finale 27.4, Dorico 5.2, Notion 6, Overture 5, TwistedWave, DSP-Q 5, SmartScore64 Pro, Toast 20 Pro
User avatar
Shooshie
Posts: 19820
Joined: Sat Oct 16, 2004 10:01 pm
Primary DAW OS: MacOS
Location: Dallas
Contact:

Re: 2009 Mac Pro frequent kernel panics

Post by Shooshie »

Replaced the bad RAM today. So far, things seem actually better. I was having trouble with things like clicks not working, applications suddenly quitting (no warning, no dialogs, no reports), and constant spinning wheels. Those things seem to have cleared up. I put the new 8Gig in slot 1, thinking that would logically be the most often used slot. The full arrangement:

8GB: 1—5
8GB: 2—6
2GB: 3—7
2GB: 4—8
Symmetrical arrangement putting the 8Gb DIMMS first in line, and the 2GB DIMMS last. Less page swapping that way, I presume.

Anyway, so far so good. We'll see what happens. And yeah, I ought to replace that CR 2032 battery for no other reason than that it's old.

Shooshie
|l| OS X 10.12.6 |l| DP 10.0 |l| 2.4 GHz 12-Core MacPro Mid-2012 |l| 40GB RAM |l| Mach5.3 |l| Waves 9.x |l| Altiverb |l| Ivory 2 New York Steinway |l| Wallander WIVI 2.30 Winds, Brass, Saxes |l| Garritan Aria |l| VSL 5.3.1 and VSL Pro 2.3.1 |l| Yamaha WX-5 MIDI Wind Controller |l| Roland FC-300 |l|
BobK
Posts: 1529
Joined: Mon Oct 18, 2004 10:01 pm
Primary DAW OS: MacOS
Location: Oakland, CA
Contact:

Re: 2009 Mac Pro frequent kernel panics - RESOLVED

Post by BobK »

Ack! Sorry for not responding sooner - I didn't receive any notifications because I forgot to update my Motunation account with my new email address. (Canceled the old one a few weeks ago because of excessive, unstoppable spam.)

Thanks for the replies! Problem seems to be resolved.

Around the time I posted here, I posted some kernel panic logs over at the official Apple discussions, and a guy there said they all pointed to a problem with RAM. That seemed odd, though, because my tests included separately removing old and new batches of RAM, and in both cases, I still got kernel panics. It seemed unlikely that both sets of RAM contained one or more bad sticks.

Still, I gave OWC a call, because at this point, I was running RAM that was less than a year old and which has a lifetime warranty.

They recommended running Apple Hardware Test.

I ran it from the DVD that came with the computer, and it threw an error pretty quickly. Google searches didn't help [note: should've searched the Apple and Mac Rumors forums, because they DID have posts about this particular error code], nor did Apple, but the guys at OWC said the code referred to the Northbridge temperature sensor. I had no idea what that was, and they didn't either.

I posted this info at the Apple forum, and the same guy posted an article that turned out to be spot on:

http://www.xlr8yourmac.com/archives/jul ... verheating

It all hinges on two plastic spring-loaded push pins which attach the heat sink for the Northbridge chip to the logic board. (I later learned that this chip controls the traffic to the CPUs and naturally runs quite hot.)

These pins are the Achilles heel of the 2009 dual-processor Mac Pro. One of them is visible in between the CPUs - and in my computer, this one looked fine.

The other pin is covered by the heat sink for CPU-A and is thus exposed to a lot of heat.

If even one of these pins fails, the heat sink isn't sufficiently in contact with the Northbridge chip to cool it. When the chip gets too hot the computer shuts down.

I re-installed iStat Menus (which I'd uninstalled a couple years ago having never used it!), and found in a forum post that the Northbridge sensor appears as "IO Hub Tdiode". It was running at around 240ºF; one web site said the chip is spec'd to run at 220.

I bought a long-stem 3mm hex wrench and some thermal paste and opened up the patient yesterday. There was no sign of the second pushpin, the one under CPU A. It was completely gone. The only trace of it was the spring, which I found on the CPU tray underneath the logic board.

Then I remembered that one day a while back I found a thin piece of black plastic about 1/2" long on the floor under the computer, or maybe it fell out of the tray when I went to install new RAM. I had no idea what it was at the time. Turns out that was a shard of the push pin shaft, which had split in half lengthwise. (I still don't know where the rest of that pin went!)

Following the advice of a post at Mac Rumors, I used a 4-40 x 1/2" machine screw because spring-loaded pushpins aren't readily available. (As a backup, I ordered some pushpins from Hong Kong on eBay.)

I removed all the old thermal paste on the CPUs and Northbridge chip and the heat-sinks, then cleaned the surfaces, reapplied the paste, put everything back together, restarted, and....

...got a solid gray screen and a red LED glowing on the logic board.

Fortunately, earlier in the day I'd watched a YouTube video on CPU upgrades where the guy resolved startup problems by tightening the CPU heat sink screws.

I tightened the screws carefully - over tightening can apparently trash the CPU socket(s) - and that fixed it.

Northbridge temperature is now steady at 160-163º, and I haven't yet had a crash. Fingers crossed.
Last edited by BobK on Wed Aug 09, 2017 4:44 pm, edited 1 time in total.
Bob

M1 Max Mac Studio - 64 GB RAM - macOS 14.1.2
MacBook Pro (15-inch, Mid 2012) - 2.6 GHz Intel Core i7 - 16 GB RAM - macOS 10.15.7
DP 11.23
Metric Halo ULN-8 mk4
User avatar
HCMarkus
Posts: 9712
Joined: Tue Jan 10, 2006 9:01 am
Primary DAW OS: MacOS
Location: Rancho Bohemia, California
Contact:

Re: 2009 Mac Pro frequent kernel panics

Post by HCMarkus »

Good job Bob!

Glad you got pointed in the right direction. I haven't had the problem on either of my 2009 Mac Pros (yet), but have a packet of those pushpins on hand for the day they are needed.

For anyone else looking for the pushpins, here's what I ordered:

http://www.ebay.com/itm/10x-PC-Heatsink ... 1eba6cec03

My reading has suggested that the plastic spring shields must be trimmed ("cirumsized") to fit the Mac Pro, but they should then work fine.
HC Markus
M1 Mac Studio Ultra • 64GB RAM • 828es • macOS 13.6.4 • DP 11.31
BobK
Posts: 1529
Joined: Mon Oct 18, 2004 10:01 pm
Primary DAW OS: MacOS
Location: Oakland, CA
Contact:

Re: 2009 Mac Pro frequent kernel panics

Post by BobK »

HCMarkus wrote:Good job Bob!
Thanks, Mark! It was a bit nerve-wracking...
HCMarkus wrote:Glad you got pointed in the right direction. I haven't had the problem on either of my 2009 Mac Pros (yet), but have a packet of those pushpins on hand for the day they are needed.

For anyone else looking for the pushpins, here's what I ordered:

http://www.ebay.com/itm/10x-PC-Heatsink ... 1eba6cec03

My reading has suggested that the plastic spring shields must be trimmed ("cirumsized") to fit the Mac Pro, but they should then work fine.
"Circumsized" :lol: I read about that too. Also that the one under CPU A heat sink needs to have the top cut down to fit.

Actually, after this success, I've started to wonder if I shouldn't upgrade the CPUs, and have been checking out your thread, Mark. Thanks for all the helpful info.

This guy sells a pair of x5690s, de-lidded and guaranteed to work, for $431. The ~$200 premium (over used, non-delidded ones) gets you the de-lidding, a guarantee, and, FWIW, a USB drive with the firmware update.

https://www.dncomputers.com/mac-pro-200 ... slbvx.html

I'm not up for doing the de-lidding myself, both because of the risk of trashing the processor and also because I have no way of testing a processor to make sure it works on arrival.

I read that some people skip de-lidding, but it involves some extra fiddling at install. At this point, I'm not willing to take that risk, but might consider it.

What do you think - is it worth it to upgrade this machine?

If I recall the stats correctly, this upgrade could double the processing power of this machine, based on Geekbench scores.

A 2017 iMac would beat the upgraded Mac Pro in the single-core Geekbench score, but the Mac Pro would be way ahead in multi-core, which I realize is important for DAWs and VIs.

I'm wondering what limitations there might be in an upgraded MP vs new iMac - like, are there factors to consider beyond processor speed (not counting the display)?
Bob

M1 Max Mac Studio - 64 GB RAM - macOS 14.1.2
MacBook Pro (15-inch, Mid 2012) - 2.6 GHz Intel Core i7 - 16 GB RAM - macOS 10.15.7
DP 11.23
Metric Halo ULN-8 mk4
User avatar
HCMarkus
Posts: 9712
Joined: Tue Jan 10, 2006 9:01 am
Primary DAW OS: MacOS
Location: Rancho Bohemia, California
Contact:

Re: 2009 Mac Pro frequent kernel panics

Post by HCMarkus »

I think the CPU upgrade is a no-brainer if you need the horsepower.

Buying pre-de-lidded CPUs eliminates the risk; you can reportedly de-lid the CPUs yourself fairly easily. Using the vise method looks to be the winner, but Michael Canavan of MOTUNation did so successfully on first try using the hot iron method.

Downsides are eventual non-support by macOS (but support looks to be present thru at least High Sierra once your 4,1 cMP s flashed to 5,1), slower single thread performance, and lack of Thunderbolt ports.

On the upside are the low cost and expandability of the cMP. I'm running a 4k monitor with an AMD RX460 on my hex 2009 and, due to the low cost of the machines, have a second cMP ready to roll in the event of failure of the studio mac. I've got multiple SSDs running, some on SATA2 and some on SATA3 (via PCIe card) along with internal SATA HDs for daily project backup and external USB HDs for Time Machine.

The 4,1>5,1 upgrade is a good way to tide oneself over to the reportedly forthcoming 7,1 modular Mac Pro and, potentially, well further into the future. I have found upgrading my Macs to be gratifying, in both financial and performance terms.
BobK
Posts: 1529
Joined: Mon Oct 18, 2004 10:01 pm
Primary DAW OS: MacOS
Location: Oakland, CA
Contact:

Re: 2009 Mac Pro frequent kernel panics

Post by BobK »

Have you heard of any problems with iCloud services or the Mac App store from people flashing 4,1 to 5,1 and then doing the CPU upgrade? I saw one post - I think at Netkas - but I don't recall the specifics.

This post is very helpful and described the exact upgrade I'm considering: http://forum.netkas.org/index.php/topic ... l#msg24313

He - and at least one other person there - says that if you have the 2.26GHz "Nehalem" Mac Pro, don't upgrade to a processor with higher power consumption. The Nehalem is rated at 95W, and the 2010 Westmere at 130W. One post says the best upgrade for the dual 4,1 is the 5675. (A pair costs only around $100 now.)

Pretty sure I've read accounts of people using 5690s in the 2.26, but I'd rather play it safe.

That post also recommends against de-lidding and provides what looks like a pretty straightforward solution: a 5mm thick thermal pad on the bottom of the heat sink; two 1mm thick washers on each post; and removing clips from the fan connector. Tempting.
Bob

M1 Max Mac Studio - 64 GB RAM - macOS 14.1.2
MacBook Pro (15-inch, Mid 2012) - 2.6 GHz Intel Core i7 - 16 GB RAM - macOS 10.15.7
DP 11.23
Metric Halo ULN-8 mk4
User avatar
mikehalloran
Posts: 15132
Joined: Sun Jan 25, 2009 5:08 pm
Primary DAW OS: MacOS
Location: Sillie Con Valley

Re: 2009 Mac Pro frequent kernel panics

Post by mikehalloran »

Wow, thanks for reporting back. I used to fix G3–G5 PPCs by scraping off dried out, failed, heat sink pads and replacing with a dab of Arctic Silver. I'd forgotten that kernel panics were the normal heads-up.
DP 11.31; 828mkII FW, micro lite, M4, MTP/AV USB Firmware 2.0.1
2023 Mac Studio M2 8TB, 192GB RAM, OS Sonoma 14.4, USB4 8TB external, M-Audio AIR 192|14, Mackie ProFxv3 6/10/12; 2012 MBPs Catalina, Mojave
IK-NI-Izotope-PSP-Garritan-Antares, LogicPro X, Finale 27.4, Dorico 5.2, Notion 6, Overture 5, TwistedWave, DSP-Q 5, SmartScore64 Pro, Toast 20 Pro
Post Reply