Windows 8 DAWBench results for UltraLite-mk3 and PCIe-424

[UltimateOutsider]

I've posted this information on a couple other forums, but I've seen a lot of mention of Windows 8 here, and thought some folks might find my test results interesting.

The following two posts represent two separate rounds of tests I did. The first round was just with a pre-hybrid UltraLite, and the second test included the PCIe-424 and a couple of other interfaces, with slightly different Windows configurations (all detailed in the posts).

The super-quick summary is that Windows 8 seems to operate just fine for typical DAW projects, but at least with the MOTU interfaces I tested, performs significantly worse than Windows 7 when pushed to its limits (with hundreds of plugins and/or notes of polyphony).

I'm still trying to figure out if this is a MOTU-specific problem or a Windows 8 thing. But there's a lot of data to chew on in the mean time.

[UltimateOutsider]

(First round- did these tests 2 1/2 weeks ago.)

Before I replaced a couple 512GB hard drives on my DAW system with bigger ones, I thought it would be interesting to see how well Cubase would run on my system with Windows 8. I imaged my existing system onto the new drives via eSata, but before swapping the drives out I did DAWBench testing, first on my existing Windows 7 installation, and then after upgrading to Windows 8.

First off, my system/HW specs:

Interface: MOTU UltraLite-mk3 (pre-hybrid, FireWire 400 only)
Processor: Intel Core i7-950 @3.06 GHz (Bloomfield family, 4 cores, 8 threads). No overclocking, and I'm using the factory heat sink.
Motherboard: Gigabyte GA-EX58-UD5. (I have EIST and the C-states disabled.)
FireWire Chipset: On-board Texas Instruments 1394 OHCI Compliant Host Controller
FireWire Driver: 1394ohci.sys, Microsoft, 6/21/2006 (same driver on both OSes)
Case: Antec P183.
Hard drives: Three Western Digital Caviar Black drives, at 7200RPM @ 500 GB.
Power Supply: Corsair HX650.
RAM: Two Kingston DDR3 1333 sticks @ 6 GB each, for 12 GB total.
Wireless: Linksys WMP600N Wireless-N PCI Adapter with Dual-Band.
Optical: Plextor PX-B320SA Blu-ray Disc Combo.
Video: EVGA NVIDIA GeForce GTX 570.
OS Versions: Windows 7 Pro 64-bit with SP-1 / Windows 8 Pro 64-bit RTM
DAW Version: Cubase 6.5.3 32-bit (the DAWBench plugins were 32-bit only)

Before running the Windows 7 tests, I uninstalled Microsoft Security Essentials and Acronis True Image 2012, because I knew those apps aren't compatible with Windows 8, and they both add a little overhead. I also made sure both systems had the “High Performance” power scheme selected in the Power Options control panel before testing to ensure top CPU availability.

*** DSP TESTS ***
In the DAWBench DSP tests, you load a project that has some audible audio tracks, plus 40 additional tracks with 8 of the same insert effect loaded (but disabled), for a total of 320 possible instances of that effect. You play the audio and enable the insert effects one by one until you start hearing pops or dropouts. The “score” for that test is the number of instances you can enable before the audio breaks up. The DAWBench DSP suite includes six projects for six effects, but I only tested two, because the public DAWBench database currently only lists reference results for three of them, and the latest version of one of those three (Elysia mpressor), is very broken and evil, compared to the one the DAWBench author used for his tests. It literally takes 5 seconds now to load a single instance of the plug-in, meaning it takes half an hour just to load the damn project.

I also conducted all of my tests at 256 samples, because that’s what I currently use in my work. At 256 samples, my interface’s reported latency numbers are 7.098 ms input, 12.018 output, 19.116 total.

DSP TEST 1: URS Classic Console Strip Pro (CSP)

Instances on Windows 7: 133
Instances on Windows 8: 93
Percent drop in performance: 30%

DSP TEST 2: Wave Arts MutiDynamics 5 (MD5)

Instances on Windows 7: 157
Instances on Windows 8: 102
Percent drop in performance: 35%

*** POLYPHONY TESTS ***
The DAWBench Polyphony projects include dozens of initially muted polyphonic tracks using Native Instruments Kontakt 4. While you listen to the handful of audible audio tracks, you unmute the polyphonic tracks until the audio breaks up. There are two flavors of this test: One with no effects applied, and one with a convolution reverb effect.

POLYPHONY TEST 1: No Convolution Reverb (NCV)

Notes of polyphony on Windows 7: 540
Notes of polyphony on Windows 8: 340
Percent drop in performance: 37%

POLYPHONY TEST 2: Convolution Reverb (CV)

Notes of polyphony on Windows 7: 520
Notes of polyphony on Windows 8: 260
Percent drop in performance: 50%

*** OBSERVATIONS ***

• Cubase and my plugins mostly survived the upgrade, although the first few times I launched Cubase I got errors that the soft eLicenser didn’t have sufficient permissions to run. I modified the Cubase shortcut to run as Administrator and that made the warnings go away.
• My Waves plugins didn’t work after the upgrade. I didn’t investigate this, as they weren’t needed for the DAWBench tests.
• The Cubase ASIO meter was a bit more erratic on Windows 8 than on Windows 7. In my previous experiences with DAWBench, I never saw the ASIO meter “redline” even when audio began breaking up. But on Windows 8 the little red emergency light blinked every second or so once the meter got above around 70%. This almost always accompanied pops or dropouts.
• Normally on the DAWBench polyphony tests, I never see more than a 20-note difference between the “with convolution reverb” and “without convolution reverb” projects. But the gap between these two tests on Windows 8 was huge- 80 notes; a 24% difference.

*** SUMMARY ***



While Cubase and my audio hardware functioned largely without error on Windows 8, the overall performance during testing was 30-50% worse than on Windows 7 on the same system. What accounts for the different results? Is it Cubase? Is it the OS? Is it the drivers? I don’t know why I got the results I got. I even re-ran most of the tests, just to make sure the numbers were consistent.

I am curious whether FireWire has anything to do with it, because I’ve heard mumblings about Microsoft wanting to deprecate FireWire support- although the driver appears to be the same on both OSes. (They have different version stamps- 6.1.7601.17514 on Win 7 and 6.2.9200.16384 on Win 8, but those are just generic stamps applied to every inbox driver that coincide with the OS build.) I don’t have any USB or PCIe interfaces to compare against though, so I wasn’t able to rule that out.

To be honest, despite the test results, Windows 8 would probably actually work okay for my day-to-day recording, considering that I never have more than a couple dozen plugins or notes of polyphony in my songs, whereas the DAWBench tests were breaking down in the hundreds. I’ve already swapped my drives out and returned to Windows 7, though. It’s fast and it works.

Please post your own before-and-afters if you have the time and interest! You can compare my results against the main DAWBench reference results available at the "DAWbench Universal Suite - Results Database : In Progress" thread at the DAWBench forums.

[UltimateOutsider]

(Second round, which I did over last weekend.)

The Windows 7 vs. Windows 8 DAW benchmarks I did a week ago left me with more questions than answers. There was an obvious performance gap between the two operating systems in my tests, but none of my data gave me any hints as to why. In order to gain some more clarity on the issue (and satisfy my GAS), I picked up a few new interfaces of different kinds and ran some new tests.

The interfaces I tested for this round are:

• MOTU PCIe-424 with 24I/O expander. (PCI Express)
• MOTU UltraLite-mk3 pre-hybrid (FireWire 400)
• Native Instruments Komplete Audio 6 (USB)
• Virus TI Desktop interface mode (USB)

** PREPARATION **
In last week’s tests, I started out with my production DAW Windows 7 image on my original 500GB hard drive. After doing my Win 7 tests, I upgraded that image to Windows 8 for the second round.

I made several changes to my PC since then that necessitated re-testing my UltraLite results, so here’s what’s different between the tests I ran then and now:

• I did not uninstall Microsoft Security Essentials or Acronis True Image before doing the Windows 7 tests, so Windows 7 had a little more of a resource burden this time around. (Turns out those programs result in around a 6% performance penalty for Windows 7. More on that later.)
• This time I was running Windows 8 as a fresh install on a separate partition with only the minimum drivers and apps required for the tests.
• These tests were all done with a new 2TB hard drive as the system drive.
• Because of some equipment shuffling I did last week, I had to switch from using a 6-foot FireWire cable to a 15 foot one for the UltraLite tests. The 24I/O also came with a 15-foot cable.
• Due to some new hardware I purchased there were more drivers and services loaded on both operating systems this time.

So, I expected (and observed) both some slightly reduced performance results versus last week’s on the UltraLite, and a slightly smaller gap between the Win 7 and Win 8 rounds.

** THE TESTS **
I performed each test at least once for each interface, on each operating system. If I encountered results that were better or worse than expected, I rebooted the computer and re-tested to confirm the results were accurate. Doing this never significantly changed the results. Just like last week the tests were the following.

• CSP - Instances of URS Classic Console Strip Pro
• MD5 - Instances of Wave Arts MultiDynamice 5
• NCV - Notes of polyphony without effects on Kontakt 4
• CV - Notes of polyphony with convolution reverb on Kontakt 4

RAW DATA



PER-INTERFACE RESULTS

24I/O


KOMPLETE AUDIO 6


ULTRALITE


VIRUS


** OBSERVATIONS **

• The FireWire and PCIe interfaces saw a similar performance drop on Win 8 as on my previous tests. As I expected, the gap wasn’t as big because my Win 7 partition had more apps and services loaded than the Win 8 partition. The difference was still significant- a 21%-44% drop-off in performance on Windows 8.
• The two USB interfaces fared significantly better on Windows 8, in a sense. The Komplete Audio 6 only suffered a 4-12% drop, and the Virus lost only 0-4%- and even scored 2% higher on the CSP test!
• The goal of every test is to find the highest number of plugin instances or notes of polyphony without any audible pops or dropouts in the audio tracks of the project. With the PCIe and FireWire interfaces, if you were at that maximum, enabling one additional plugin or one more row of polyphony introduced maybe one pop every 3-4 seconds or so. Enabling a couple more would result in pops maybe every 2-3 seconds perhaps, and so on- but it took a while before you ended up with a simply unplayable result. The “no pops” threshold with these interfaces generally happened well before Cubase’s ASIO meter “redlined.” But with the USB interfaces, I could drive the ASIO meter all the way up into the red without hearing any pops- however, adding just a single plugin or polyphony track beyond that threshold would result in completely unplayable results; harsh buzzing, silent gaps, etc.
  
• While USB seemed to have a higher playable threshold than FireWire and USB in terms of ability to perform under severe ASIO strain on Windows 8, the environment was much more fragile. For example, while playing at the “no pops” threshold on the PCIe/FireWire, you could tab around, launch programs, etc without interrupting the audio, but with the USB interfaces, even something as trivial as launching Notepad.exe- sometimes even just moving the mouse- resulted in severe disruption of the audio.
• I used a program called RTL Utility, which measures actual round-trip latency on audio interfaces, giving you more accurate numbers than what you would see by looking in your DAW’s audio interface panel. I found that the latency numbers (both reported and real) between the two operating systems were virtually identical. The difference between Win 7 and Win 8 seems to be more a matter of load/bandwidth than raw speed.
• In an effort to see if PCI/PCIe saturation had anything to do with the poor showing for the MOTU interfaces on Windows 8, I replaced my high-end NVIDIA GeForce GT 570 card with a much more modest GeForce GT 430, and also removed my PCI WiFi card. This improved the PCIe interface tests by around 3.5 percent overall, but had no effect on the FireWire results. Also, the PCIe performance boost occurred on both operating systems, so it didn’t help explain the gap.
• Due to the differences in my Windows 7 setup between last week’s tests and this week’s tests, my Windows 7 tests in this round were around 6% worse. In other words, the extra software I normally have running for antivirus and backups incur a 6% performance hit. My Windows 8 test results between the two weeks for that interface were basically the same. So keep in mind that my Windows 7 results this week sort of have a 6% handicap. The Virus may have scored the same on both operating systems this week, but if I had uninstalled Microsoft Security Essentials and Acronis True Image, it very likely would be 6% worse on Windows 8. (Which is still much better than the other interfaces.)

** CONCLUSIONS **
It’s still way too early to declare that there’s a serious issue in Windows 8 (and if there is, what exactly is the root cause), but I think we can at least rule some potential culprits out.

• It’s not Cubase. The TI Virus just about the same on both operating systems (with the caveat that the Windows 7 results have around a 6% handicap). If Cubase were the problem, I’d have expected the performance gap to be consistent across interfaces.
• It’s not JUST FireWire. Both my PCIe and FireWire interfaces underperformed similarly on Windows 8. Efforts to reduce PCI saturation only slightly improved results for the PCIe interface, and didn’t affect the FireWire interface at all.
• It’s not necessarily drivers. Both my PCIe and FireWire interfaces are MOTU products, but the two devices have different features, different hardware, and different drivers. It COULD be a case of unoptimized drivers in both cases... but if that’s the case, what’s different about Windows 8 that it would cause these logo-compliant Windows 7 drivers to underperform?

[UltimateOutsider]

I forgot to mention in my previous benchmark post that as before, all tests were performed at 256 samples, with 44.1kHz sample rate, at 32-bits. I recorded both the reported and measured latency of each interface on both operating systems. The reported numbers (as shown in the Cubase interface panel) were identical on both OSes, and the measured numbers, as computer by Oblique Audio's RTL Utility, were very nearly identical.

For reasons I haven't been able to figure out yet, I wasn't able to complete the RTL (round-trip latency) tests on the Virus TI Desktop. That's why there's no data for it on the "actual" test results.

RAW DATA (lower numbers are better)


CHART (shorter bars are better)


That's it for now. Excuse me while I go off to ponder whether a 5.041ms latency advantage is worth the $1200 USD price difference between my PCIe-424 core system and the Komplete Audio 6.

[UltimateOutsider]

So, Vin (DAWBench) did do some testing with an RME PCIe card, and also did SONAR vs. Cubase testing on both OSes. His findings are here.

His PCIe card performed more or less the same between Win 7 and Win 8. He also found that Cubase and SONAR performed the same between the two operating systems.

So, my poor PCIe results on Win 8 appear to be driver related. My FireWire results could be a problem with the driver, FireWire (this is still a controversial Win 8 topic), or both. Vin says he might test a FireWire interface to be sure.

Also, as we both suspected, Windows 8 does not improve SONAR performance- but doesn't appear to hurt it either.

I will re-test my UltraLite and PCIe-424 when/if MOTU ever releases Windows 8-approved drivers for them.

[billruys]

Hey UltimateOutsider,

Just wondering if you have any news? Have things improved since the last round of MS patches (I noted they listed improved AV performance and better driver compatibility among other things).

Or, are we still waiting for new drivers to fix the poor performance?

[UltimateOutsider]

Hi. I literally haven't booted into Windows 8 on my DAW system since I made my last post in this thread. I am indeed waiting for some new MOTU drivers (which reminds me, I need to add MOTU's RSS feed to my reader), and I will test them out of curiosity.

I have been developing professionally for Windows 8 for a year now, and know it pretty well. It's a fine OS (although I still don't like the UI changes, and most HW vendors still haven't released Win 8 drivers or officially acknowledged Win 8 support) and since none of its improvements affect DAW performance in any positive manner, I'm avoiding it at home- except when I get the benchmarking itch.

[billruys]

Thanks for the update.

I gave Windows 8 a try early on with Sonar X1 and X2. One thing I noticed was if I opened the PCI Console (running 2408 Mk2 & 24I/O), the hard drive would start to chatter with activity which took several minutes to stop. I never ran any diagnostics, but something was causing an awful lot of disk activity.

I have since gone back to Windows 7 and unless there is a compelling reason to move to 8, I can't see myself going there.

I think a lot of Sonar users have made the jump due to a post from the Cakewalk CTO that claim huge performance improvements. Problem is, nobody seems to be seeing those improvements in the real world.

[UltimateOutsider]

Thanks for the update.

I gave Windows 8 a try early on with Sonar X1 and X2. One thing I noticed was if I opened the PCI Console (running 2408 Mk2 & 24I/O), the hard drive would start to chatter with activity which took several minutes to stop. I never ran any diagnostics, but something was causing an awful lot of disk activity.

I have observed increased disk activity as well. Haven't done any real investigation into the cause, but for example on Win 8 I saw the disk working a lot during my Kontakt benchmarks- while under Win 7 the HD doesn't get hit at all.

I reckon it's one or more of Win 8's caching or indexing services.