Setting up Hard Drives for recording and Sample Libraries

[rosindabow]

Hi all. Looking for some thoughts on what is the most efficient way to set up your hard drives for recording and for sample playback. I am a long time user and looking to reorganize some things. I own many, many sample libraries, over 4Tb worth. I also have approximately 3Tb of active or fairly recent audio recorded on my record drives. For years I have been using multiple hard drives, most of them 2Tb in size, usually with 2 partitions on each. I of course have to keep clones of each drive plus my system drive. So I end up with audio and sample libraries spread across both internal and external hard drives. My system drive is an OWC 1Tb SSD. I just recently purchased some 4Tb bare drives and I was thinking of consolidating my record drive as a single, non-partitioned 4Tb internal hard drive. I also want to consolidate most of my sample libraries onto a single, non-partitioned 4Tb internal hard drive. I have the 4 internal bays on my Mac Pro so there is no problem with space. I also run OWC hard drives in their Mercury Pro cases so that there is no case fan noise, just platter spinning noise. However, even that little noise can be a factor as I do have live players in my control room with me most of the time and every now and then, you can hear a hard drive working hard at finding something.

My question, does this make sense? I had bee told in the past to partition my drives so that they were more efficient. I was also told that managing hard drives larger than 2Tb can be a pain. I have now recently been told by a number of professionals not to partition my hard drives as the search time rises because each time a file is searched for, it must scan each partition separately, as if it were a separate hard drive.
I'm looking for feedback as to what others are doing. I know that we all have these massive libraries and hard drive management can be a bear.
One other note, I use a Nitro AV as my external firewire 800 hub and that helps because of the consistent speed.
I know that one day I will be migrating to the new trash can Mac but that's a couple of years down the road.
Will I lose playback speed if most of my libraries are on one massive internal hard drive?
Will I run into any record issues?
Thanks.

[mikehalloran]

There is no reason to partition those drives except to have a repair partition -- and the OS installs that for you unless you created your boot drive by making a clone. I keep a small 15G repair partition on one of my drives with a copy of TechTool Pro, Disk Utility and Disk Warrion. Even though DW is useless for repairs, it is still useful for HD diagnostics but nowhere near as good as TTP.

As for external drives, USB 3 and eSATA are just as fast as the internal bus on a Mac Pro. eSATA has an advantage when it comes to drive maintenance while USB 3 does not require shutting down the Mac to reconfigure. Either requires a PCIe card.

[rosindabow]

HI - thanks for the response. It doesn't quite answer my questions however.

Anyone else out there running a similar setup streaming large libraries via firewire?

[mikehalloran]

HI - thanks for the response. It doesn't quite answer my questions however.

Anyone else out there running a similar setup streaming large libraries via firewire?

Many of us are, also USB 2. It works but it is sloooooow. Don't partition since it can slow things a little and cannot improve performance.

The advice is the same: go to eSATA or USB 3 on your externals if you want to see any real world performance performance or lower latency. If you're going to spend the money on drives, spend a little more on a PCIe card that adds USB 3 and/or eSATA. Cards are cheap. Housings that support either or both protocols are dirt cheap. If you get a dock, no fan noise.

There has been no reason to partition drives to improve performance since 1.5G SATA I became available with the G5. Partitioning a HD can make things a little slower if that's what you want.

eSATA and USB 3 through the PCIe bus have the potential to be faster than the internal 3G SATA II. You cannot achieve the maximum 6G SATA III speeds because mechanical HDs are not capable of it except in some flavors of RAID – you are limited by the platter speed.

The only thing faster is external SSDs and USB 3 holds a slight edge in speed while eSATA has a big advantage that it supports TRIM.

It's also true that you are limited by the maximum load that the PCIe bus can handle but you would need to be running a server with a lot of traffic to achieve that. It's not an issue with DP.

[rosindabow]

I'm not looking to change anything. I've never used USB2 - I am running Firewire 800. Everything is running just fine. I cannot install a PCIe card inside the Mac as all my slots are taken for other things including a UAD card and my MOTU 424 card. I was just looking on feedback on using larger hard drives to stream massive libraries and record audio to. I own 30 or 40 hard drives, have 4 computers connected via VSL and MIDIoverLAN. It is a fairly complex setup and is working just fine - just wanted to consolidate to bigger hard drives and make sure that I was not going to run into any issues. Thanks.

[mikehalloran]

Feedback: They work fine.

I'm not looking to change anything.

Then the only improvement you can make is not to partition the new drives. It's a minor improvement at best.

[jloeb]

If you have no immediate plans to go the SSD route, then you are likely to take a performance hit by consolidating from more to fewer platter hard drives, at least with those drives that are on the internal bus. Don't know how complex your sessions get, but if you are using big templates and/or lots of voices then yes, there's a very good reason that the advice in the pre-SSD world has always been to spread your samples out over multiple drives. Platter drives can be slow relative to the performance demands of DAW work, and using multiple physical drives (not partitions, which don't help) spreads the throughput load.
With your FireWire drives, it's the connection that will slow you down before the drive ever does.

I'd say: departition, create an organizational system for yourself that you like, and stay multiple to maximize your performance. Getting your FireWire drives onto an eSATA bus using a card would help the performance of those drives specifically. You then might be able to reduce the external drive number a little without taking a performance hit. Overall, you could reduce your drive number, but I wouldn't go as far as possible in that direction.

[mikehalloran]

You can add 2 up to 16 PCIe slots to that Mac Pro using an expansion chassis. It's not cheap but, if you are on the lookout, you can find used units as old systems are strapped. It's not like the TB versions for the nMP where your 424 card and many other audio cards will never work.

eSATA is nearly 8x the speed of FW800. As I wrote earlier, you won't see those speeds with mechanical HDs but it will be a significant, noticeable improvement.

You asked a number of questions in your OP but the answer to most it that, as long as you are running FW, it doesn't matter. There's your bottleneck.

Running your libraries on as many multiple drives as possible was excellent advice in the old ATA/EIDE (now known as PATA) days. It just isn't a real world issue with SATA bus speeds.

[HCMarkus]

Although I am now all SSD but for backup chores, I believe there are still two good reasons to partition spinning HDs:

1) Data Management
Quick restoration from clones. For example, I use a spinner to store various OS Clones. I split a large spinner into partitions that are sized to fit my boot drive and clone multiple iterations of my OS/Applications, one per partition, to the HD. I can quickly restore an older OS/DP version if necessary, or boot into an older OS for temporary use without messing with my current OS drive.

2) Access Speed
Viewing partitions in Apple Disk Utility from the top of the partition list down represents outer > inner partitions on the physical drive. If you are looking to favor read/write speed for certain data (i.e. sample sets over backup data), using outer partitions will result in significantly faster access to the data stored there. Cross-backing up of data makes sense in some situations, using a fast outer partition of one drive for daily use data and a slow inner partition of a second drive to back up that data.

[mikehalloran]

If you have a need for having different Operating systems on a drive, then partitioning is the only way to manage them. There is no other reason to do it anymore.

The slower / faster hard drive partitions debate ended with SATA. I'm just the messenger. Partitioning library drives slows everything down but only a little. Does any of this really matter with DP? No.

The OP is on an SATA II bus and FW800. Until he gets past those speed festrictions, none of the rest of it matters.

Latency is the only real issue with audio. Streaming speed is easily handled ore USB 1, let alone USB 2.

The only other is cloning. This is where any speed / performance increase will be noted. Like many others, I clone my library drives on occasion -- it's not necessary but will save time if one ever went down. Since I only have one eSATA bus, one FW800 bus and one USB 2 bus with 4 ports, I have a great deal of experience here.

My recent work product is backed up with three Time Capsules. I would never trust normal backups to clones.

[rosindabow]

Speaking from my own personal experience, streaming and recording off of USB 1 or 2 does not work. When you get into libraries that are close to 1Tb in size, the bus speed does matter and they are just too slow. In fact, many professional libraries recommend placing sections of the orchestra on different drives so they can playback without choking the system. That is one of my big challenges. As far as recording, I have close to zero latency - runs like a top. I'll post back once I switch some stuff around.

[HCMarkus]

The slower / faster hard drive partitions debate ended with SATA. I'm just the messenger. Partitioning library drives slows everything down but only a little.

With all respect Mike, I have observed the difference in speed from outer to inner when transferring files… it's a physical rotational thing, nothing to do with drive interface. Just sayin'.

[bayswater]

Speaking from my own personal experience, streaming and recording off of USB 1 or 2 does not work. When you get into libraries that are close to 1Tb in size, the bus speed does matter and they are just too slow.

Why would the amount of data on the drive matter? If the seek times, buffers, etc are the same, it's the transfer rate that matters, not the number and size of the files sitting on the drive.

On the matter of FW800, yes it works fine. Again, I don't see why it would make a difference whether you have a 1TB or a 4TB drive, as long as seek times, buffers and transfer rates are the same. If you access more samples at the same time, you'll hit a limit, but that will happen regardless of the size of the drive or how full it is.

I can do quite a lot with a FW 800 drive, but it requires allocating a lot of RAM to preloads, etc, otherwise M5 chokes very quickly, and shows up as processor overload in DP. A reason to move away from FW -- as Mike has noted a few times, Apple support for it is disappearing, for both hardware and software, and Windows never did embrace it. With USB 3 in full swing, FW is a dying parrot.

[rosindabow]

Hello - with all due respect - in my original post, I never asked about USB3. That was not the intent of my discussion. I was just trying to determine if the size of the drives would effect my record or sample playback speed. USB3 is a great discussion to be had and I'm learning a whole heckuva lot about it. It was just not my original question. I think I have picked up that the size of the drive should have no bearing on my ability to record.

Do you think that track count would change with multiple record drives vs a single drive (again, using Firewire 800 for the moment.) Also, why do sample libraries recommend spreading there samples out on a number of different drives? It seems to me (again from personal experience) that once I did that, I had less hiccups. I put different libraries on separate drives and that seemed to help.

Anyway, if we could please address those issues first, that would be great. I would love to continue the discussion about USB3 at some other time. Problem is, I would have to change my entire setup to move into USB3 and eSATA and I'm not quite ready to do that.
Thanks.

[musicman691]

Hello - with all due respect - in my original post, I never asked about USB3. That was not the intent of my discussion. I was just trying to determine if the size of the drives would effect my record or sample playback speed. USB3 is a great discussion to be had and I'm learning a whole heckuva lot about it. It was just not my original question. I think I have picked up that the size of the drive should have no bearing on my ability to record.

Do you think that track count would change with multiple record drives vs a single drive (again, using Firewire 800 for the moment.) Also, why do sample libraries recommend spreading there samples out on a number of different drives? It seems to me (again from personal experience) that once I did that, I had less hiccups. I put different libraries on separate drives and that seemed to help.

Anyway, if we could please address those issues first, that would be great. I would love to continue the discussion about USB3 at some other time. Problem is, I would have to change my entire setup to move into USB3 and eSATA and I'm not quite ready to do that.
Thanks.

If the connection protocol and speed stay the same as well as drive speed a bigger drive won't really be any better or worse than a smaller drive.

As to track count versus multiple drives I've only ever run into or even seen that with ProTools. That's such a finicky daw anyways.

Spreading out samples on multiple drives that's a biggie that East West recommends with the bigger Play 4 libraries like the Symphonic and Hollywood Orchestras, particularly the ones that have the multiple mic positions. With slower connection protocols like FW800 it's still a good thing to do. Thunderbolt/Thunderbolt 2 and the faster Esata and an ssd you could keep a whole library on one drive.