DP7 Disaster in a live show

[meredithsdg]

I had a DP7 disaster on Saturday evening. Never had anything like this happen before. I was using DP to run pit music for a musical. We've been in rehearsals since mid September, had no issues. Saturday was closing night.
I'm running DP on a 13in MacBook Pro (mid-2009). The instrumental tracks were all MIDI playing through a Korg Kronos. There were a few stage sound effects (engine sounds, bells, water splashing, etc) as audio tracks. I used a TASCAM 428 as audio/MIDI interface. The entire show was in one DP project, separate chunks for each tune, overture, entr'act, scene change music, etc.
Played the overture, opening scene underscore, go to the 3rd chunk, there's a vamp at the beginning with memory loop, the cast joins in singing, I load the next memory loop for a one measure loop under dialogue about 1/3 into the song, come out of the vamp, get a few more measures and disaster strikes. Notes stick on, no new notes are sounding. I send "all notes off", notes remain stuck. I change sequences on the Kronos and stuck notes quit. (I have different track layouts as sequences with no MIDI data in the Kronos.) The MIDI input/output indicators on the 428 appear frozen. I suspect it might be the culprit. Since we're in the opening number, the director informs the audience that in live theater, stuff happens. We all chuckle and I reboot the Kronos, my Mac and the 428. We start again from the opening number, get past the glitch, life is good.
Then the really weird thing happens! I do a manual vamp on scene change music halfway through act 1. I let the sequence play to the end, but I choose a measure where I think we will vamp from and enter it into the counter with the keypad. I wait and hit return on the downbeat and the sequence jumps. But when it reached the end of the tune the second time, I notice the tempo slowed down on the last several beats. Strange, but hoped it was my imagination.
I go to the next chunk, start it part way through the scene. At measure 25, the tempo begins to slow down for no apparent reason. It slows to a crawl. The singer is mystified (let alone the music director!) I have no choice but to stop and start again. It slows down again at measure 25. The chunk is set to Conductor track, I switch to manual, back up to measure 20 or so, play again, AND IT STILL SLOWS DOWN at measure 25.
The Kronos and the 428 seem to be fine, so I quit DP, relaunch and load the project again, all the while muttering my apologies to cast and audience. We run again and everything is fine. I barely breathe until intermission. During intermission, I shut down everything, but all plugs and wires, reattach everything and reboot. Second act, all is fine!
I suspect a voltage spike may have caused the first issue, but for the life of me cannot think of what caused the tempo issue.
Any ideas?

[David Polich]

Yeah - it's the audio interface. It "lost its mind" and had to be rebooted,
as you discovered.

Seems pretty clear to me..the symptoms you describe are exactly what I've experienced when my audio interface "went AWOL".

[meredithsdg]

Yeah - it's the audio interface. It "lost its mind" and had to be rebooted,
as you discovered.

Seems pretty clear to me..the symptoms you describe are exactly what I've experienced when my audio interface "went AWOL".

Even the tempo issues?

[David Polich]

Yeah - it's the audio interface. It "lost its mind" and had to be rebooted,
as you discovered.

Seems pretty clear to me..the symptoms you describe are exactly what I've experienced when my audio interface "went AWOL".

Even the tempo issues?

Yes, even the tempo issues. Not saying I'm 1000% certain it's your audio interface, just that it's what I'd suspect was the problem, based on my
experience.

[Shooshie]

I did this sort of thing for 20 years on the professional touring circuits, and while I never had even a slight glitch during concert, it was always a fear of mine. I just tried to trust in MOTU and believe it would work… and it did. But there is one thing I did consciously and relentlessly: I insisted on MOTU interfaces. MIDI and Audio, if it was connected to Digital Performer, it was going to be a MOTU box.

My reasons were entirely hunches and a "feeling." I never conducted scientific tests to determine that MOTU boxes performed better with MOTU software, but it just seemed a given that their design and testing was integrated with their own products. I'm not going to tell you that this wouldn't have happened if you'd used MOTU interfaces; I have no idea what happened to you in concert, but I know it must have been terrifying while it was happening, and I empathize.

All I can say is that I ran redundant systems -- two complete computers with DP, two MIDI interfaces, two sets of audio interfaces, and all the cables to keep them live -- but I never once had to use the redundant set. It may have been dumb luck, or it may have been the security gained from using hardware and software that were designed and tested around each other.

I hope you can figure out exactly what happened. If you do, please let us know. That's worth knowing if you're on the road with this stuff, playing for thousands of people every night. My condolences that it had to happen to you at all!



Shooshie

[James Steele]

Yeah... I have to echo Shooshie here. I've always used MOTU interfaces because I just felt like they had to have been tested more thoroughly WITH DP than others. Also, you mentioned voltage spike? It may or may not be overkill, but I'd be plugged into a line conditioner if that's a concern.

[mikehalloran]

A voltage spike can certainly be an issue but other problems causing low voltage are far more likely. One of the things I never trust in a live situation is bus power.

[James Steele]

A voltage spike can certainly be an issue but other problems causing low voltage are far more likely. One of the things I never trust in a live situation is bus power.

Do you mean powering an interface off the USB port power only? Yeah... I don't know the technical reasons against that, but intuitively it seems risky. I know we have some professionals on the this board and the MOTU-MAC list who run DP on major tours (Roger Waters, Beyonce, Madonna) and they drive backup rigs they can switch to on the fly. I don't know if they attempt to drop repeat locations and engage loops on the fly though. That seems scary and perhaps a DAW designed for live interactive performance like "Live" might be better?

[meredithsdg]

Thank you, friends, for the thoughtful replies. I confess that I used my words a bit loosely when I said "voltage spike." More accurately, a power fluctuation may have occurred.

I doubt I will ever know exactly what caused the glitches, but here are the steps I've taken:

1. Added a UPS power supply.

2. Retired the TASCAM 428. (That device was well past it's prime.)

3. All MIDI tracks are directly through USB to the Kronos, no MIDI Interface.

4. The audio is going through the KRONOS well. (This may be risky. I'll be watching closely during rehearsals leading up to our next production in December.)

BTW, the Kronos was added to the pit in mid-September. I was already a bit uneasy about changing gear mid-production, so I kept the older gear in place because that's what we were accustomed to.

One final thought, I concur about using MOTU hardware with MOTU software and do so in my own studio. In this case, my employer makes the budgeting decisions and at the time, the TASCAM was our most affordable option. Having said that, I have had issues in years gone by with MOTU Fastlane interfaces and even MOTU tech support indicated driver issues were at fault. (this is ancient history, by the way. 10 or more years ago.)

I am very interested to see how things will proceed from here. I'll keep you posted. Again, thanks everyone for your replies.

[James Steele]

2. Retired the TASCAM 428. (That device was well past it's prime.)

That's probably the best move of all. Still, I'm sorry to hear you had problems. I wasn't aware of issues with the Fastland, but that's a MIDI only interface, right?

I'm in a situation where I'll probably be using backing keyboard tracks in a live situation and sending a click to the drummer. For the sake of "fewer moving parts" I plan on using an iPad with mono mix on one channel and cue/click track on the other. Using computers in a live performance situation just seems rather frightening to me. Especially when so many clubs that you might perform in as a band have such sketchy power. The UPS is key... but I hope you spent some extra to get one that does some voltage regulation as well. That will help.

Anecdote: I once was in a band that rehearsed at one of those rehearsal studios where bands rented rooms by the month. When the band on either side of my band was rehearsing, the voltage would sag so much my amp would sound different. I started plugging my amp into a this Tripp Light unit that would do makeup voltage and it sounded normal again. Ugh.

[meredithsdg]

I hear you on the "moving parts" thing. I've used the iPad for live accompaniment, but where I've run into trouble is in getting an accurate downbeat on the first beat. In the case of the show, I had to cue singers with one hand and start the track with the other. I find that easier to do with from the computer.

I have noticed that in the iOS music app and with DPs control app, that letting go of the "button" triggers the music. (as opposed to actually "pushing" the button.) So, I would press and hold the button and then release when I cued the downbeat.

If it hadn't been for measures that I needed to vamp, I had originally intended to put all the sequences in the Kronos. No moving parts there since it has an SSD. I've been using it for my personal performances and LOVE it.