.0 releases, early adopters and beta testing

philbrown · Post by **philbrown** » Sun Jul 12, 2015 5:57 am

I hope no one takes these questions as trolling, I'm seriously curious about a couple of things and have wanted to ask these questions for years. First off let me say, I NEVER purchase or install a .0 release of any program that's important to me like DP, e.g. I installed DP 8 when it was 8.07 IIRC, last November I believe.

1. Anyone who's been around a while, especially long-time DP users: what is your motivation for installing a .0 release when there is a long history of bugs at each .0 release? This question applies doubly to people who upgrade in the middle of important projects (and it does happen every time). There must be reasons, as so many really smart DP-savvy people here do it pretty consistently.

2. With so many *extremely* experienced DP power-users here, some of whom obviously want to be on the leading (bleeding) edge, why wouldn't MOTU use that resource and approach a dozen or half-dozen power-users here to beta test and find the most obvious important bugs BEFORE releasing, vs risking page after page of (public) complaints of bugs in a forum like this, not to mention a large volume of tech support requests, phone calls, complaints, possible bad press, etc. I'm seriously asking, as it seems they must have reasons that make sense from their company's perspective.

daniel.sneed · Post by **daniel.sneed** » Sun Jul 12, 2015 6:34 am

That's a heavy question, Phil.
I'm sure Motu already has a beta tester team.
The question turns to be: What is really important to them ? and what is not.
We can't even imagine how many wrecks and messes they go thru during testing time.
But in the end, paying customers, including me, are stumped to see *some* bugs in .0 release.
Computer and software world is definitely a strange world...

Phil O · Post by **Phil O** » Sun Jul 12, 2015 6:43 am

Yeah, it confuses me too, Phil. You'd think that many of the reproducible bugs we see would be caught in beta testing. Is there a bug threshold for releases? I can imagine someone saying, "Well, we haven't fond a new bug in over 3 days. Let's release on the 15th." or "We've fixed all the serious bugs and 85% of the minor bugs, lets release and start working on .01."

Or maybe their beta testers suck...

Phil

Robert Randolph · Post by **Robert Randolph** » Sun Jul 12, 2015 6:59 am

philbrown wrote:I hope no one takes these questions as trolling, I'm seriously curious about a couple of things and have wanted to ask these questions for years. First off let me say, I NEVER purchase or install a .0 release of any program that's important to me like DP, e.g. I installed DP 8 when it was 8.07 IIRC, last November I believe.

1. Anyone who's been around a while, especially long-time DP users: what is your motivation for installing a .0 release when there is a long history of bugs at each .0 release? This question applies doubly to people who upgrade in the middle of important projects (and it does happen every time). There must be reasons, as so many really smart DP-savvy people here do it pretty consistently.

Because it's rarely an issue. You may be biased by the vocal minority that experiences bugs. There's plenty of us that have little to complain about though. Why would I bother posting about the bugs that I don't have?

I also keep the last stable version installed if I need to. Sometimes I do, sometimes I don't. DP9 has been very stable for me so far.

philbrown wrote: 2. With so many *extremely* experienced DP power-users here, some of whom obviously want to be on the leading (bleeding) edge, why wouldn't MOTU use that resource and approach a dozen or half-dozen power-users here to beta test and find the most obvious important bugs BEFORE releasing, vs risking page after page of (public) complaints of bugs in a forum like this, not to mention a large volume of tech support requests, phone calls, complaints, possible bad press, etc. I'm seriously asking, as it seems they must have reasons that make sense from their company's perspective.

Marketing. Development.

Firstly, public betas can tend to create a public perception of instability of software. It's also difficult to market something as new and amazing if it's already been in the public's hands for months upon months. You could argue that a buggy x.0 release is also bad for marketing, but that's nothing compared to running an easily accessible open beta.

The second issue is development. Managing a large beta test, if you want to do it properly, is very difficult. A LOT of time ends up spent chasing ghosts. For companies with a limited number of eyes on the code and tracker, fewer testers is preferred usually. It also helps a lot when you can choose your testers to only include people that understand the process and are capable of providing clear, concise and reproducible bug reports. Yes some bugs get missed, but from my observations as a DP user and as a developer (formerly professional, currently hobby), it's largely system configuration issues that get passed on as 'major bugs'. These are very difficult to find in a closed beta, and it's rarely worth the hassles of a large open beta to try and deal with poorly configured systems.

FMiguelez · Post by **FMiguelez** » Sun Jul 12, 2015 7:16 am

I'm one of those early adopters, but I only do it in a "test clone" drive.
And I'm one of those who is not too happy with 9.0, as its numerous little bugs are greater than its new features.
Why do I do it? I can't help it... Most of the new release features MOTU comes up with, are very often quite desirable for me.
However, this is the very first DP upgrade I regret installing. Quite buggy. I'm one more bug away from going back to 8.07.

OTOH, it's a shame MOTU doesn't take advantage of the numerous experts and power users we have here in MOTUNation. If they made some of us beta testers, as you asked, we would have reported most, if not all, DP9's bugs.

I can't believe so many obvious bugs went right past the beta testers this time... Like the bug where the Track Selrctor and Channel Strip windows can not be closed with a shortcut. How can they not notice these things?

Michael Canavan · Post by **Michael Canavan** » Sun Jul 12, 2015 7:40 am

The other DAW I use all the time is Ableton Live. Live goes through a public beta. Even with a public beta things can go haywire, Live 8 beta a few years ago was stable on most peoples systems, I was one of the few throwing up red flags saying it wasn't ready for release while most people were on the forums talking about how great it was. it didn't take but a couple of months before the forums and the whole user base really was complaining about Live 8 being an unstable mess. That beta period was about two months long, and it still got released buggy.

MOTU could easily run into that problem, people in a community like this could test it and have no issues, then dude using configuration x variant different than anyone who used the beta, who uses other plug ins that people on the beta aren't using, and bam, even the public beta doesn't solve the bugs.

For what it's worth DP9 is about the level of stability that 8.0 had for me, not great but workable.

mikehalloran · Post by **mikehalloran** » Sun Jul 12, 2015 9:20 am

...Like the bug where the Track Selrctor and Channel Strip windows can not be closed with a shortcut. How can they not notice these things?

Thats easy. You can't notice something that you never tried. I've never closed a channel strip window with a short cut, for example. Apparently, none of the beta testers did either.

Any program this complex needs a huge beta program to catch the bugs. It is clear that MOTU doesn't have as large a beta program as it needs. Not your decision or mine either.

A lot of the reported bugs are not. There are certain users who scream "bug!" whenever they encounter something that doesn't work the same as it did in DP 8 or 7, both of which also made many changes and went through large growing pains. Every user base has its minority of those who continually remind us that the sky is always falling. To be fair, early DP releases don't often help. Hiding old features in new sub menus needs to be fully explained.

MOTU did no one a favor by failing to rewrite the documentation. 90% or even 95% correct concordance with the DP 8 manual is not good enough when you go to the PDF file and something doesn't work as written. Putting all the changes in the Getting Started Guide isn't good enough.

9 will get better, I've no doubt. I'm finishing older projects in 8 but I'll start my next project in 9 to give it a workout. I've never used the drum editor and am holding off for now. Hopefully that will get fixed.

philbrown · Post by **philbrown** » Sun Jul 12, 2015 9:25 am

Thanks for the thoughtful relies and not bitch-slapping me for asking these questions.

FWIW, I wasn't talking about or suggesting public beta testing, just using our top 10 or 15 power users here as beta testers. I do understand the challenges of dealing with so many different systems- I'm sure it's a daunting task! Having said that, if it was me deciding, I know exactly who I would approach here and (this is just conjecture and my personal feeling) feel that they could do a bit better job of diving in deeply and reporting more of these glitches before a major release.

Obviously there is no 100% solution and of course I'm not implying that at all, nor am I implying that everyone is experiencing deal-breaker bugs with this release.

And I still love DP and recommend it (just did a couple of days ago).

EDIT: @Mike - we posted at the same time.

bayswater · Post by **bayswater** » Sun Jul 12, 2015 9:55 am

These discussions rarely get around to question of money. I don't know how MOTU is run, but if it works like most companies, there is a constant review of cash in and cash out, and inevitable conflict between development and testing, who want to keep refining the code, and others who want to get on with the release. At some point, money for development runs out, and the decision to release is forced.

I've been on both sides of the argument many times, and rarely saw a case where both sides agreed when it was time to go to production. But compared to Cubase, Logic, Aperture, FCP, TurboCad, Office, and so on, MOTU does as well or better than most others with .00 releases.

That doesn't really address Phil's question about why we might buy software that isn't ready. For me, it's curiosity. I know I'm going to spend the money at some point, the cost of buying early rather than late is negligible (what's 6 months interest on $200?), and I can keep using V8 if 9 doesn't cut it.

So what's my total risk? $5 and a couple of hours not spent wasting money on something else? And then there's wanting to see MOTU stay healthy.

mikehalloran · Post by **mikehalloran** » Sun Jul 12, 2015 10:13 am

I wasn't talking about or suggesting public beta testing, just using our top 10 or 15 power users here as beta testers

I couldn't agree more.

It wouldn't take too much research to figure out who those people should be. (Frankly, I wouldn't be one of them since my expertise is on the Mac and not DP).

Robert Randolph · Post by **Robert Randolph** » Sun Jul 12, 2015 1:15 pm

philbrown wrote:Thanks for the thoughtful relies and not bitch-slapping me for asking these questions.

FWIW, I wasn't talking about or suggesting public beta testing, just using our top 10 or 15 power users here as beta testers. I do understand the challenges of dealing with so many different systems- I'm sure it's a daunting task! Having said that, if it was me deciding, I know exactly who I would approach here and (this is just conjecture and my personal feeling) feel that they could do a bit better job of diving in deeply and reporting more of these glitches before a major release.

Obviously there is no 100% solution and of course I'm not implying that at all, nor am I implying that everyone is experiencing deal-breaker bugs with this release.

And I still love DP and recommend it (just did a couple of days ago).

EDIT: @Mike - we posted at the same time.

I'd be curious to know if those people are even interested in beta testing.

It's going to be pretty rare to find someone that is: highly proficient in using DP, has lots of free time to use buggy software, knows how to file good bug reports and reaches out to join the beta program then is accepted.

philbrown · Post by **philbrown** » Sun Jul 12, 2015 1:59 pm

Robert Randolph wrote:It's going to be pretty rare to find someone that is: highly proficient in using DP, has lots of free time to use buggy software, knows how to file good bug reports and reaches out to join the beta program then is accepted.

That's you're assumption, and probably based on experience, but still- think of all the time some people here devote week after week to help people troubleshoot problems and answer people's questions– even hand-holding newbie questions and some of those folks enjoy being on the leading edge. To people who are drawn to such activities anyway, a $50 or half-price discount might be a source of pride and appreciation for the input as much as the cash. But.. what do I know, it's guesswork for all of us really, we're not MOTU. But... it's the internet, so we can opinionate! My personal gut feeling is there are expert people here who would be willing to contribute.

BKK-OZ · Post by **BKK-OZ** » Sun Jul 12, 2015 2:09 pm

As a beta tester for a pretty complex piece of software, I can tell you that it isn't something that is as onerous or difficult as some of the posts above suggest.

There are about a dozen of us on this particular beta program. We all file simple weekly reports. The lead developer communicates directly with us through a (private) forum. Hardly any bugs sneak through to public release versions, and the beta team are, in effect, extensions of the development team. Whenever a tricky issue arises, all of us chip in with testing and suggestions. The variety of configurations we all run gives the development team a chance to have their code checked in a way they couldn't possibly replicate in-house. Issues like changes that confound users are picked up early, and they are then highlighted in 'what's new' notes and intro videos.

All of us testers are using this software in our day-day work, and as a result, we can feed in ideas about practical improvements and new features. For example, something I have been wanting to see some developer do for about 5 years is now in the next build. I took some time about a month ago to map out in detail what I thought was needed, and voila! it's now here. Great for me, for other users, and another selling point for the software.

It all requires a moderate investment of time and energy by the company and the beta testers. This one new feature on its own justifies my time.

In short, it ain't rocket surgery.

toodamnhip · Post by **toodamnhip** » Sun Jul 12, 2015 3:16 pm

I have to honestly express an opinion here, and that is that MOTU appears to be quite biased in how they choose their beta testers and who they allow as spokespersons for the product.
I can understand that I have had opinionated brawls a time or two at the forums and maybe that is why they never allowed me to Beta, (even thought TECHNICALLY, I am involved with many other companies testing and really NEED to be able to get MOTU data on a backline channel). But I know of at least TWO heavy cats here that who want to, but are not allowed to beta and it makes absolutely no sense. It has always been difficult to get MOTU to be more public about anything when it comes to bugs and developments, the companies general tendency is to be closed off and somewhat inaccessible.
I have discovered major bugs in the past that were a danger to a projects very survival, (such as when an upgrade in DP 5 resulted in sys ex data being lost on quit.).
Such lack of public information on some bugs is borderline negligent. For ex, right now in DP 9, plug in mute is sometimes working, sometimes not working. OK, how about a public bug announcement so that users relying on their mixes working don’t release messed up mixes from DP 8? Such a bug is serious.
As the yrs have gone by, MOTU has slowly become a bit more open to the public, even allowing the words” bug fix” to come out occasionally.
But the company has a long way to go when it comes to reaching out to anyone not in their preferred click. And how any given beta tester becomes one must be one of the great secrets of the musical universe..lol right on up their with understanding pythagorean string theory.
That said, anyone can write to Magic Dave at MOTU and get a personal response..that IS great.
But I 100% agree with your question about MOTU adding MORE beta testers. As a long time user of DP, I’d rather not have these discussions in public.
By the way, the other thing MOTU can do to IMPROVE their products is make their tech links MORE forgiving. If one little box isn;t filled in like the system wants it, you can’t submit. The other day I had to give up on filing a bug report because the stupid interface got stuck on some small question and would NOT accept data input. As a result, I could not report and gave up due to lack of time that day. (Often, when someone files a tech link they are MID work session and don’t have time for finicky, unforgiving input forms).
Now I m annoyed at this and don’t want to go back to the tech links. (I am sure I will eventually).
Tech links should be set up to accept whatever data a user wishes to put in it. Because users are providing crucial data on their own time, MOTU should respect that and TAKE the data..even if a “fill in” box is incomplete. Such obvious buggy-ness in DP 9 and a tech link system that would ever boot out one’s report are clear signs this company needs to open up it;s ears to a wider array of input.

MIDI Life Crisis · Post by **MIDI Life Crisis** » Sun Jul 12, 2015 4:39 pm

FWIW, I've approached MOTU to be a beta tester and have been referred by other testers twice. Butt no, so when I find a bug in a release on day one I'm going to be ruthless.

That isn't the problem in 'not' using me as I also know beta testers who have told me they test for MOTU but put up an innocent front when reporting serious bugs in an open forum, as if they're not testing.

Maybe they're covering the asses in not wanting to violate their NDA? Whatever. Maybe MOTU thinks I'll be revealing or something? Again, it's clearly not my loss. It could also be that they suspect that some of us might find (read: REPORT) bugs in betas they can't fix by release, or ever, for that matter.

They get no quarter from me in that regard. If I'm not testing and find bugs on day one of a paid update (or any update for that matter) then it's going to get a post and no gentle wording in the report.

And if that sounds too "brutal" or "harsh" remember it's a company that is taking good money for a product. Nothing more. Nothing less. If it's got a dent in an otherwise chrome plated ball, it just ain't gonna roll correctly. That's an angry customer you see in the reflection.

MOTUnation.com