Rethinking Virtual Instrument design

[Shooshie]

This is a continuation of a discussion that splintered off another discussion. I've moved it here to avoid further derailment of that thread.

One more thing.

Let's quit using MIDI velocity.

Period.

Thanks.

Now that's just way off base. Maybe you
prefer not to use velocity, but I'm pretty sure
99% of all keyboard players (and drummers)
prefer velocity to control something.

It's easy to program velocity to control different
sound aspects..filter cutoff, modulation, envelope
times, release noise levels, etc. Or you can program
velocity to control nothing, if that floats your boat.

Sorry about the "period" in the "let's quit using MIDI velocity" sentence. You're right; it has its place. But MIDI Velocity works like this: it sets the initial attack level, while acting as a "note-on" marker. That's basically all it does. So, when a VI uses it to control expressivity, there is a problem from the design point-of-view. We need the note-on, but if you're using an expressive instrument, velocity is absolutely the worst thing on earth for setting a level. It sets the level and then it's out of there. It does nothing else. So, the designer uses it to choose the "velocity layer." That is, the set of samples recorded at a particular loudness range. Most library interfaces now have velocity X-fade, but why even force you to turn it on? It should be on by default. If you do a crescendo, you must be able to rise through velocity layers as you get louder. Likewise, as you play softer, you go down through velocity layers. It has to be this way.

So, why do we even bother with velocity? Why aren't expressive instruments played with Expression or BC? Why do we have to set them up to do so? They should be designed from the ground up to be played with either of the two, with Expression as the default. Leave Velocity out of the picture for anything other than a note-on marker.

Velocity was designed for keyboards back when all expression was programmed into each individual "patch," by way of an envelope. We all remember the ASDR (Attack, Sustain, Decay, Release) envelope. It's ideal for pianos, drums, marimbas, and such. I would never remove Velocity from those instruments. It works for that. It was designed for that. But for violins? Saxes? Flutes? Give me a break. Velocity for expressive instruments makes as much sense as painting pictures with a hammer. (by "expressive" I mean instruments with which the player is constantly in touch with the sound, expressing it as he/she plays. I don't mean to imply that pianos or drums are not expressive. It's a MIDI term) Let's REdesign them around Expression (with a BC option) and not even get velocity mixed up in it.

Of course, the argument against this will be "but everyone has a velocity keyboard." So, because everyone has a velocity keyboard, we're going to keep making multi-thousand-dollar libraries that are mediocre by design? You know what that is? Stupid. End of discussion, vis-a-vis velocity.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Changing the subject

Let's talk about what we'd do differently with VI libraries if we were to redesign them from the ground up. Let's start by looking at what Wallander did. Basically, he identified most of the possible variables that go into an instrument's sound and character, and he created a set of controls for each. This is huge. This should have been the turning point at which all VI libraries were redesigned, but instead, the world ignored Wallander with an indignant huff. VI designers apparently have been hoping Wallander would go away ever since he came on the scene. That's just sad. It's also stupid. But mostly it's sad, because it reveals a level of willful ignorance in the VI developers which pretty much disqualifies them from the very profession they've chosen. It's a choice to go with the status quo, believing that they cannot design something that someone would want badly enough to buy and learn an expressive controller. So, they stick to mediocrity, and try to convince us through marketing that their mediocre VI is sooooo superior to others that we should pay them thousands of dollars for the privilege of limiting ourselves to mediocrity.

I'm sick of it.

But when I say that, someone invariably pipes up and says "don't forget... these are just VI's for mockups. They cannot replace real instruments. Nobody's trying to sound great on these." Again, a stupid argument for mediocrity. Nobody's sounding great on these things, because they are made in such a way that it's VERY, VERY difficult to sound great. But Wallander showed us that this is not necessarily the only possibility.

Greatness in a VI is a possibility. It's right there waiting for someone to make it. Some have come very close. VSL is close. It's maddeningly close, but just... not quite there. Wallander is close. Some of his instruments are truly amazing. Sample Modeling is VERY close. (I don't own it, but I've heard people doing fantastic things with it.) But when it comes to a solo flute or violin... nobody is close, at least, not that I've heard. Yet I'm convinced that the technology is there. The samples are there. It's the programming and design that are lacking, because nobody is willing to put that kind of investment into what could be considered esoteric. The market at the top, for the finest players, is pretty damned small. But there aren't going to be any great players if nobody makes great instruments! Catch 22? Maybe. But maybe not, if we could come up with some design standards that make deep programming possible in ALL virtual instruments, but not necessary if you want to stick with mediocre. (nothing wrong with that if you really are just doing mockups)

So where do we start?

One thing that makes Wallander's instruments great is the ability to change the formant. You can modify an instrument's sound to a great degree. For example, if you are setting up a saxophone sound, messing with the formant is like experimenting with mouthpiece and reed. You can get a dark sound which is more classically oriented, or go the other way and get a brighter, more harmonically snarky sound for rock or jazz. Go very far and you'll ruin it; same goes for mouthpieces and reeds, really. Imagine if all instruments had this ability. No two players would sound alike. Everyone could set the instrument to sound the way they want it to.

• 

Then there are little details like chirps in attacks, pitch variations during sustain, vibrato random variation by some small percentage both in speed and amplitude. Real players don't sound like an LFO when they make vibrato. The peaks vary. The speed may vary — all very slightly — many times per second. Wallander allows this kind of control, and as long as it's set to low levels, <10%, it's very effective. This is on top of your normal speed and amplitude controls, which are of course separate. Of course, I prefer vibrato that I create, but I've found very few MIDI instruments which can respond to that kind of subtlety at that speed and not get mixed up in the latency of either the VI, the controller, or DP. Here are the controls for setting up vibrato:

• 
  
  
  
  

This is just the tip of the iceberg. Of course, you can set up an instrument to be as complex or as simple as you want it. Naturally, you don't need or want a dozen controllers on every section instrument in your orchestra. That's just for solo instruments. The section instruments can be played with 3 or 4 controllers, or even less if you want.

Here is the control for mutes. It changes as you select different kinds of mutes. A Harmon mute is not the same as a plunger mute, for example.

• 

Let's look at a partial list of the parameters available in Wallander Instruments' WIVI interface:

4.1 General Settings

• - Overview
  - Amplification
  - Hard Panning
  - MIDI Channel
  - Breath CC
  - Breath CC (LSB)
  - Pitch Bend CC
  - Instrument Range
  - Pitch Bend Depth

4.2 Instrument Tuning

• - Overview
  - Tuning Editor
  - Reference Frequency
  - Octave Stretching
  - Transpose Instrument
  - Edit Tuning
  - Pressure Dependence
  - Out Of Tune
  - New Detuning
  - Scale Override

4.3 Vibrato Control

• - Overview
  - LFO
  - Vibrato
  - Tremolo
  - Vibrato Delay
  - Vibrato Speed
  - Vibrato Skew
  - Vibrato Amount CC
  - Vibrato Speed CC
  - Randomize Depth
  - Randomize Speed

4.4 Tone/Timbre

• - Overview
  - Mute
  - Wah-wah CC
  - Wah-wah Detuning
  - Tone
  - Breath Noise
  - Chiff Noise
  - Key/Valve Noise
  - Lip Buzz Amount
  - Lip Buzz Depth
  - Lip Buzz Speed
  - Angle Dependence
  - Falloff
  - Left/Right Amplitude Separation
  - Left/Right Phase Separation

4.5 Control Parameters

• - Overview
  - Sequence Editor
  - Beats
  - Quantize
  - Synchronize Start
  - Note-breaking
  - Sequence Time
  - Time CC
  - Velocity Mode
  - Use Breath CC
  - Use Pitch Bend CC
  - CC 3/CC 4
  - Sequence Mode
  - Polyphonic Mode
  - Voice
  - Always Play
  - Release Trigger
  - Key Priority
  - Retrigger Notes
  - Release Velocity
  - Graphics Layer
  - MIDI Channels 2-5
  - Volume/Pan/Sustain CC
  - Volume CC 1-2
  - PanCC
  - Sustain CC
  - Breath Envelope (ADSR)
  - Starting Level
  - Starting Level CC
  - Starting Level CC Dependence
  - Attack Time
  - Attack Time CC
  - Attack Time CC Dependence
  - Attack Shape
  - 1st Sustain Level
  - 1st Sustain Level Time
  - Decay Time
  - Decay Shape
  - 2nd Sustain Level
  - 2nd Sustain Level Time
  - Instrument Delay
  - Release Time
  - Release Shape
  - Breath Controller Speed
  - Breath Pressure Threshold
  - Key Attack Speed
  - Velocity Range
  - Key Release Speed
  - Randomizer
  - Randomize Time
  - Randomize Pitch
  - Drift Into Tune
  - Performance Jitter
  - Performance Shimmer
  - Sustain Time
  - Key/CC Switch
  - Breath Compression
  - Breath Max Value
  - Portamento
  - Transitions
  - Legato
  - Frequency Jitter
  - Level Shimmer
  - Pitch Bend Alignment

5 Templates

• - Overview
  - Run/Export Template
  - Simple Example
  - Advanced Example
  - Randomization
  - Conditionals
  - Scala Tunings
  - File Depository

In each of the above options, there are usually more than one CC controller involved. They all use MIDI Learn, so they are very easy to set up, and except where not appropriate, they all have on-screen knobs or other handles for setting them and forgetting. Once you get it set as you like it, you can save the settings and recall them, even as templates for another instrument. Your playing preferences are easily portable.

I post these as examples of what can be done. I envision a developer applying these to their own VIs in ways that make sense to them, making them more transparent and easy to use for the final user, but still accessible for those who wish to dig deeper.

How would a new VI, a synthesis of sampling and modeling, work? What would it look like? I don't know. I'm not a Virtual Instrument engineer. But it would involve getting people like me (not necessarily me) to work with the programmers during the design phase, then to test the instruments and offer suggestions. What's different about this than previous design processes? By design, we need to incorporate EXPRESSIVE MIDI INSTRUMENTS. One should not have to compromise the VI in order to use an expressive instrument. The capabilities of expressive MIDI instrument controllers should be reviewed and taken into consideration from page 1 of the design phase, all the way to the final testing.

Expressive instruments should not be designed primarily for keyboards, but keyboards should definitely be included, using as many expressive elements as possible on any given keyboard. Breath Control should be considered a standard, not an option, for keyboards playing expressive instruments. Expressive Pedal boards should be considered common accessories.

I'll stop here for now. Please, someone else respond and say something. I talk too much as it is.

Shooshie

[HCMarkus]

Clearly focused from a wind player's perspective Shooshie, but great stuff all around. We need programmers who play instruments other than keyboards, as we keyboardists get used to thinking in terms of velocity instead of continuous control. The thing I find most endearing in a VI, that makes the VI a true instrument, is the range of detailed, pleasing sounds it produces and how it responds to my input. The possibilities offered by such an instrument induce the performer to take the time to master it.

Most VIs generate sound with sample playback. To get the level of control Wallander gives us, we must use more basic building blocks, and we are back to the FM vs Sampling discussion we had earlier… sampling sounds ultimately real, but static. FM is so expressive, but lacking the character a sample can contain.

If mankind can AutoTune, and we can fit a close approximation of a complex recording into an itty-bitty MP3, we can design a synth, likely using Fast Fourier Transform, that will give us the expressivity of FM along with the complexity of sampling.

[Shooshie]

If mankind can AutoTune, and we can fit a close approximation of a complex recording into an itty-bitty MP3, we can design a synth, likely using Fast Fourier Transform, that will give us the expressivity of FM along with the complexity of sampling.

And if Google can map the world and photograph it, seamlessly, so that you can just about take a vacation at your own computer, why can't we map sounds and assemble them seamlessly, too?

Imagine a spectral display. Now, cut it up in pieces. Make each piece a recorded fragment. Make such recorded fragments of instruments playing all kinds of music, and develop an algorithm for creating those fragments of sound — I'm talking bits like harmonics, air, bowstrings, finger slides, reed buzz, string buzz, fundamentals and entire harmonic series of each pitch — then make layers of these representing the same sounds under duress, or caress, and so forth; then make an algorithm that puts them all back together in real time. Does that sound impossible? It's not unlike what Google has done in "Street View" of their mapping systems, or in the satellite imagery that appears mapped over terrain, as if you're really flying over it. It just happens a lot faster... perhaps. That's at least one idea. Maybe there are better ones, but I'm just trying to get away from the "model OR sample, but not both" mentality. There has to be a way to combine the best of both worlds. IRCAM is doing the impossible. Why not incorporate some of their ideas?

With the right approach, sample libraries could be made in hours or days, not years. Maybe even minutes.

For wind players, imagine an instrument with not ONE transducer in the mouthpiece, but 5 or more. Each senses an area of the pressure of the lips, and changes the sound accordingly. Bowings could be mapped to such pressure points, making fairly natural performance. The ideas are just starting to flow.

Shoosh

[David Polich]

Shooshie, I understand your points. There are two things which
are a reality about the whole virtual instrument "industry".

One..wind and breath controllers are gone
and are never coming back. Yamaha invented
the concept and they discontinued production
of those, for the simple reason that they
weren't selling.

Two, the entire VI industry grosses less in a year
than a single Albertson's grocery store does in
a week. There just isnt that much money in it.
It takes money to pay talent to sample stuff,
edit the data (which does take longer than "a few
days", trust me), design GUI's, etc. We should
all be thankful there are companies and people
who even want to make these things for us.
Because they really need to sell as much
as they can, they have to make stuff that
does not require months to learn, is easy to
understand, and can be played from a conventional
keyboard controller, which is what the
majority of customers have.

Everything you listed, software developers
have thought of. Its not a question of whether
they want to implementit, it's a question of
do they have the resources and time, and
will it sell. If the answer is no to either question,
then it isn't going to happen.

Please dont take this the wrong way..but if
you're so keen to have VI's that respond the
way you want and sound the way you want,
why not develop them yourself? It's a valid
question.

[stubbsonic]

Velocity is GREAT for what it does. It lets a piano player or keyboardist think in terms of "percussion" and set the initial attack level. That is fine, as far as that goes. And it can be used to set kind of a general dynamic for playing, but as a brass player, there is so much that happens after the attack that we need at least one (probably two) other streams of data to control the sound post attack. In other words, I completely agree that it is WAY overdue to let velocity join a range of other controls to either crossfade sample layers or otherwise change dynamics. But "quit using MIDI velocity"? Balderdash.

After playing with Reaktor's Silverwood instrument with a breath controller, I got a satisfying sense of what is possible. Though it was not "realistic" sounding per se, it was expressive.

There are a bunch of ways people can physically train/adapt to controllers-- breath control, foot control, aftertouch, faders, knobs, ribbons, gesture sensors like Leap Motion, touch screens, and a myriad of DIY hacked devices could each be employed to control amplitude, timbre (waveshape, filter cutoff, res, distortion), pitch, effect mix or depth, flutter, growl, etc. etc.

In the VI development world, we need to see more kinds of physical modeling hybrids. The examples Shooshie gives have always impressed me. I'd like to see more wind instrument models out there in the VI world-- especially those that let you design unreal new sounds.

In the MIDI controller world, we are seeing some new controllers that offer more dimensions of control, but in watching them being played, they seem difficult to control. A bit like what happens when I watch & listen to theremin players. I think, well, that must be hard.

I'd like to see keyboards with EXCELLENT and adjustable velocity and polyAT response, but also lots of inputs for CCs.

Finally, people who use additional CC's to good effect need to be seen & heard doing it to good effect in the mainstream culture. People will say, I've never seen or heard anything like that. I want to learn how to do that!

[SixStringGeek]

OK, I get it - you want more control parameters. That's cool. But for every one of those parameters I have to have a way to tweak it in real time.

I'm gonna go with your formant example - mostly because I get it.

Guitarists are fond of saying "the tone is in the hands". It is to an extent. When I pick up a guitar, my picking sounds like me. BB King sounds like him. Why? On a technique level a lot has to do with how you hold your pick. How tight and how much you choke up on it. Stiffness and snap. Because when you pick the string, your thumb will graze it just a bit and cancel certain modes of vibration or overtones. Tiny changes in the way I hold my pick can let a string ring with all overtones like a plain old open pluck - or I can choke way up on it and do pinch harmonics - that squealy thing rock guitarists do when they actually cancel the fundamental and leave the upper overtones only.

K - nifty. So I have to replicate this very subtle lil thing I do as a control voltage encoded into a number mapped to a dial or some other sensor and feed it to my virtual instrument and its just one channel of data I'm generating while I play.

The bit that kills me is you've listed a zillion channels of data you want to feed into this model to vary things - which I get - we all do it on our real instruments - but I don't know how to manage that many data streams in real time on anything but a flippin guitar. I got a breath controller - that's cool. Swells and stuff. Its one more data stream.

But I don't think the problem is the instruments - its the controllers we have available to manipulate them. Imogen Heap and her experiments in data gloves and gestures might be on the right track. But its still very experimental.

[Shooshie]

Really, 99% of those parameters are set-and-forget. There are just a few that you need in real time. Loudness, pitch, vibrato (if separate from pitch), and maybe a wild-card controller you use for different things in different instruments. But the ultimate goal is to pick up the controller (the actual instrument in your hands) and just play.

It seems harder than it is. On my Yamaha WX5 there are several continuous data streams that I control in real time. Add to that my two expression pedals. The thing is, after a while, you don't think about the controller you need to be moving. You think about the sound you want, and you find your hands and feet "just doing it." That's the way all instruments are. At first you have to learn them, but what CAUSES you to learn them is the desire to hear a certain effect. Suddenly your brain wants to complete those neural pathways, and it clears the way, because IT wants to hear the result. Bam... you've learned it.

It's just like the pick scenario you described. Very few people know what you just told us. They just do it. But if they stopped and thought about it, they'd figure it out and say, "yeah, that's what I'm doing. It's how I'm holding the pick, and where I'm picking the string, and the pressure I'm using in my left hand, and how much of the string I'm covering with my left hand's fingers, and so on and so on. But they didn't learn it that way. They just wanted to hear something a certain way, and kept at it until they got it. The MIDI instruments work that way, too.

I heard a guy in a country band play sliding steel guitar on a Kurzweil 2600 one time, and it sounded like the effing real deal. I watched how he did it, and it was a trick that required split second timing with his fingers. But what you heard was a steel guitar with one string sliding up while the other one stayed stationary, and he made it work. That was something he just did until it clicked. To this day I cannot repeat how he did it, but I can still see his fingers doing this little opposing flip, and you heard that characteristic steel guitar slide and twang.

These things ARE instruments, and we do learn tricks on them like we do on all other instruments.

I wish my son would get rich and retire early. He's a primo programmer, and if he were willing to work with his dad, we might develop something. David Polich is right; there's little money to be had in this. Whatever you do has got to be a labor of love.

Shooshie