Does converting MIDI files to MP3 files require creativity?

Question

MIDI files record which keys are pressed when, how hard and for how long. Given a sufficiently detailed physics model of the instrument, it should be possible to simulate it in silico (at least in theory, although this particular approach might be computationally expensive in practice) to produce the sound with no creativity whatsoever. If this is incorrect, then why?

Answer 1

In theory, perhaps just about if it was a simple performance eg. solo piano, and you got very very lucky with the sample set you paired to the midi.

However, bear this in mind, even if you have a fantastic piano player playing on the most expensive digital stage piano available then most experienced/professional musicians would still be able to detect that the 'keyboard instrument' was not a real piano. Now it's possible, with a powerful computer, fast hard disks and an expensive piano sample library that you could get quite a lot closer, however this might take a fair bit of tweaking to get rid of any last little artefacts that sound artificial. The resultant midi file would probably end up being fairly specific to THAT piano sample set, and may not work so well if you tried it using a different sound. Sure it would sound 95% there if you switched sounds but you might have to go back and tweak it again. Playing music is after all an interaction with an instrument and a feedback loop, how you play is adjusted in real time depending on the sounds you hear coming back from the instrument so a midi file captured from an Upright acoustic piano (one of those models with a midi module of course!) is unlikely to CONVINCINGLY translate directly to a Steinway Grand, as the performances would be different.

So for the MIDI file to sound suitably flawless you would need a soundset very close to what the original musician played it on.

Additionally, there is just SO MUCH to model/sample on a piano that even the biggest and most expensive sample sets don't paint a complete picture, though some of them are very good. As a point of reference I've seen some super over the top piano sample libraries aimed at the professional MIDI worker to top out close to 500Gb, though in truth 50Gb is a bit more normal ;)

Moving onto something like drums and you have a situation where drummers routinely use more sounds than are commonly included in a midi drum soundset. For example a snare soundfont might have center snare, rimshot, maybe a sidestick, each at all the different velocities. But a drummer may be intentionally varying where they are playing on the snare skin all the time, to put in ghost notes and fill out the groove with many subtleties. Dead center for the backbeat, off to the side for ghost notes etc. These details would probably not be picked up on the midi file recorded from the drummers inputs. Similarly there are many ways to play each cymbal, with drummers often taking many years to get their 'jazz ride' playing sounding good, attacking the cymbal in different ways, sometimes while it is already moving with different levels of energy to get it doing what they want. On a midi drum track you generally just get 'Standard Ride' and 'Bell', with a sample set that doesn't typically take into consideration what 'state' the cymbal was previously in when it is hit again (it just mutes the previous sample and plays the new one, this is not how it works in real life).

Of course, you can set up your own midi drum tracks to include more sounds, and experts at creating midi based music for film scores or similar do just that. They may use 6 different midi notes for the snare (say for a funk track) and expand the palette out to include different ghost notes etc, and can, with work, create something very convincing. This is a highly skilled job though, requiring in depth knowledge of the instruments you are trying to emulate. Whats more, the midi file created in such a way would be very non-standard, and would not work being played back on the general midi soundset. It would have to be delivered along with a set of instructions of what sounds had been mapped to where. And then, again, we come up against the problem that that the highly detailed midi file would probably only sound RIGHT with a pretty similar soundset to that which the midi file was created for, and for exactly the same set of velocity curves etc. on each of the samples, otherwise all that detailed editing to make it sound 'real' would be messed up by a scaling of velocity values etc.

If you own the same virtual instrument that the midi file was created for, then you may have more luck. Many modelling/sampling virtual instruments use extra midi data to control various articulations/details SPECIFIC to that particular software which would be junk on any other software (unless they use the same standard) but, I'm getting into the long grass now as such files would have to have been creatively worked on at some point, anyway.

So it's possible to re-create near-convincingly, though I have only covered two of the easiest instruments to do it with here. Guitar, voice, wind etc. become much harder and are notoriously difficult to convincingly model/sample. Firstly because there's no real interface that can capture player input, eg. on-off note info, in a realistic way as there is for piano and drums (though they have tried over the years, see midi breath controllers, or the Suzuki Omnichord), and secondly because those instruments have various physical or performance complexities like non-fixed pitch or complex lung-sax interactions etc. This is where you REALLY need MUCH more control than just note on-off and volume, probably in the form of a dedicated piece of software that tries to emulate all the specifics for the instrument in question, there's lots of such software out there though, and it's always getting better!

And note, finally, this is with a midi expert with a great degree of instrumental knowledge, a collection of high quality sound-sets or instrument-specific modelling software, sitting at Logic for extended periods of time tweaking to create something that sounds 'real'.. By that I mean for a full song it may sound good, but an experienced musician will still be able to tell it's MIDI, as it stands in 2022 that is! That said, solo piano, I could probably be fooled..

Grabbing a standard MIDI version of a song off the internet and chucking some sounds on top of it will always sound like midi. Those files are rarely accurate to the original song anywhere near enough to sound real with a generic sound added. Though there are of course good and bad midi files to be found they won't tend to have anywhere near the detail needed to sound fully authentic, or if they do, they would need to come with some info on what sort of instruments to hook them up to and so on. Many such net-downloadable MIDI files are created by one hobbyist individual inputting notes from a score with mouse-clicks, for the general midi soundset or similar. Usually left pretty much in that 'quantised' state and uploaded. That's never going to sound like a real musical performance without significant, creative, editing!

Answer 2

It might not require creativity but it does require effort. This is because anything but the most basic MIDI data is instrument specific.

For example, suppose you want to take MIDI data that is meant to be used with a violin sound. Each different virtual violin instrument will react different to different MIDI parameters. It’s mostly standard that CC 1 and 11 are used for dynamic layer and volume, but which specific CC values are mapped to which volume varies from instrument to instrument. Key switches are often very different between instruments. And pretty much every other CC number will map differently, as will the velocity values. So MIDI data that sounds perfect when used for one virtual violin may sound terrible when used for a different one.

Furthermore, the virtual violins are actually closer to being standardized in terms of MIDI than other types of instruments. If we look at synthesizers (virtual or physical), all bets are off. Almost any CC number could be mapped to any parameter or no parameter. Velocity could control loudness or filter cutoff or envelope attack time or nothing.

Last but probably most, MIDI data usually doesn’t include the selection of sounds in the first place, unless the MIDI data includes General MIDI sound selection. So if you have a General MIDI sound bank and a MIDI piece that selects sounds for each channel from a General MIDI sound bank, you’ll at least get the right sounds. Unfortunately, General MIDI sound banks are simplified sound sets, which they have to be to comply with the General MIDI standard. So while the results using General MIDI will be more predictable, they will also be less musical. There will be very little expression or dynamics that will come through a General MIDI sound bank.

In summary, the quality of the audio that results from MIDI data is directly proportional to the work put in by a human interpreter on adjusting the MIDI data to work best with the particular instruments used. It’s a bit like a conductor interpreting the music on the page and directing the orchestra on the details of interpreting it. The life of the piece is created in the room by the humans, the score itself is just a guide that requires a lot of work to convert to actual sound.

Answer 3

It depends. MIDI contains only a very limited subset of what goes into actually playing an instrument, just enough for piano-like ones but not for most instruments. If you take the effort to edit the synth's parameters to make the result sound better and maybe edit the recording to fix things afterwards that should count as creativity. As far as I know, "sufficiently detailed physics models" also aren't there yet. Just pressing the button to feed a MIDI file into General MIDI synth does not count.

This could be compared to recording engineer's job: Just putting a mic in front of musician and pressing rec doesn't take much creativity but getting the best recording out of a performance is an art on its own.

Answer 4

You say

Given a sufficiently detailed physics model of the instrument, it should be possible.... to produce the sound with no creativity whatsoever.

And you are right that given a model that converts MIDI input to audio output, little further creativity may be required. (The model doesn't necessarily need to be a physical model of the instrument - there are various ways of synthesizing sound).

However, creating a model that converts from MIDI note data to audio would normally be seen as a creative endeavour in itself, and in fact that activity has a name - synthesizer programming.

Answer 5

Your question is about keyboard instruments. Yes, assuming that the keyboard is the only device through which we play the instrument, MIDI can do a pretty good job of capturing and reproducing what is played. And yes, 'Given a sufficiently detailed physics model of the instrument' no creativity is required for playback.

That's your question answered. We could go on to discuss how WELL we have achieved this modelling for different instruments, and extend the scope beyond keyboard instruments. I'd say we've done pretty well on piano, organ etc. Reasonably well on most orchestral instruments. Solo strings, saxophone and voice still need work.

We can focus on how well the job has been done, or on how badly! I'd just remind you that much of the music we hear today demonstrates that it hasn't been TOO badly! But the continuing existence of real orchestras performing live music shows that it's not yet a perfect job.