Math news: Using math to blend musical notes seamlessly

news Mar 31, 2020

Portamento is a musical term that refers to the gliding of a note from a single pitch to another. Unfortunately, the only instruments that can accomplish this are those that can continuously vary in pitch. These include stringed instruments, trombones, as well as your own voice!

This is where a brilliant MIT student called Trevor Henderson comes in. He managed to create an algorithm that can produce a portamento effect between any two given audio signals. Not only that, but it can do this in real-time. For example, this algorithm would be able to go from a piano note to a voice without a hitch. This also applies to multiple different songs.

This year, his paper won the “Best Student Paper” award at the International Conference on Digital Audio Effects, showing the importance of his work.

How does it work?

His algorithm is based upon the geometry-based framework used to determine the most efficient way to move data points between two different configurations called “Optimal Transport.” This framework dates all the way back to the 1700’s, and today is found in most production chains, 3D modeling institutions, and even your PC’s graphics.

Today, Trevor is a graduate student of computer science, and this algorithm is his crowning achievement. It works by first breaking the audio into small pieces. Afterward, it finds the ideal way to move the pitches in each of these so that they produce the desired effect. Naturally, this is only a simplified version, and the algorithm itself is much more complex.

“Optimal transport is used here to determine how to map pitches in one sound to the pitches in the other,” said Henderson, who completed classical organ training, and is currently playing at MIT’s radio station “If it’s transforming one chord into a chord with a different harmony, or with more notes, for instance, the notes will split from the first chord and find a position to seamlessly glide to in the other chord.”

Is This Useful?

Henderson states that this is among the first techniques that has successfully applied the optimal transport framework to the transformation of audio. He’s currently using this technology to transition from song to song on his radio show without anyone noticing.

This is one of the potential uses of this algorithm, DJs could utilize it to seamlessly go between tracks during their performances. Potentially, this could even spawn new genres of music.

Henderson’s mentor states that whenever Henderson found a subject interesting, he would immediately jump to ask how it can be related to music. Luckily, the field of mathematics is linked with practically every aspect of the human condition.

Because of this, Henderson has had no shortage of musical discoveries due to math. In fact, most chord progressions, and all of music theory are rooted in mathematics. There are even studies that show musical practice enhances mathematical ability.

The best thing about math is precisely this, it can weave itself into any pursuit. There’s no shortage of successful mathematicians going on to pursue a career in something else, and having math help them out on that journey.

Think about it, even something as seemingly unrelated as a game of football is built upon math. In fact, professional bettors and casinos thrive off of this. Subjects such as game theory are often used in order to determine the winner ahead of time.

Henderson’s achievement is much like this. He claims that optimal transport is much like a lazy way to build a sandcastle. The framework is used in order to figure out where every grain of sand should be in the sandcastle while making you move as little as possible from your chair.

Essentially, it is best approximated by a model that will tell you how to get one shape from another using the least amount of motion. Now, applying this to audio signals isn’t exactly easy, seeing as we usually don’t interpret sounds as shapes. Luckily, Henderson noted that since sounds travel through air, he could use the frequency and amplitude in order to present them as shapes.

Usually, song transitions are done by fade. This means that a given signal is slowly silenced, while the other gets louder. Henderson’s algorithm is a huge improvement upon this technology. It smoothly transitions from one song to the next, so smoothly in fact, that the human ear cannot detect it.

Even though for most students, this would be the peak of their student life, Henderson isn’t quite done with it yet. He believes that he can create more than just portamento using his algorithm. By taking the output of this effect, and feeding it back into the input, he thinks he’ll be able to create yet another classic musical effect- the legato. Legato describes a seamless transition between two distinct notes, so it is essentially the opposite of portamento. While portamento plays all of the existing notes between the two, legato just transitions.

If he manages to do this, there’s no telling to where Henderson will take his math/music combination next!

Closing Words

Henderson’s work stands as a testament that there is not a single field of human endeavor math is irrelevant in. Even the arts, which are often regarded as the opposite of mathematics take influence from it. For example, color theory in visual art is ruled by mathematical principles and different light frequencies.
What this teaches us is that we should never neglect math knowledge. After all, there’s no telling at which point in our lives it might come in handy. Even if it isn’t making a revolution like Henderson’s work, it might still be making our lives easier by say, automating a part of our job in Excel. On the other hand, who knows, it might be one of you that makes the next big musical revolution!

Its versatility is math’s greatest strength, and anyone who doesn’t seek to take advantage of it is naught but a fool.