Mediaboss Marketing

Singing is an extraordinary human skill. It requires the ability to form words, then the ability to vocalize them at a certain pitch and finally the ability to synchronize this with the notes. For many, it comes naturally humans seemed to be hard-wired for singing.Not so machines. Teaching a computer to sing to turn a musical score into vocalized song has turned out to be hugely frustrating. First, these devices must master the ability to turn text into speech, which is itself an ongoing challenge in computer science. They must then match the words to the notes at the level of syllables and even at the level of phenomes. Finally, these phenomes, syllables and words need to be vocalized at the correct pitch and for the right duration. That turns out to be hard. Various groups around the world have attempted it, sometimes with impressive results. But in each case the final output requires significant tweaking to achieve any level of realism.

That could be about to change thanks to the work of Peiling Lu and colleagues at the Microsoft Technology Center Asia. This team has been working on a way to give the companys chatbot, Xiaoice (pronounced Shao-ice), the ability to sing. The new singing bot is called XiaoiceSing and the results are impressive.First some background. The task in singing voice synthesis is to turn a musical score into a voiced song that is indistinguishable from a human effort. Lu and co point out that a score consists of the song lyrics along with the song notes and note duration. For a professional human singer, it is straightforward to turn this written information into a song.But for a computer the task begins by translating the score into machine readable form. XiaoiceSing does this by dividing the worlds into phonemes and then allocating a pitch and duration to each. This can be expressed in the form of a vector that a computer can read.But this translation process is tricky. Every word is made of syllables and these, in turn, are formed from phonemes. For example, the word sing is a single syllable made up of three phonemes. The score could suggest the entire word be sung for several beats. But the problem for XiaoiceSing is to divide those beats between the phonemes. Should it place equal emphasis on each phoneme or more on the middle or final phonemes? Just as important are the pauses between notes when nothing is sung. The human ear is hugely sensitive to this pattern which plays an important role in the rhythm of the sing. That makes small differences generated by a machine all-too-obvious. Then there is the problem of hitting the right note. When a human sings, the sound is made up of lots of frequencies. The combination of frequencies differs as the note and its quality changes, for example, when singing different phonemes.

In general, the actual note is the lowest frequency sound the fundamental frequency. This tends to be the loudest and the one the human ear most easily picks out. But the quality of the sound is determined by the other frequencies which form a kind of envelope around the fundamental frequency. The task of producing the correct envelope for a given phenome and the correct pitch is far from easy. And any mistake gives the impression of signing out of tune.Pu and colleagues tackle all this using a variety of machine learning techniques and applying them to proven technologies. For example, XiaoiceSing uses a text-to-speech system called FastSpeech, a technology that many of this team developed at Microsoft. The output from FastSpeech must then be decoded and vocalized, or vocoded. And for this, XiaoiceSing uses a speech synthesis vocoder called WORLD, which must be trained to produce a human-like sound.All this training is done with a dataset of 2297 Mandarin pop songs recorded by a female professional singer and then divided into 10 second sections. The machine essentially learns by associating the spectral features of the human song with the machine-readable score. And repeating this with over 10,000 samples from the dataset.Then, given a new score the machine has never seen, it can produce a human-like output.The results are impressive. Here is a short song sung by a human. And here is the same song produced by XiaoiceSing from the score.Not bad! And for comparison purposes, the team also output the same song using more conventional machine-learning techniques. Judge for yourself. But in their own tests which involved asking listeners which machine-sung version they prefer, XiaoiceSing repeatedly came out on top. The team have more examples here.That sets up an interesting future for singing. Songs sung entirely by computer-generate characters are already a feature of certain pop scenes. But they are far from perfect. XiaoiceSing isnt perfect either but it is an interesting step forward. A potential popstar in the making? Ref: arxiv.org/abs/2006.06261 : XiaoiceSing: A High-Quality and Integrated Singing Voice Synthesis System

Read more from the original source:
The AI Popstar Learning To Sing Like A Human - Discover Magazine