{"id":9930,"date":"2019-11-25T20:45:26","date_gmt":"2019-11-26T01:45:26","guid":{"rendered":"https:\/\/digital.hbs.edu\/platform-digit\/submission\/deezers-spleeter-deconstructing-music-with-ai\/"},"modified":"2019-11-25T20:45:26","modified_gmt":"2019-11-26T01:45:26","slug":"deezers-spleeter-deconstructing-music-with-ai","status":"publish","type":"hck-submission","link":"https:\/\/d3.harvard.edu\/platform-digit\/submission\/deezers-spleeter-deconstructing-music-with-ai\/","title":{"rendered":"Deezer\u2019s Spleeter: Deconstructing music with AI"},"content":{"rendered":"

About Deezer<\/strong><\/h3>\n

Founded in August 2007, Deezer is an online music streaming service based in Paris, France. While Spotify has gained market leadership in the online music steaming space, Deezer continues to hold its own with 14 million monthly active users (MAU) in over 180 countries as of January 2019 [1]. Per Similarweb.com estimates, 35% of Deezer\u2019s users are in France, 11% in Brazil and the remainder spread out across the world.<\/p>\n

What is Spleeter?<\/strong><\/h3>\n
\"\"<\/a>
Credit: Deezer.com<\/figcaption><\/figure>\n

On November 4th<\/sup> 2019, Deezer released Spleeter, a machine learning tool for source separation. Spleeter is a project from Deezer\u2019s research division and is made available online as a Python library based on Tensorflow. Although source separation remains a relatively obscure topic, its applications in music information retrieval (MIR) has the potential to have a far reaching impact on the way we are able to produce and consume music.<\/p>\n

Source separation 101<\/strong><\/h3>\n

At its core, source separation is the separation of a desired signal from a set of mixed signals. In music, this means deconstructing a recorded musical piece into its components by isolating each instrumental layer on the track. For example, 4-stem source separation on Coldplay\u2019s hit single, \u201cYellow\u201d, would yield the following layers (aka \u201cstems\u201d) in isolation:<\/p>\n

    \n
  1. Vocals by Chris Martin, singing about the stars and such<\/li>\n
  2. Drums\/percussion by Will Champion<\/li>\n
  3. Bass guitar by Guy Berryman<\/li>\n
  4. Electric guitar by Jonny Buckland<\/li>\n<\/ol>\n

    While it may seem like a relatively straightforward process, accurate source separation is difficult to accomplish. Today, most professionally recorded music is made by recording each instrument on a separate channel, and then the final combined track is produced in a step called the \u201cmixdown\u201d. In this final step, all the individual tracks are blended together for mastering and then digitally compressed for delivery. All the sound waveforms are meshed together, in a process akin to an irreversible chemical reaction that is it impossible to undo.<\/span>Nonetheless, Spleeter has made the impossibly difficult task of source separation a lot easier using machine learning.<\/p>\n

    How does Spleeter work?<\/strong><\/h3>\n

    A common technique used source separation is time-frequency (TF) masking. Different types of sounds in the musical track correspond to varying frequencies. For instance, the lead vocals would occupy different frequency bands as compared with the drums. Using TF masking, the mixture of frequencies that make up a piece of music is filtered, allowing us to pick and choose which frequencies to keep. What remains after this process is the separated stem of the instrument that we want to isolate.<\/p>\n

    \"\"<\/a>
    Spectrogram showing separation (a) & (b) and mixture (c) of stems. Credit in footnote [2].<\/figcaption><\/figure>The tricky part of this process is being able to approximate which frequencies correspond to which instruments. Given that the audible range of frequencies for humans is 20 to 20,000 Hz, a lot of processing is needed to accurately classify the broad range of frequencies contained in a musical track. Traditionally, this step was done manually by using snippets of isolated vocals (which are hard to find) to approximate the frequencies that should be left unmasked, thereby making a \u201cminus one\u201d track commonly used for bootleg karaoke.<\/p>\n

    Today, Spleeter does the heavy lifting as it comes with pretrained models for standard 2,4 and 5-stem separation. Using Spleeter is as simple as installing the package and running the separator function on a command line interface, which then creates a .wav file for each stem. In addition, Spleeter also allows users to train custom source separation models and evaluate the models against a benchmark dataset (musDB18 available on SigSep, another open source project).<\/p>\n

    Applications and challenges<\/strong><\/h3>\n

    Deezer is in a unique position to build a generalized source separation engine because it has access to a large catalog of music that few organizations do. However, a major challenge is that Deezer is not legally or ethically able to release the vast library of stems made from its own catalog due to copyrighting. So how does Deezer capture value?<\/p>\n

    Deezer can use its musical data assets to train its own machine learning models, which can then be deployed to improve its user value proposition. In addition, Spleeter enables Deezer to perform source separation at scale. On the GPU version, one can expect separation at 100x faster than real-time [3], which allows for rapid deployment on its growing catalog of music. As such, it is conceivable that Deezer can accomplish the following:<\/p>\n

    Make novel music recommendations<\/strong>. Deezer can use the finer data granularity to cluster music by similarity for a given instrument. For example, Deezer could identify bands that have lead singers who have a similar vocal timbre as Coldplay\u2019s lead singer, Chris Martin. Coldplay fans might hence get recommendations to give Blue Merle a listen.<\/p>\n