Sitelet https://reference.wolfram.com/language/ref/netencoder/AudioMelSpectrogram.html

WOLFRAM

Wolfram Language & System Documentation Center

"AudioMelSpectrogram" (Net Encoder)

NetEncoder["AudioMelSpectrogram"]

represents an encoder that converts an audio file or object into its mel-frequency spectrogram.

NetEncoder[{"AudioMelSpectrogram","param"->val,…}]

represents an encoder with specific parameters for preprocessing and feature computation.

Details

  • The "AudioMelSpectrogram" encoder computes the magnitude spectrogram and applies to it a filter bank whose filter centers are linearly spaced on the mel-frequency scale. This is done to mimic the human perception of pitch, which is nonlinear. The number of filters is always less than the number of spectrogram bins, so the dimensionality of the feature is reduced.
  • NetEncoder[…][input] applies the encoder to an input to produce an output.
  • NetEncoder[…][{input1,input2,…}] applies the encoder to a list of inputs to produce a list of outputs.
  • The input to the encoder can be an Audio object or a File[…] expression.
  • The output is computed by filtering the spectrogram with nf bandpass filters whose center frequencies are linearly spaced on the mel scale.
  • The output of the encoder is a rank-2 tensor of dimensions {n,nf}, where n is the number of partitions after the preprocessing is applied and nf is the number of filters used for the computation.
  • An encoder can be attached to an input port of a net by specifying "port"->NetEncoder[…] when constructing the net.
    • Parameters
  • The following general parameters are supported:
  • "Augmentation"Noneaugmentation to be applied
    "Normalization"Nonewhether to apply normalization
    "SampleRate"16000target sample rate
    "TargetLength"Alltarget output length
  • Additional partitioning parameters:
  • "WindowSize"Automaticlength of the partitions
    "Offset"Automaticoffset of the partitions
    "WindowFunction"Automaticwindow to be applied to the partitions
  • Mel-spectrogram parameters:
  • "MaximumFrequency"Automaticmaximum frequency of the mel filters
    "MinimumFrequency"Automaticminimum frequency of the mel filters
    "NumberOfFilters"40number of the mel filters
  • The following settings and suboptions can be specified for each encoder parameter.
  • "Normalization" can take the following settings:
  • Noneno normalization
    "Max"absolute maximum value normalized to 1
    {"Max",val}absolute maximum value normalized to val
    {"RMS",val}RMS of input audio signal normalized to val
  • "TargetLength" can take the following settings:
  • Allsame as input signal
    durthe duration dur specified as a time quantity
    nthe first n partitions
  • If the specified "TargetLength" does not match the length of the input signal, padding or trimming are applied accordingly.
  • "Augmentation" can be specified as a list of rules with the following keys:
  • "Convolution"Noneconvolves an impulse response to the input
    "Noise"Noneadds noise to the input
    "TimeShift"Noneshifts the input by a specified amount
    "Volume"Nonemultiplies the input with a constant
    "VTLP"Noneapplies vocal tract length perturbation to the input
  • Any augmentation parameter that accepts a numeric value can also be specified as a list of two numbers or a univariate distribution. In the first case, the value will be randomized according to a uniform distribution between the given bounds. In the second, the user-provided distribution will be used.
  • Possible values for "Convolution" include:
  • Noneno augmentation
    signalFile or Audio object to be convolved with input
    {mix,signal}signal to be convolved with input and mix parameter
  • Possible values for "Noise" include:
  • Noneno augmentation
    ampwhite noise with amplitude amp
    noiseFile or Audio object containing the noise signal to be added
    {amp,noise}noise signal and its with the specified amplitude
  • Use "TimeShift"->t to shift the input by t seconds, padding or trimming if necessary. Use Scaled[s] to shift the input by s×dur seconds, where dur is the duration of the input signal. Use {t1,t2} or Scaled[{ts1,t2}] to randomize the shift between the specified times.
  • Use "Volume"->val to specify a constant multiplier.
  • Vocal tract length perturbation (VTLP) multiplies the center values of the filter frequencies in the mel-spectrogram by a fixed amount. Use "VTLP"val to specify the amount of the perturbation.
  • With the parameter "WindowSize"->Automatic, a partition length of 25 milliseconds is used. Use "WindowSize"->dur to select a partition length of duration dur. Use "WindowSize"->n to select a partition length of n samples.
  • With the parameter "Offset"->Automatic, a partition offset of 8.33 milliseconds is used. Use "Offset"->dur to select a partition offset of duration dur. Use "Offset"->n to select a partition offset of n samples.
  • Parameter "WindowFunction" applies a window to each partition. Possible settings are:
  • Noneno windowing applied to the input audio
    Automatic
    functhe window is computed using the function func
    listthe sampled window list is explicitly specified
  • With the parameter "MinimumFrequency"->Automatic, a frequency is computed as Ceiling[sr/ws], where sr is the sample rate "SampleRate" and ws is the partition length "WindowSize". Use "MinimumFrequency"f to set the minimum frequency for the filters to f.
  • With the parameter "MaximumFrequency"->Automatic, a frequency is computed as Round[Min[8000,sr/2]], where sr is the sample rate "SampleRate". Use "MaximumFrequency"f to set the maximum frequency for the filters to f.
  • With the parameter "NumberOfFilters"->n, n filters will be used in the computation of the mel-spectrogram.

Examples

open all close all

Basic Examples  (1)

Create a mel-spectrogram NetEncoder:

Wolfram Language code: enc = NetEncoder["AudioMelSpectrogram"]

Create an Audio object:

Wolfram Language code: audio = Import["ExampleData/rule30.wav"]

Apply the encoder to the Audio object:

Wolfram Language code: enc[audio]//Short

Plot the result:

Wolfram Language code: MatrixPlot[Log[Transpose[%]], IconizedObject[«opts»]]

Scope  (3)

NetEncoder["AudioMelSpectrogram"] can encode either File or Audio objects. Create a mel-spectrogram encoder:

Wolfram Language code: enc = NetEncoder["AudioMelSpectrogram"]

Apply the encoder to a File object:

Wolfram Language code: enc[File[FindFile["ExampleData/rule30.wav"]]]//Short

Apply the encoder to an in-core Audio object:

Wolfram Language code: a1 = Import["ExampleData/rule30.wav", "Audio"]
Wolfram Language code: out = enc[a1]; Dimensions[out]

Apply the encoder to an out-of-core Audio object:

Wolfram Language code: a2 = Audio["ExampleData/rule30.wav"]
Wolfram Language code: out = enc[a2]; Dimensions[out]

Create a list of Audio objects:

Wolfram Language code: audios = {\!\(\*AudioBox["![Embedded Audio Player](audio://content-3w7j6)"]\), \!\(\*AudioBox["![Embedded Audio Player](audio://content-8h61g)"]\)};

NetEncoder["AudioMelSpectrogram"] maps across a batch of inputs:

Wolfram Language code: NetEncoder["AudioMelSpectrogram"][audios]

Create a mel-spectrogram NetEncoder:

Wolfram Language code: enc = NetEncoder["AudioMelSpectrogram"]

Attach the encoder to the input of a net:

Wolfram Language code: net = NetChain[{BasicRecurrentLayer[10], SequenceLastLayer[]}, "Input" -> enc]

Apply the net to an Audio object:

Wolfram Language code: NetInitialize[net][Import["ExampleData/rule30.wav"]]

Parameters  (9)

"Normalization"  (1)

Create an Audio object:

Wolfram Language code: audio = Audio[RandomReal[.1, 800], SampleRate -> 16000]

Use an encoder with "Normalization"->None to avoid any normalization:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram", "Normalization" -> None}][audio];

Since the normalization is applied to the signal before the spectrogram is computed, there are no guarantees on the bounds of the result:

Wolfram Language code: MinMax[%]

Use an encoder with "Normalization"->Automatic to normalize the maximum absolute value of the waveform samples to 1.:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram", "Normalization" -> True}][audio];

Find the minimum and maximum value of the result:

Wolfram Language code: MinMax[%]

"SampleRate"  (2)

Create an Audio object:

Wolfram Language code: audio = Audio[RandomReal[.1, 800], SampleRate -> 16000]

Using an encoder with "SampleRate"8000 resamples the signal to 8000Hz before performing the short-time Fourier transform:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram", SampleRate -> 8000}][audio]//Short

The "SampleRate" parameter affects the computation of the default window size:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram"}]

An encoder with a lower sample rate than the original audio will result in a shorter window length:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram", "SampleRate" -> 6000}]

An encoder with a higher sample rate than the original audio will result in a longer window length:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram", "SampleRate" -> 32000}]

"TargetLength"  (1)

Create an Audio object:

Wolfram Language code: audio = Audio[RandomReal[.1, 800], SampleRate -> 16000]

Using an encoder with "TargetLength"All returns the mel-spectrogram for all the data:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram", "TargetLength" -> All}][audio]//Dimensions

Using an encoder with "TargetLength"10 zero-pads the output to be of length 10:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram", "TargetLength" -> 10}][audio]//Dimensions

Using an encoder with "TargetLength"2 takes only the first two partitions:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram", "TargetLength" -> 2}][audio]//Dimensions

"WindowSize"  (1)

The partition length is automatically computed to be 25ms:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram"}]

Using an encoder with "WindowSize"600 returns the mel-spectrogram using partitions of 600 samples:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram", "WindowSize" -> 600}]

"Offset"  (1)

Create an Audio object:

Wolfram Language code: audio = Audio[RandomReal[.1, 800], SampleRate -> 16000]

The partition offset is automatically computed to be 1/3 of the partition length:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram"}][audio]//Dimensions

Using an encoder with "Offset"10 returns the mel-spectrogram computed using partitions with an offset of 10 samples:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram", "Offset" -> 10}][audio]//Dimensions

"MinimumFrequency"  (1)

Create an Audio object:

Wolfram Language code: audio = Import["ExampleData/rule30.wav"]

The minimum frequency is automatically computed to be Ceiling[sr/ws], where sr is the sample rate "SampleRate" and ws is the partition length "WindowSize":

Wolfram Language code: MatrixPlot[NetEncoder[{"AudioMelSpectrogram"}][audio], AspectRatio -> 1]

Using an encoder with "MinimumFrequency"2000 returns the mel-spectrogram computed using filters whose minimum frequency is 2000Hz:

Wolfram Language code: MatrixPlot[NetEncoder[{"AudioMelSpectrogram", "MinimumFrequency" -> 2000}][audio], AspectRatio -> 1]

"MaximumFrequency"  (1)

Create an Audio object:

Wolfram Language code: audio = Import["ExampleData/rule30.wav"]

The maximum frequency is automatically computed to be Round[Min[8000,sr/2]], where sr is the sample rate "SampleRate":

Wolfram Language code: MatrixPlot[NetEncoder[{"AudioMelSpectrogram"}][audio], AspectRatio -> 1]

Using an encoder with "MaximumFrequency"2000 returns the mel-spectrogram computed using filters whose maximum frequency is 2000Hz:

Wolfram Language code: MatrixPlot[NetEncoder[{"AudioMelSpectrogram", "MaximumFrequency" -> 2000}][audio], AspectRatio -> 1]

"NumberOfFilters"  (1)

Create an Audio object:

Wolfram Language code: audio = Audio[RandomReal[.1, 800], SampleRate -> 16000]

By default, 40 filters are used for the computation of the mel-spectrogram:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram"}][audio]//Dimensions

Using an encoder with "NumberOfFilters"128 returns the mel-spectrogram computed using 128 filters:

Wolfram Language code: NetEncoder[{"AudioMelSpectrogram", "NumberOfFilters" -> 128}][audio]//Dimensions

Properties & Relations  (1)

Create an Audio object:

Wolfram Language code: audio = Audio[RandomReal[.1, 800], SampleRate -> 16000]

Create a mel-spectrogram NetEncoder:

Wolfram Language code: offset = 100; enc = NetEncoder[{"AudioMelSpectrogram", "Offset" -> offset}]

The length of the result can be computed as Ceiling[length/offset], where length is the length of the signal after resampling and offset is the "Offset" parameter of the encoder:

Wolfram Language code: Ceiling[QuantityMagnitude[AudioLength[audio]] / offset] == Length@enc[audio]