From b40e4a1e83245f0c2c3ad3733d538b0f73d69deb Mon Sep 17 00:00:00 2001 From: Ted Moore Date: Tue, 7 Jun 2022 12:59:09 +0100 Subject: [PATCH 1/5] NMFMorph RST and SC examples --- doc/NMFMorph.rst | 6 ++---- example-code/sc/NMFMorph.scd | 39 +++++++++++++----------------------- 2 files changed, 16 insertions(+), 29 deletions(-) diff --git a/doc/NMFMorph.rst b/doc/NMFMorph.rst index 648bd1c..1921b56 100644 --- a/doc/NMFMorph.rst +++ b/doc/NMFMorph.rst @@ -4,11 +4,10 @@ :sc-related: Classes/FluidAudioTransport, Classes/FluidBufNMFCross :see-also: :description: - Perform cross-synthesis using Nonnegative Matrix Factorization (NMF) and Optimal Transport (OT). NMF analyses of ``source`` and ``target`` sounds decompose their material in to a selectable number of components, which are in turn represented by their *bases* (spectrum) and *activations* (temporal pattern of each component). - - ``FluidNMFMorph`` provides the ability to interpolate between ``source`` and ``target`` spectra using a technique called Optimal Transport, that provides richer results than a simple linear interpolation between spectral shapes. The resulting sound is built up using a buffer of temporal activations, then resynthesised using a phase estimate. + Perform cross-synthesis using Nonnegative Matrix Factorization (NMF) and Optimal Transport (OT). NMF analyses of ``source`` and ``target`` sounds decompose their material in to a selectable number of components, which are in turn represented by their *bases* (spectrum) and *activations* (temporal pattern of each component). + ``NMFMorph`` provides the ability to interpolate between ``source`` and ``target`` spectra bases using a technique called Optimal Transport, that provides richer results than a simple linear interpolation between spectral shapes. The resulting sound is built up using a buffer of temporal activations, then resynthesised using a phase estimate. :control source: @@ -45,4 +44,3 @@ :control maxFFTSize: The maximum FFT size to allocate memory for - diff --git a/example-code/sc/NMFMorph.scd b/example-code/sc/NMFMorph.scd index 27cb273..4630b3c 100644 --- a/example-code/sc/NMFMorph.scd +++ b/example-code/sc/NMFMorph.scd @@ -1,5 +1,5 @@ -code::FluidNMFMorph:: relies on preexisting NMF analyses to generate variations between sounds. We can produce these using link::Classes/FluidBufNMF:: +code::FluidNMFMorph:: relies on preexisting NMF analyses to generate variations between sounds. Produce these using link::Classes/FluidBufNMF:: code:: //read some audio @@ -7,38 +7,27 @@ code:: ~src1 = Buffer.readChannel(s,FluidFilesPath("Nicol-LoopE-M.wav"),channels:[0]); //some drums ~src2 = Buffer.readChannel(s,FluidFilesPath("Tremblay-SA-UprightPianoPedalWide.wav"),channels:[0]);//some piano -~src1Bases = Buffer.new; -~src2Bases = Buffer.new; -~src1Activations = Buffer.new; -~src2Activations = Buffer.new; +~src1Bases = Buffer(s); +~src2Bases = Buffer(s); +~src1Activations = Buffer(s); +~src2Activations = Buffer(s); ) -//nmf analyses + +//nmf analyses (wait for this to complete!) ( -FluidBufNMF.process(s,~src1,bases:~src1Bases,activations:~src1Activations,components:5, action:{"Analysed Source 1".postln}); -FluidBufNMF.process(s,~src2,bases:~src2Bases,activations:~src2Activations, components:5, action:{"Analysed Source 2".postln}); +FluidBufNMF.processBlocking(s,~src1,bases:~src1Bases,activations:~src1Activations,components:5, action:{"Analysed Source 1".postln}); +FluidBufNMF.processBlocking(s,~src2,bases:~src2Bases,activations:~src2Activations, components:5, action:{"Analysed Source 2".postln}); ) ( -~morph = { |source, target, activations, interp, autoassign| - FluidNMFMorph.ar(source,target,activations,autoassign,interp) * 80 -}; +~synth = { |source, target, activations, autoassign| + FluidNMFMorph.ar(source,target,activations,autoassign,MouseX.kr).dup * 80 +}.play(s,args:[\source,~src1Bases,\target,~src2Bases,\activations,~src1Activations,\autoassign,1]); ) -~synth = ~morph.play(s,args:[\source,~src1Bases,\target,~src2Bases,\activations,~src2Activations,\interp,0.5,\autoassign,1]); - -//Play with different interpolation values -~synth.set(\interp,0.0); -~synth.set(\interp,1.0); -:: -warning::The following parameters current require one to change the 'autoassign' control to update the process:: -code:: -//Change the actvations -~synth.set(\activations, ~src1Activations, \autoassign,0); -~synth.set(\autoassign,1); +// Change the actvations +// when changing the activations, one needs to change the 'autoassign' control to update the process ~synth.set(\activations, ~src2Activations, \autoassign,0); ~synth.set(\autoassign,1); -//Swap source and target -~synth.set(\source,~src2Bases,\target,~src1Bases, \autoassign,0); -~synth.set(\autoassign,1); :: From c90fee81b43c00cf02fb1886dadaa2b62de17de1 Mon Sep 17 00:00:00 2001 From: Ted Moore Date: Tue, 7 Jun 2022 13:56:23 +0100 Subject: [PATCH 2/5] NMFMorph --- doc/NMFMorph.rst | 4 ++-- example-code/sc/NMFMorph.scd | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/NMFMorph.rst b/doc/NMFMorph.rst index 1921b56..b27bb1b 100644 --- a/doc/NMFMorph.rst +++ b/doc/NMFMorph.rst @@ -11,11 +11,11 @@ :control source: - A |buffer| with the spectral bases for the source sound. + A |buffer| with the spectral bases for the source sound (must be the same number of spectral bases as ``target``). :control target: - A |buffer| with the spectral bases for the target sound. + A |buffer| with the spectral bases for the target sound (must be the same number of spectral bases as ``source``). :control activations: diff --git a/example-code/sc/NMFMorph.scd b/example-code/sc/NMFMorph.scd index 4630b3c..5ef7152 100644 --- a/example-code/sc/NMFMorph.scd +++ b/example-code/sc/NMFMorph.scd @@ -16,7 +16,7 @@ code:: //nmf analyses (wait for this to complete!) ( FluidBufNMF.processBlocking(s,~src1,bases:~src1Bases,activations:~src1Activations,components:5, action:{"Analysed Source 1".postln}); -FluidBufNMF.processBlocking(s,~src2,bases:~src2Bases,activations:~src2Activations, components:5, action:{"Analysed Source 2".postln}); +FluidBufNMF.processBlocking(s,~src2,bases:~src2Bases,activations:~src2Activations, components:4, action:{"Analysed Source 2".postln}); ) ( From 49bf5a9fc2c4833173c871624728bca1282d3cfe Mon Sep 17 00:00:00 2001 From: Ted Moore Date: Tue, 7 Jun 2022 14:14:26 +0100 Subject: [PATCH 3/5] NMFMorph (and species transformer[0]) --- doc/NMFMorph.rst | 2 +- example-code/sc/NMFMorph.scd | 4 ++-- flucoma/doc/templates/schelp_transformer[0].schelp | 8 ++++++++ 3 files changed, 11 insertions(+), 3 deletions(-) create mode 100644 flucoma/doc/templates/schelp_transformer[0].schelp diff --git a/doc/NMFMorph.rst b/doc/NMFMorph.rst index b27bb1b..5098994 100644 --- a/doc/NMFMorph.rst +++ b/doc/NMFMorph.rst @@ -1,5 +1,5 @@ :digest: Morph between sounds -:species: transformer +:species: transformer[0] :sc-categories: FluidCorpusManipulation :sc-related: Classes/FluidAudioTransport, Classes/FluidBufNMFCross :see-also: diff --git a/example-code/sc/NMFMorph.scd b/example-code/sc/NMFMorph.scd index 5ef7152..454d4d4 100644 --- a/example-code/sc/NMFMorph.scd +++ b/example-code/sc/NMFMorph.scd @@ -13,10 +13,10 @@ code:: ~src2Activations = Buffer(s); ) -//nmf analyses (wait for this to complete!) +//nmf analyses -- must have the same number of components (wait for this to complete!) ( FluidBufNMF.processBlocking(s,~src1,bases:~src1Bases,activations:~src1Activations,components:5, action:{"Analysed Source 1".postln}); -FluidBufNMF.processBlocking(s,~src2,bases:~src2Bases,activations:~src2Activations, components:4, action:{"Analysed Source 2".postln}); +FluidBufNMF.processBlocking(s,~src2,bases:~src2Bases,activations:~src2Activations, components:5, action:{"Analysed Source 2".postln}); ) ( diff --git a/flucoma/doc/templates/schelp_transformer[0].schelp b/flucoma/doc/templates/schelp_transformer[0].schelp new file mode 100644 index 0000000..d49bfb9 --- /dev/null +++ b/flucoma/doc/templates/schelp_transformer[0].schelp @@ -0,0 +1,8 @@ +{% extends "schelp_base.schelp" %} +{% block classmethods %} +CLASSMETHODS:: + +METHOD:: ar + +{% include "schelp_controls.schelp" %} +{% endblock %} From ff4b34117259573ca529d32b8c700c00c0fe39e9 Mon Sep 17 00:00:00 2001 From: James Bradbury Date: Tue, 7 Jun 2022 15:31:09 +0100 Subject: [PATCH 4/5] separate discussion and description --- doc/NMFMorph.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/NMFMorph.rst b/doc/NMFMorph.rst index 5098994..4b7679d 100644 --- a/doc/NMFMorph.rst +++ b/doc/NMFMorph.rst @@ -2,12 +2,12 @@ :species: transformer[0] :sc-categories: FluidCorpusManipulation :sc-related: Classes/FluidAudioTransport, Classes/FluidBufNMFCross -:see-also: +:see-also: BufNMF, NMFCross, AudioTransport, BufAudioTransport :description: + Perform cross-synthesis using Nonnegative Matrix Factorization (NMF) and Optimal Transport (OT). - Perform cross-synthesis using Nonnegative Matrix Factorization (NMF) and Optimal Transport (OT). NMF analyses of ``source`` and ``target`` sounds decompose their material in to a selectable number of components, which are in turn represented by their *bases* (spectrum) and *activations* (temporal pattern of each component). - - ``NMFMorph`` provides the ability to interpolate between ``source`` and ``target`` spectra bases using a technique called Optimal Transport, that provides richer results than a simple linear interpolation between spectral shapes. The resulting sound is built up using a buffer of temporal activations, then resynthesised using a phase estimate. +:discussion: + The algorithm uses NMF analyses of the ``source`` and ``target`` sounds. It decomposes their material in to a selectable number of components, which are in turn represented by their *bases* (spectrum) and *activations* (temporal pattern of each component). ``NMFMorph`` provides the ability to interpolate between ``source`` and ``target`` bases using a technique called Optimal Transport, that provides richer results than a simple linear interpolation between spectral shapes. The resulting sound is built up using a buffer of temporal activations, then resynthesised using a phase estimate. :control source: From 4218848f7231b490977faca7ce42d0c14a42b237 Mon Sep 17 00:00:00 2001 From: Ted Moore Date: Wed, 8 Jun 2022 12:07:55 +0100 Subject: [PATCH 5/5] added comment about why autoassign is set twice --- example-code/sc/NMFMorph.scd | 2 ++ 1 file changed, 2 insertions(+) diff --git a/example-code/sc/NMFMorph.scd b/example-code/sc/NMFMorph.scd index 454d4d4..371f6f5 100644 --- a/example-code/sc/NMFMorph.scd +++ b/example-code/sc/NMFMorph.scd @@ -28,6 +28,8 @@ FluidBufNMF.processBlocking(s,~src2,bases:~src2Bases,activations:~src2Activation // Change the actvations // when changing the activations, one needs to change the 'autoassign' control to update the process ~synth.set(\activations, ~src2Activations, \autoassign,0); + +// change autoassign back to 1 to hear the difference ~synth.set(\autoassign,1); ::