昇思25天学习打卡营第19天|热门LLM及其他AI应用-基于MindNLP+MusicGen生成自己的个性化音乐

模型准备

from mindnlp.transformers import MusicgenForConditionalGeneration

model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
#可选
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-medium")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-big")

无提示音频生成与播放

%%time
unconditional_inputs = model.get_unconditional_inputs(num_samples=1)

audio_values = model.generate(**unconditional_inputs, do_sample=True, max_new_tokens=256)
import scipy
sampling_rate = model.config.audio_encoder.sampling_rate
scipy.io.wavfile.write("musicgen_out.wav", rate=sampling_rate, data=audio_values[0, 0].asnumpy())
#直接播放
from IPython.display import Audio
# 要收听生成的音频样本，可以使用 Audio 在 notebook 进行播放
Audio(audio_values[0].asnumpy(), rate=sampling_rate)

1. model.get_unconditional_inputs:生成输入样本，数目为1个

2. model.generate：根据输入样本生成输出，do_sample：采取采样模式， max_new_tokens：token最大数目，类似音频的长度

3. model.config.audio_encoder.sampling_rate：获取模型采样模式下的采样率

4. scipy.io.wavfile.write：将生成的音频数据保存为音频文件

   • audio_values[0, 0] ：提取了第一维和第二维的第一个元素（假设音频数据是多维的），根据模型输出是 a Torch tensor of shape (batch_size, num_channels, sequence_length)。因此，这样是代表的一个音频的第一个通道的输出

   • asnumpy()：将张量数据转换为 NumPy 数组，适应音频数据的要求

文本提示生成

from mindnlp.transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("facebook/musicgen-small")

inputs = processor(
    text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],
    padding=True,
    return_tensors="ms",
)

audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)

音频提示生成与播放

%%time
from datasets import load_dataset

processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
dataset = load_dataset("sanchit-gandhi/gtzan", split="train", streaming=True)
sample = next(iter(dataset))["audio"]

# take the first half of the audio sample
sample["array"] = sample["array"][: len(sample["array"]) // 2]

inputs = processor(
    audio=sample["array"],
    sampling_rate=sample["sampling_rate"],
    text=["80s blues track with groovy saxophone"],
    padding=True,
    return_tensors="ms",
)

audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)
scipy.io.wavfile.write("musicgen_out_audio.wav", rate=sampling_rate, data=audio_values[0, 0].asnumpy())
from IPython.display import Audio
# 要收听生成的音频样本，可以使用 Audio 在 notebook 进行播放
Audio(audio_values[0].asnumpy(), rate=sampling_rate)

1. AutoProcessor.from_pretrained：对应的处理器
2. next(iter(dataset))[“audio”]：利用迭代器在数据集中取样
3. processor
   • audio：音频提示
   • text：文本描述，指明生成音乐的风格
   • padding：启用填充，确保所有数据都是一样的长度
   • return_tensors：指定返回的张量类型
4. model.generate
   • guidance_scale：指导尺度参数，越高会使结果越逼近描述

批量音频提示生成

sample = next(iter(dataset))["audio"]

# take the first quater of the audio sample
sample_1 = sample["array"][: len(sample["array"]) // 4]

# take the first half of the audio sample
sample_2 = sample["array"][: len(sample["array"]) // 2]

inputs = processor(
    audio=[sample_1, sample_2],
    sampling_rate=sample["sampling_rate"],
    text=["80s blues track with groovy saxophone", "90s rock song with loud guitars and heavy drums"],
    padding=True,
    return_tensors="ms",
)

audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=256)

# post-process to remove padding from the batched audio
audio_values = processor.batch_decode(audio_values, padding_mask=inputs.padding_mask)
Audio(audio_values[0], rate=sampling_rate)