Learn how to generate lyrics using deep (multi-layer) LSTM in this article by Matthew Lamons, founder, and CEO of Skejul — the AI platform to help people manage their activities, and Rahul Kumar, an AI scientist, deep learning practitioner, and independent researcher.
This article will show you how to create a deep LSTM model suited for the task of generating music lyrics. Here’s your goal: to build and train a model that outputs entirely new and original lyrics that is in the style of an arbitrary number of artists. You can refer to the code file found at Lyrics-ai (https://github.com/PacktPublishing/Python-Deep-Learning-Projects/tree/master/Chapter06/Lyrics-ai) for this exercise.
Data pre-processing
To build a model that can generate lyrics, you’ll need a huge amount of lyric data, which can easily be extracted from various sources. You can find the code files at https://github.com/PacktPublishing/Python-Deep-Learning-Projects/tree/master/Chapter06/Lyrics-ai. These files contain a text file called lyrics_data.txt which includes lyrics from around 10,000 songs and stored them in.
Now that you have your data, convert this raw text into the one-hot encoding version:
import numpy as np
import codecs
# Class to perform all preprocessing operations
class Preprocessing:
vocabulary = {}
binary_vocabulary = {}
char_lookup = {}
size = 0
separator = '->'
# This will take the data file and convert data into one hot encoding and dump the vocab into the file.
def generate(self, input_file_path):
input_file = codecs.open(input_file_path, 'r', 'utf_8')
index = 0
for line in input_file:
for char in line:
if char not in self.vocabulary:
self.vocabulary[char] = index
self.char_lookup[index] = char
index += 1
input_file.close()
self.set_vocabulary_size()
self.create_binary_representation()
# This method is to load the vocab into the memory
def retrieve(self, input_file_path):
input_file = codecs.open(input_file_path, 'r', 'utf_8')
buffer = ""
for line in input_file:
try:
separator_position = len(buffer) + line.index(self.separator)
buffer += line
key = buffer[:separator_position]
value = buffer[separator_position + len(self.separator):]
value = np.fromstring(value, sep=',')
self.binary_vocabulary[key] = value
self.vocabulary[key] = np.where(value == 1)[0][0]
self.char_lookup[np.where(value == 1)[0][0]] = key
buffer = ""
except ValueError:
buffer += line
input_file.close()
self.set_vocabulary_size()
# Below are some helper functions to perform pre-processing.
def create_binary_representation(self):
for key, value in self.vocabulary.iteritems():
binary = np.zeros(self.size)
binary[value] = 1
self.binary_vocabulary[key] = binary
def set_vocabulary_size(self):
self.size = len(self.vocabulary)
print "Vocabulary size: {}".format(self.size)
def get_serialized_binary_representation(self):
string = ""
np.set_printoptions(threshold='nan')
for key, value in self.binary_vocabulary.iteritems():
array_as_string = np.array2string(value, separator=',', max_line_width=self.size * self.size)
string += "{}{}{}\n".format(key.encode('utf-8'), self.separator, array_as_string[1:len(array_as_string) - 1])
return string
The overall objective of the pre-processing module is to convert the raw text data into one-hot encoding, as shown in the following diagram:
This figure represents the data preprocessing part. The raw lyrics data is used to build the vocabulary mapping which is further been transformed into the on-hot encoding. After the successful execution of the pre-processing module, a binary file will be dumped as {dataset_filename}.vocab. This vocab file is one of the mandatory files that need to be fed into the model during the training process, along with the dataset.
Defining the model
This article will take an approach from the Keras model and use TensorFlow to write each layer from scratch. TensorFlow gives you a more fine-tuned control over your model’s architecture. For this model, use the code in the following block to create two placeholders that will store the input and output values:
import tensorflow as tf
import pickle
from tensorflow.contrib import rnn
def build(self, input_number, sequence_length, layers_number, units_number, output_number):
self.x = tf.placeholder("float", [None, sequence_length, input_number])
self.y = tf.placeholder("float", [None, output_number])
self.sequence_length = sequence_length
Next, store the weights and bias in the variables that you’ve created:
self.weights = {
'out': tf.Variable(tf.random_normal([units_number, output_number]))
}
self.biases = {
'out': tf.Variable(tf.random_normal([output_number]))
}
x = tf.transpose(self.x, [1, 0, 2])
x = tf.reshape(x, [-1, input_number])
x = tf.split(x, sequence_length, 0)
You can build this model using multiple LSTM layers, with the basic LSTM cells assigning each layer with the specified number of cells, as shown in the following diagram:
Tensorboard visualization of the LSTM architecture
The following is the code for this:
lstm_layers = []
for i in range(0, layers_number):
lstm_layer = rnn.BasicLSTMCell(units_number)
lstm_layers.append(lstm_layer)
deep_lstm = rnn.MultiRNNCell(lstm_layers)
self.outputs, states = rnn.static_rnn(deep_lstm, x, dtype=tf.float32)
print "Build model with input_number: {}, sequence_length: {}, layers_number: {}, " \
"units_number: {}, output_number: {}".format(input_number, sequence_length, layers_number,
units_number, output_number)
# This method is using to dump the model configurations
self.save(input_number, sequence_length, layers_number, units_number, output_number)
Training the deep TensorFlow-based LSTM model
Now that you have the mandatory inputs, that is, the dataset file path, the vocab file path, and the model name initiate the training process. Define all hyperparameters for the model:
import os
import argparse
from modules.Model import *
from modules.Batch import *
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--training_file', type=str, required=True)
parser.add_argument('--vocabulary_file', type=str, required=True)
parser.add_argument('--model_name', type=str, required=True)
parser.add_argument('--epoch', type=int, default=200)
parser.add_argument('--batch_size', type=int, default=50)
parser.add_argument('--sequence_length', type=int, default=50)
parser.add_argument('--log_frequency', type=int, default=100)
parser.add_argument('--learning_rate', type=int, default=0.002)
parser.add_argument('--units_number', type=int, default=128)
parser.add_argument('--layers_number', type=int, default=2)
args = parser.parse_args()
Since this is a batch training the model, divide the dataset into batches of a defined batch_size using the Batch module:
batch = Batch(training_file, vocabulary_file, batch_size, sequence_length)
Each batch will return two arrays. One will be the input vector of the input sequence, which will have a shape of [batch_size, sequence_length, vocab_size], and the other array will hold the label vector, which will have a shape of [batch_size, vocab_size].
Now, initialize your model and create the optimizer function. In this model, you used the Adam Optimizer. Then, you’ll train your model and perform the optimization over each batch:
# Building model instance and classifier
model = Model(model_name)
model.build(input_number, sequence_length, layers_number, units_number, classes_number)
classifier = model.get_classifier()
**# Building cost functions**
cost = tf.reduce_mean(tf.square(classifier - model.y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
# Computing the accuracy metrics
expected_prediction = tf.equal(tf.argmax(classifier, 1), tf.argmax(model.y, 1))
accuracy = tf.reduce_mean(tf.cast(expected_prediction, tf.float32))
# Preparing logs for Tensorboard
loss_summary = tf.summary.scalar("loss", cost)
acc_summary = tf.summary.scalar("accuracy", accuracy)
train_summary_op = tf.summary.merge_all()
out_dir = "{}/{}".format(model_name, model_name)
train_summary_dir = os.path.join(out_dir, "summaries")
##
# Initializing the session and executing the training
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
iteration = 0
while batch.dataset_full_passes < epoch:
iteration += 1
batch_x, batch_y = batch.get_next_batch()
batch_x = batch_x.reshape((batch_size, sequence_length, input_number))
sess.run(optimizer, feed_dict={model.x: batch_x, model.y: batch_y})
if iteration % log_frequency == 0:
acc = sess.run(accuracy, feed_dict={model.x: batch_x, model.y: batch_y})
loss = sess.run(cost, feed_dict={model.x: batch_x, model.y: batch_y})
print("Iteration {}, batch loss: {:.6f}, training accuracy: {:.5f}".format(iteration * batch_size,
loss, acc))
batch.clean()
Once the model completes its training, the checkpoints are stored. You can use them later for inferencing. The following is a graph of the accuracy and the loss that occurred during the training process:
The accuracy (top) and the loss (bottom) plot with respect to the time
You can see that accuracy getting increased and loss getting reduced over the period of time.
Inference
Now that the model is ready, you can use it to make predictions. Start by defining all parameters. While building inference, you need to provide some seed text, just like you did in the previous model. Along with this, you should also provide the path of the vocab file and the output file in which you will store the generated lyrics. Also, provide the length of the text that you need to generate:
import argparse
import codecs
from modules.Model import *
from modules.Preprocessing import *
from collections import deque
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--model_name', type=str, required=True)
parser.add_argument('--vocabulary_file', type=str, required=True)
parser.add_argument('--output_file', type=str, required=True)
parser.add_argument('--seed', type=str, default="Yeah, oho ")
parser.add_argument('--sample_length', type=int, default=1500)
parser.add_argument('--log_frequency', type=int, default=100)
Next, load the model by providing the name of model that you used in the training step in the preceding code, and restore the vocabulary from the file:
model = Model(model_name)
model.restore()
classifier = model.get_classifier()
vocabulary = Preprocessing()
vocabulary.retrieve(vocabulary_file)
Use the stack methods to store the generated characters, append the stack, and then use the same stack to feed it into the model in an interactive fashion:
# Preparing the raw input data
for char in seed:
if char not in vocabulary.vocabulary:
print char,"is not in vocabulary file"
char = u' '
stack.append(char)
sample_file.write(char)
# Restoring the models and making inferences
with tf.Session() as sess:
tf.global_variables_initializer().run()
saver = tf.train.Saver(tf.global_variables())
ckpt = tf.train.get_checkpoint_state(model_name)
if ckpt and ckpt.model_checkpoint_path:
saver.restore(sess, ckpt.model_checkpoint_path)
for i in range(0, sample_length):
vector = []
for char in stack:
vector.append(vocabulary.binary_vocabulary[char])
vector = np.array([vector])
prediction = sess.run(classifier, feed_dict={model.x: vector})
predicted_char = vocabulary.char_lookup[np.argmax(prediction)]
stack.popleft()
stack.append(predicted_char)
sample_file.write(predicted_char)
if i % log_frequency == 0:
print "Progress: {}%".format((i * 100) / sample_length)
sample_file.close()
print "Sample saved in {}".format(output_file)
Output
After successful execution, you’ll get your own freshly brewed, AI-generated lyrics reviewed and published. The following is one sample of such lyrics. Some spellings have been modified so that the sentences make sense:
Yeah, oho once upon a time, on ir intasd
I got monk that wear your good
So heard me down in my clipp
Cure me out brick
Coway got baby, I wanna sheart in faic
I could sink awlrook and heart your all feeling in the firing of to the still hild, gavelly mind, have before you, their lead
Oh, oh shor,s sheld be you und make
Oh, fseh where sufl gone for the runtome
Weaaabe the ligavus I feed themust of hear
Here, you can see that the model has learned in the way it has generated the paragraphs and sentences with appropriate spacing. It still lacks perfection and also doesn’t make sense.
Seeing signs of success — The first task is to create a model that can learn, and then the second one is used to improve on that model. This can be obtained by training the model with a larger training dataset and longer training durations.
If you found this article interesting, you can explore Python Deep Learning Projects to master deep learning and neural network architectures using Python and Keras. Python Deep Learning Projects imparts all the knowledge needed to implement complex deep learning projects in the field of computational linguistics and computer vision.