Learn how to summarize text in this article by Rajdeep Dua who currently leads the developer relations team at Salesforce India, and Manpreet Singh Ghotra who is currently working at Salesforce developing a machine learning platform/APIs.
Text summarization is a method in natural language processing (NLP) for generating a short and precise summary of a reference document. Producing a summary of a large document manually is a very difficult task. Summarization of a text using machine learning techniques is still an active research topic. Before proceeding to discuss text summarization and how we do it, here is a definition of summary.
A summary is a text output that is generated from one or more texts that conveys relevant information from the original text in a shorter form. The goal of automatic text summarization is to transform the source text into a shorter version using semantics.
Lately, various approaches have been developed for automated text summarization using NLP techniques, and they have been implemented widely in various domains. Some examples include search engines creating summaries for use in previews of documents and news websites producing consolidated descriptions of news topics, usually as headlines, to help users browse.
To summarize text effectively, deep learning models need to be able to understand documents and discern and distill the important information. These methods are highly challenging and complex, particularly as the length of a document increases.
Text summarization for reviews
This article will show you how to work on the problem of text summarization to create relevant summaries for product reviews about fine food sold on the world’s largest e-commerce platform, Amazon. Reviews include product and user information, ratings, and a plain text review. It also includes reviews from all other Amazon categories. Develop a basic character-level sequence-to-sequence (seq2seq) model by defining the encoder-decoder recurrent neural network (RNN) architecture.
The dataset used in this article can be found at https://www.kaggle.com/snap/amazon-fine-food-reviews/. Your dataset will include the following:
· 568,454 reviews
· 256,059 users
· 74,258 products
How to do it…
You’ll develop a modeling pipeline and encoder-decoder architecture that tries to create relevant summaries for a given set of reviews. The modeling pipelines use RNN models written using the Keras functional API. The pipelines also use various data manipulation libraries.
The encoder-decoder architecture is used as a way of building RNNs for sequence predictions. It involves two major components: an encoder and a decoder. The encoder reads the complete input sequence and encodes it into an internal representation, usually a fixed-length vector, described as the context vector. The decoder, on the other hand, reads the encoded input sequence from the encoder and generates the output sequence. Various types of encoders can be used — more commonly, bidirectional RNNs, such as LSTMs, are used.
Data processing
It is crucial that you serve the right data as input to the neural architecture for training and validation. Make sure that data is on a useful scale and format, and that meaningful features are included. This will lead to better and more consistent results.
Employ the following workflow for data preprocessing:
1. Load the dataset using pandas
2. Split the dataset into input and output variables for machine learning
3. Apply a preprocessing transform to the input variables
4. Summarize the data to show the change
Now get started step by step:
1. Get started by importing important packages and your dataset. Use the pandas library to load data and review the shape of your dataset—it includes 10 features and 5 million data points:
import pandas as pd
import re
from nltk.corpus import stopwords
from pickle import dump, load
reviews = pd.read_csv("/deeplearning-keras/ch09/summarization/Reviews.csv")
print(reviews.shape)
print(reviews.head())
print(reviews.isnull().sum())
The output will be as follows:
(568454, 10)Id 0
ProductId 0
UserId 0
ProfileName 16
HelpfulnessNumerator 0
HelpfulnessDenominator 0
Score 0
Time 0
Summary 27
Text 0
2. Remove null values and unneeded features, as shown in the following snippet:
reviews = reviews.dropna()
reviews = reviews.drop(['Id','ProductId','UserId','ProfileName','HelpfulnessNumerator','HelpfulnessDenominator', 'Score','Time'], 1)
reviews = reviews.reset_index(drop=True) print(reviews.head())
for i in range(5):
print("Review #",i+1)
print(reviews.Summary[i])
print(reviews.Text[i])
print()
The output will be as follows:
Summary Text
0 Good Quality Dog Food I have bought several of the Vitality canned d...
1 Not as Advertised Product arrived labeled as Jumbo Salted Peanut...
2 "Delight," says it all This is a confection that has been around a fe...
3 Cough Medicine If you are looking for the secret ingredient i...
Review # 1
Not as Advertised - Product arrived labeled as Jumbo Salted Peanuts...the peanuts were actually small sized unsalted. Not sure if this was an error or if the vendor intended to represent the product as "Jumbo".
Review # 2
"Delight" says it all - This is a confection that has been around a few centuries. It is a light, pillowy citrus gelatin with nuts - in this case, Filberts. And it is cut into tiny squares and then liberally coated with powdered sugar. And it is a tiny mouthful of heaven. Not too chewy, and very flavorful. I highly recommend this yummy treat. If you are familiar with the story of C.S. Lewis' "The Lion, The Witch, and The Wardrobe" - this is the treat that seduces Edmund into selling out his Brother and Sisters to the Witch.
Review # 3
Cough Medicine - If you are looking for the secret ingredient in Robitussin I believe I have found it. I got this in addition to the Root Beer Extract I ordered (which was good) and made some cherry soda. The flavor is very medicinal.
By definition, a contraction is the combination of two words into a reduced form, with the omission of some internal letters and the use of an apostrophe. You can get the list of contractions from http://stackoverflow.com/questions/19790188/expanding-english-language-contractions-in-python.
3. Replace contractions with their longer forms, as shown here:
contractions = {
"ain't": "am not",
"aren't": "are not",
"can't": "cannot",
"can't've": "cannot have",
"'cause": "because",
"could've": "could have",
"couldn't": "could not",
"couldn't've": "could not have",
"didn't": "did not",
"doesn't": "does not",
"don't": "do not",
"hadn't": "had not",
"hadn't've": "had not have",
"hasn't": "has not",
"haven't": "have not",
"he'd": "he would",
"he'd've": "he would have",
4. Clean the text documents by replacing contractions and removing stop words:
def
# Convert words to lower case
text = text.lower()
if True:
text = text.split()
new_text = []
for word in text:
if word in contractions:
new_text.append(contractions[word])
else:
new_text.append(word)
text = " ".join(new_text)
text = re.sub(r'https?:\/\/.*[\r\n]*', '', text, flags=re.MULTILINE)
text = re.sub(r'\<a href', ' ', text)
text = re.sub(r'
text = re.sub(r'[_"\-;%()|+&=*%.,!?:#$@\[\]/]', ' ', text)
text = re.sub(r'
text = re.sub(r'\'', ' ', text)
if remove_stopwords:
text = text.split()
stops = set(stopwords.words("english"))
text = [w for w in text if not w in stops]
text = " ".join(text)
return text
5. Remove unwanted characters and, optionally, stop words. Also, make sure to replace the contractions, as shown previously. You can get the list of stop words from Natural Language Toolkit (NLTK), which helps with splitting sentences from paragraphs, splitting up words, and recognizing parts of speech. Import the toolkit using the following commands:
import nltk
nltk.download('stopwords')
6. Clean the summaries as shown in the following snippet:
# Clean the summaries and texts
clean_summaries = []
for summary in reviews.Summary:
clean_summaries.append(clean_text(summary, remove_stopwords=False))
print("Summaries are complete.")
clean_texts = []
for text in reviews.Text:
clean_texts.append(clean_text(text))
print("Texts are complete.")
7. Finally, save all the reviews into a pickle file. pickle serializes objects so they can be saved to a file and loaded in a program again later on:
stories = list()
for i, text in enumerate(clean_texts):
stories.append({'story': text, 'highlights': clean_summaries[i]})
# save to file
dump(stories, open('/deeplearning-keras/ch09/summarization/review_dataset.pkl', 'wb'))
Encoder-decoder architecture
Develop a basic character-level seq2seq model for text summarization. Use a word-level model, which is quite common in the domain of text processing. For this article, use character level models. As mentioned earlier, encoder and decoder architecture is a way of creating RNNs for sequence prediction. Encoders read the entire input sequence and encode it into an internal representation, usually, a fixed-length vector named the context vector. The decoder, on the other hand, reads the encoded input sequence from the encoder and produces the output sequence.
The encoder-decoder architecture consists of two primary models: one reads the input sequence and encodes it to a fixed-length vector, and the second decodes the fixed-length vector and outputs the predicted sequence. This architecture is designed for seq2seq problems.
1. Firstly, define the hyperparameters such as batch size, number of epochs for training, and number of samples to train:
batch_size = 64
epochs = 110
latent_dim = 256
num_samples = 10000
2. Next, load the review dataset from the pickle file:
stories = load(open('/deeplearning-keras/ch09/summarization/review_dataset.pkl', 'rb'))
print('Loaded Stories %d' % len(stories))
print(type(stories))
The output will be as follows:
Loaded Stories
3. Then, vectorize the data:
input_texts = []
target_texts = []
input_characters = set()
target_characters = set()
for story in stories:
input_text = story['story']
for highlight in story['highlights']:
target_text = highlight
# We use "tab" as the "start sequence" character
# for the targets, and "\n" as "end sequence" character.
target_text = '\t' + target_text + '\n'
input_texts.append(input_text)
target_texts.append(target_text)
for char in input_text:
if char not in input_characters:
input_characters.add(char)
for char in target_text:
if char not in target_characters:
target_characters.add(char)
input_characters = sorted(list(input_characters))
target_characters = sorted(list(target_characters))
num_encoder_tokens = len(input_characters)
num_decoder_tokens = len(target_characters)
max_encoder_seq_length = max([len(txt) for txt in input_texts])
max_decoder_seq_length = max([len(txt) for txt in target_texts])
print('Number of samples:', len(input_texts))
print('Number of unique input tokens:', num_encoder_tokens)
print('Number of unique output tokens:', num_decoder_tokens)
print('Max sequence length for inputs:', max_encoder_seq_length)
print('Max sequence length for outputs:', max_decoder_seq_length)
The output will be as follows:
Number of samples: 568411
Number of unique input tokens: 84
Number of unique output tokens: 48
Max sequence length for inputs: 15074
Max sequence length for outputs: 5
4. Now, create a generic function to define an encoder-decoder RNN:
def
# define training encoder
encoder_inputs = Input(shape=(None, n_input))
encoder = LSTM(n_units, return_state=True)
encoder_outputs, state_h, state_c = encoder(encoder_inputs)
encoder_states = [state_h, state_c]
# define training decoder
decoder_inputs = Input(shape=(None, n_output))
decoder_lstm = LSTM(n_units, return_sequences=True, return_state=True)
decoder_outputs, _, _ = decoder_lstm(decoder_inputs, initial_state=encoder_states)
decoder_dense = Dense(n_output, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)
# define inference encoder
encoder_model = Model(encoder_inputs, encoder_states)
# define inference decoder
decoder_state_input_h = Input(shape=(n_units,))
decoder_state_input_c = Input(shape=(n_units,))
decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]
decoder_outputs, state_h, state_c = decoder_lstm(decoder_inputs, initial_state=decoder_states_inputs)
decoder_states = [state_h, state_c]
decoder_outputs = decoder_dense(decoder_outputs)
decoder_model = Model([decoder_inputs] + decoder_states_inputs, [decoder_outputs] + decoder_states)
# return all models
return model, encoder_model, decoder_model
Training
1. For running the training, use rmsprop optimizer and categorical_crossentropy as the loss function:
# Run training
model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
model.fit([encoder_input_data, decoder_input_data], decoder_target_data,
batch_size=batch_size,
epochs=epochs,
validation_split=0.2)
# Save model
model.save('/deeplearning-keras/ch09/summarization/model2.h5')
The output will be as follows:
64/800 [=>............................] - ETA: 22:05 - loss: 2.1460
128/800 [===>..........................] - ETA: 18:51 - loss: 2.1234
192/800 [======>.......................] - ETA: 16:36 - loss: 2.0878
256/800 [========>.....................] - ETA: 14:38 - loss: 2.1215
320/800 [===========>..................] - ETA: 12:47 - loss: 1.9832
384/800 [=============>................] - ETA: 11:01 - loss: 1.8665
448/800 [===============>..............] - ETA: 9:17 - loss: 1.7547
512/800 [==================>...........] - ETA: 7:35 - loss: 1.6619
576/800 [====================>.........] - ETA: 5:53 - loss: 1.5820
512/800 [==================>...........] - ETA: 7:19 - loss: 0.7519
576/800 [====================>.........] - ETA: 5:42 - loss: 0.7493
640/800 [=======================>......] - ETA: 4:06 - loss: 0.7528
704/800 [=========================>....] - ETA: 2:28 - loss: 0.7553
768/800 [===========================>..] - ETA: 50s - loss: 0.7554
2. For inference, use the following method:
# generate target given source sequence
def predict_sequence(infenc, infdec, source, n_steps, cardinality):
# encode
state = infenc.predict(source)
# start of sequence input
target_seq = array([0.0 for _ in range(cardinality)]).reshape(1, 1, cardinality)
# collect predictions
output = list()
for t in range(n_steps):
# predict next char
yhat, h, c = infdec.predict([target_seq] + state)
# store prediction
output.append(yhat[0,0,:])
# update state
state = [h, c]
# update target sequence
target_seq = yhat
return array(output)
The output will be as follows:
Review(1): The coffee tasted great and was at such a good price! I highly recommend this to everyone!
Summary(1): great coffee
Review(2): This is the worst cheese that I have ever bought! I will never buy it again and I hope you won't either!
Summary(2): omg gross gross
Review(3): love individual oatmeal cups found years ago sam quit selling sound big lots quit selling found target expensive buy individually trilled get entire case time go anywhere need water microwave spoon to know quaker flavor packets
Summary(3): love it
If you found this article interesting, you can explore Keras Deep Learning Cookbook to leverage the power of deep learning and Keras to develop smarter and more efficient data models. Keras Deep Learning Cookbook shows you how to tackle different problems encountered while training efficient deep learning models, with the help of the popular Keras library.