Chapter 16 - Natural Language Processing with RNNs and Attention

Responsible for the session: Pex Tufvesson

Note: The third release of O'Reilly's book "Hands-on Machine Learning with Scikit-Learn, Keras & TensorFlow" was published in November 2019. A year has passed since, and in addition to the NLP models that the book refers to such as OpenAI's GPT-2 model Links to an external site. (which was updated in Nov 2019 as a 1.5B parameter model Links to an external site. with source-code) and BERT Links to an external site., we should keep an eye on at least Open-AI's GPT-3 Links to an external site. (175B parameters). 

Chapter summary

Your task is to read chapter 16 in the book, then read my summary below. When we meet, let's discuss if you agree with me or not on how I see things! I have made two experiments based on the theories in this chapter:

• Generating Swedish proverbs using character RNNs, aka "are you drunk?"
• Translating English idioms to Swedish using Sentiment Analysis, aka "Better than Google Translate"

Enjoy! / Pex

model = keras.models.Sequential([
  keras.layers.GRU(128, return_sequences=True, input_shape=[None, max_id], 
                   dropout=0.2, recurrent_dropout=0.2),
  keras.layers.GRU(128, return_sequences=True, 
                   dropout=0.2, recurrent_dropout=0.2),
  keras.layers.TimeDistributed(keras.layers.Dense(max_id, activation="softmax"))
])
model.compile(loss="sparse_categorical_crossentropy", optimizer="adam") 
history = model.fit(dataset, epochs=20) 

...and training takes a couple of hours.

The resulting text generator let's you choose two things:

1. The seed. The initial state of the RNN - the first couple of words to set the tone of the generated text to come.
2. The temperature. The allowed randomness. A temperature of 0 will always choose the next character as the one with the highest probability from the last softmax layer. The more you raise the temperature, the more likely the generator is to choose another char, from the probability vector gotten from the last softmax layer.

Choosing a low temperature will probably push the RNN into an endless loop of repeating the same chars over and over again.

Let the fun begin! aka: Are you drunk?

The course is Hands-On, so let's get started: I want to generate text myself. As you may or may not know, I am the webmaster of the largest Swedish proverb collection at https://livet.se/ord Links to an external site. - which has been online since 2005. I grabbed the latest MySql dump from the site, and ran

mysql -uroot -p -Dpex.livet -s -e"SELECT prov_se FROM ord_proverbs WHERE NOT ((prov_se == '') || (prov_se == '_'));" > mysql_log.txt

...which results in a text file with 3146kB of Swedish text, from 42318 Swedish proverbs.

I will not paste the code in here, since it's quite long. Total training time on an Intel Corei9 laptop was 30 hours. Training time on an AMD Ryzen9 3900XT was 60 hours.

I did use an nVidia 1080 Ti GPU for training, and it took 1h45m. However, RNNs trained in Tensorflow have quite strict rules on what features you're allowed to use, so recurrent_dropout was supposed to be set to 0.2, but tensorflow's GPU implementation does not allow that. You can read more about that here: https://www.tensorflow.org/api_docs/python/tf/keras/layers/GRU Links to an external site.

Before doing anything clever, the characters in the source text were:

!"#%&'()*+,-./0123456789:;<=>?[\]_`abcdefghijklmnopqrstuvwxyz|}~ ¤¥¨«­´¶·»½¿àãäåæçèéëíïôöøüý–—‘’”…

However, to keep the number of tokens down, I replaced or removed some characters to make the set smaller. All characters in the source text, after replacing:

!"#%&'()*+,-./0123456789:;<=>?[\]_`abcdefghijklmnopqrstuvwxyz|}~ «­´ãäåéöøü

Using these chars, tokenizing a string like "Skulle gå ud etter öl?" becomes: [[7, 14, 22, 9, 9, 3, 1, 13, 21, 1, 22, 10, 1, 3, 5, 5, 3, 6, 1, 23, 9, 34]] ...and then back to "['skulle gå ud efter öl?']" again. There are 77 unique tokens in my encoding.

The neural network I used was

Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param # 
=================================================================
gru (GRU) (None, None, 128) 79488 
_________________________________________________________________
gru_1 (GRU) (None, None, 128) 99072 
_________________________________________________________________
time_distributed (TimeDistri (None, None, 77) 9933 
=================================================================
Total params: 188,493
Trainable params: 188,493
Non-trainable params: 0
_________________________________________________________________

As a first text, I predicted one character to complete the sentence "hon är kvinn". It chose "o", which seems fair. I was aiming for "a", but I should be happy with an "o" as well.

Playing around with the temperature, leads to results like:

### Using seed_text='e' t=0.200000,
generated_text='en av det som ska vara som för att spela sig själv.
beskull: en person som ska vara stora problem och'

### Using seed_text='e' t=1.000000,
generated_text='er framgång skulle titta jord för att många nationer förstår att göra.
uttami: elemerings pirsom att '

### Using seed_text='e' t=2.000000,
generated_text='ers djug: göds vnamt scekjofin.
scojöbuke: hnviextmåb.. utmansata advinuten. så lämj "naskel: klok.
v'


For the rest of my generated proverbs, I choose 0.3 as the temperature. And, I have manually removed all text after the first "." character.
The generated proverbs, when using letters "a"-"k" as seed, are:

'a stället av stället som kan göra det som för att man är beskyller det att man ska bli värd.'
'ber som för att man inte har till dem.'
'cken med en stor förstånd.'
'de för att de ska bli valje som ska man inte vara som den man som gör någon in för att spela en skill'
'finner sig själva.'
'g som man inte har till den som ska på det som inte behöver honom.'
'h som man inte har till den som ska på det som inte behöver honom.'
'inte med en stor första känniskor som gör en vidskepelse.'
'ja för att de ska bli valje som ska man inte vara som den man som gör någon in för att spela en skill'
'ka för min själv.'


To get texts that makes more sense, I used longer seeds:


'En sommar ' => 'En sommar som gör aldrig något som de ska vara så länge ett genom att skydda den som inte har till den som ska'
'Som man ' => 'Som man inte har gjort någonting.
fisk för att man ska vara sälja som talar om det som lever något annat än '
'Tidigare ' => 'Tidigare är det inte skillnaden mellan två saken som vill ha ett litet till den som inte har till den.
man sk'
'Man tar ' => 'Man tar sig själva.
det är bete vädret som gör det som är betänga som talar om det som lever något.
det är b'
'Utanför ' => 'Utanför att man inte har till den som är betende men inte betänga att ta andra att skilja sig av att glömma '
'I Lund ' => 'I Lund för att man inte har till folk som gör det som är betänga som talar om det som lever något.
det är b'
'Tvinga ' => 'Tvinga som gör det som för tiden för att spela sinnet för att skilja sig en största första första graden.
v'
'Efteråt ' => 'Efteråt som gör det som för tiden för att spela sinnet för att skilja sig en största första första graden.
v'
'Liggande ' => 'Liggande som gör det som för tiden för att spela sinnet för att skilja sig en största första första graden.
v'
'Tango ' => 'Tango sig själva.
det är bete vädret som gör det som är betänga som talar om det som lever något.
det är b'
'En Dans, ' => 'En Dans, men det är betenare att vara för att man inte har gått ut som till den.
en stor man är en stor som v'
'Fredag ' => 'Fredag som gör aldrig något som de ska vara så länge ett genom att skydda den som inte har till den som ska'
'Vi ska gå ' => 'Vi ska gå i filmen.
det är bete som finns i spillning av att betrakta sig en av det med som ska avskilla sig a'
'Spännande ' => 'Spännande springer och skillnaden med vatten på det som inte behöver honom.
det är inte samma som man ska beta'
'Tomten ' => 'Tomten som gör aldrig något som de ska vara så länge ett genom att skydda den som inte har till den som ska'
'Lucia ' => 'Lucia som de första vänner.
det finns inget som får inte betraktar att ge det som för mycket som ger sig e'
'En dvärg ' => 'En dvärg och skydda sig själva.
kvinnor skulle hålla vilja som inte har till dem.
det är bete när du har till'
'Med svärd ' => 'Med svärd för att man inte har till folk som gör det som är betänga som talar om det som lever något.
det är b'
'Jul ' => 'Jul som gör det som för tiden för att spela sinnet för att skilja sig en största första första graden.
v'
'Glögg ' => 'Glögg i hemma.
konsten att tro att de inte har fått en man som man inte har till den.
man ska vara skillna'
'Slut ' => 'Slut är det som kan göra sig själv.
det är bete när det är en sallan som inte har för mycket stor skapar '

Looking at texts coming out of a badly trained RNN like this actually helps you in identifying what's wrong with texts coming out of a well trained neural network like GPT-2 or GPT-3. The same kind of "I have no clue of what this text is supposed to say" is present in more advanced text generators as well. However, they do fewer grammatical errors.

I was forced to use recurrent_dropout = 0.0 instead of 0.2. Regardless, creating sentences from two GRU layers may not be the ideal way of inventing proverbial masterpieces.

model = keras.models.Sequential([
  keras.layers.GRU(128, return_sequences=True, stateful=True, dropout=0.2, 
                   recurrent_dropout=0.2, batch_input_shape=[batch_size, None, max_id]),
  keras.layers.GRU(128, return_sequences=True, stateful=True, dropout=0.2,
                   recurrent_dropout=0.2), 
  keras.layers.TimeDistributed(keras.layers.Dense(max_id, activation="softmax"))
])

...and the shuffling of windows cannot be done as wildly as before, since the GRU layers will keep their state between windows. The upside is that the state will be meaningful for every character that it gets as input, speeding up the training process. The downside is that batching and handling of the input text needs to be done more carefully.

To improve their small IMDb-example, they use a pre-trained "nnlm-en-dim50" sentence embedding that's already packed with knowledge by having synonyms grouped using a 7 billion word corpus.

It feels like they are cheating big time. But, I guess we should see pre-trained embeddings as just another tool in the tool box.

Let the fun begin! aka: Better than Google Translate.

The course is for getting our hands dirty, so let's get started. I happen to collect proverbs, and one certain class of proverbs are idioms Links to an external site.. They are really difficult to translate, since often the words you see in a sentence is not at all related to common knowledge about these words. For instance, take these examples:

That´s a close shave. -> Det var nära ögat.
That´s another cup of tea. -> Det är en annan femma.
You look like a dying duck in a thunder­storm.
    -> Du ser ut som om du sålt smöret och tap­pat pengarna.
Put that in your pipe and smoke it! -> Där fick du så du teg!
Nothing ventured, nothing gained. -> Friskt vågat, hälften vunnet.
Make a mountain out of a molehill. -> Göra en höna av en fjäder.
Have a finger in the pie. -> Ha ett finger med i spe­let.
Give somebody the benefit of the doubt. -> Hellre fria än fälla.
I smell a rat. -> Här ligger en hund be­graven.
Like a bolt from the blue. -> Som en blixt från en klar himmel.

Putting these into Google Translate Links to an external site. gives us:

That's a close shave. -> Det är en nära rakning.
That´s another cup of tea. -> Det är ännu en kopp te.
You look like a dying duck in a thunder­storm. -> Du ser ut som en döende anka i åskväder.
Put that in your pipe and smoke it! -> Lägg det i röret och rök det!
Nothing ventured, nothing gained. -> Inget vågat ingenting vunnit. ***
Make a mountain out of a molehill. -> Gör ett berg av en mullvad.
Have a finger in the pie. -> Ha ett finger i pajen.
Give somebody the benefit of the doubt. -> Ge någon fördelen av tvivel.
I smell a rat -> jag känner lukten av en råtta.
Like a bolt from the blue. -> Som en bult från det blå.

...and it's obvious that one of the world's richest company doesn't know a thing about idioms. At least not when it comes to translating them to Swedish.

*** Google is actually boasting about having "verified translations" that are clearly dead wrong:

Can we do better than Google - with the help of Google? Let's find out:

We could for instance take a look at tokenizing sentences. Google has published their Universal Sentence Encoder Links to an external site.. It will for instance help you in making a confusion matrix for sentences like this:

...how can we utilize this for translating English idioms to Swedish? I have a list of translated idioms at https://livet.se/ord/källa/Idiom Links to an external site. - there's 1890 "perfect" translations there (for a certain definition of perfect). Let's see if we can get a hold of these in a way that's more manageable for Python than a webpage.

sudo mysql -uroot -p -Dpex.livet -s -e"SELECT prov_en,prov_se FROM ord_proverbs WHERE ((author_id = 20402) AND NOT ((prov_en = '') || (prov_se = '') || (prov_se = '_')));" > idioms_log.txt
(vpex1) pex@p16 ~/phd/201205_Translate_English_idioms_to_Swedish $ head idioms_log.txt
Good riddance!	Ajöss med den!
An/one's Achilles' heel	En/någons akilleshäl
Not on your life/nelly!	Aldrig i livet!
All roads lead to Rome	Alla vägar bär till Rom
All hands to the pumps!	Alle man till pumparna!
All or nothing	Allt eller intet
All told	Allt som allt
For all one is worth	Allt vad tygen håller
All that glitters is not gold	Allt är inte guld som glimmar
Always fall on one's feet	Alltid komma ned på fötterna

...seems like a good start. We now have a text file, where one English idiom with its Swedish translation is separated with a tab character on every line.

    # Encoding English_idioms text with the Universal Sentence Encoder
    # The Universal Sentence Encoder is a powerful Transformer model (in its large version)
    # allowing to extract embeddings directly from sentences instead of from individual words.

    # Load a pre-trained model for sentence tokenizing from tensorflow hub:
    print("### Grabbing pre-trained sentence encoder:")
    model = tfhub.load("https://tfhub.dev/google/universal-sentence-encoder/4")

    # Grab the English Idioms that we fully know a prefect translation of:
    print("### Loading idioms:")
    English_idioms = []
    Swedish_idioms = []
    with open("idioms_log.txt") as f:
        all_lines = f.readlines()
        for line in all_lines:
            sp = line.split("\t")
            English_idioms.append(sp[0])
            Swedish_idioms.append(sp[1][:-1])

    # We are first using the model to encode all the idioms that the user shall search for:
    print("### Translating all English_idioms to embeddings:")
    print("There are %d idioms to process" % len(English_idioms))
    batch_size = 10
    embeddings = []
    for i in range(0, len(English_idioms), batch_size):
        embeddings.append(model(English_idioms[i : i + batch_size]))
    English_idioms_embeddings = tf.concat(embeddings, axis=0)

    # When a user is searching for an idiom, we can extract
    # its embedding and find the most similar idiom
    # in our database of translated idioms. In our case
    # we use a simple vector dot product as a similary function:
    def find_best_translation(English_idiom: str) -> str:
        embedding = model([English_idiom])
        # compute dot product with each English idiom:
        scores = English_idioms_embeddings @ tf.transpose(embedding)
        idiom_number = np.argmax(tf.squeeze(scores).numpy())
        return idiom_number

    search_texts = [
        "Sweat like a horse",
        "Work for a week",
        "My cup of tea",
        "child of time",
        "six and a half dozen",
        "penny",
        "pretty",
    ]
    for search_text in search_texts:
        idiom_no = find_best_translation(search_text)
        print(
            "You searched for '%s', and I found: '%s' -> '%s'."
            % (search_text, English_idioms[idiom_no], Swedish_idioms[idiom_no])
        )

    while search_text != "quit":
        search_text = input(
            "Write an English idiom you'd like to translate to Swedish, 'quit' to quit:\n"
        )
        idiom_no = find_best_translation(search_text)
        print(
            "You searched for '%s', and I found: '%s' -> '%s'."
            % (search_text, English_idioms[idiom_no], Swedish_idioms[idiom_no])
        )

How well does this perform, then? Let's run it:

### Grabbing pre-trained sentence encoder:
2020-12-05 10:54:45.915011: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-12-05 10:54:45.926230: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f9b5fe629c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-12-05 10:54:45.926247: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
### Loading idioms:
### Translating all English_idioms to embeddings:
There are 1890 idioms to process
You searched for 'Sweat like a horse', and I found: 'Rough it; work like a horse' -> 'Slita hund'.
You searched for 'Work for a week', and I found: 'Work like a slave' -> 'Arbeta som en slav'.
You searched for 'My cup of tea', and I found: 'Not be somebody's cup of tea' -> 'Inte vara någons likör'.
You searched for 'child of time', and I found: 'Child's play' -> 'En barnlek'.
You searched for 'six and a half dozen', and I found: 'It's as broad as it's long; it's six of one and half a dozen of the other' -> 'Det går på ett ut'.
You searched for 'penny', and I found: 'A pretty penny' -> 'En vacker slant'.
You searched for 'pretty', and I found: 'A raving beauty' -> 'En strålande skönhet'.
Write an English idiom you'd like to translate to Swedish, 'quit' to quit:
flower
You searched for 'flower', and I found: 'Hair-splitting' -> 'Hårklyverier'.
Write an English idiom you'd like to translate to Swedish, 'quit' to quit:
welcome
You searched for 'welcome', and I found: 'There you are!' -> 'Där ser du!'.
Write an English idiom you'd like to translate to Swedish, 'quit' to quit:
first and last
You searched for 'first and last', and I found: 'First and foremost' -> 'Först och främst'.
Write an English idiom you'd like to translate to Swedish, 'quit' to quit:
split hair
You searched for 'split hair', and I found: 'Split hairs' -> 'Ägna sig åt hårklyverier'.
Write an English idiom you'd like to translate to Swedish, 'quit' to quit:
haircut
You searched for 'haircut', and I found: 'Tear one's hair' -> 'Slita sitt hår'.
Write an English idiom you'd like to translate to Swedish, 'quit' to quit:
quit
You searched for 'quit', and I found: 'Kill (the) time' -> 'Fördriva tiden'.

Well, it's not perfect, but it's a lot better than Google Translate!

Encoder-Decoder NN for Machine Translation

This sub-chapter describes a paper on Sequence to Sequence Learning with Neural Networks Links to an external site. for translating English to French. The author of the book is French, btw.

The encoder is fed the source sentences in reversed order "because doing so introduced many short term dependencies between the source and the target sentence". Translation behaves like a LIFO - last word in is the one that's supposed to be translated and output first.

The source sentences are tokenized into multiple word ids, and then an embedding layer will turn each word into a n-dimentional vector.

There are a lot of fine details missing. They use a mixed bag of tricks to speed up:

• Start-of-sequence (SOS) tokens, End-of-sequence (EOS) tokes, to handle multiple length sentences.
• Feeding the decoder "the correct French word" after each step at training time, to speed up learning.
• Grouping sentences to be translated into buckets of similar length, and use differently trained NNs for different lengths. Use of padding to make a too short sentence fit into a NN trained for longer sentences.
• Masking out EOS tokens.
• Use sampled softmax Links to an external site. instead of softmax. 
• Bidirectional RNNs.

My general NLP observations (...and others' observations)

GPT-3 has trouble with Bias & Unreliability:

• If you ask it for writing a kid's story, it will deviate and start writing a horror plot text.
• If we give it a task, the solution is chosen that looks most likely given the training data.

Francois Chollet author of "Deep learning with python" states:
"It is only constrained by plausability and not other important things such as factfullness or consitency, which is why it's so easy to generate things with GPT-3 that are untrue or even self contradictory."

Commercial uses of NLP

GPT-2 and BERT can be downloaded and run locally. OpenAI.com has a price-list for using their GPT-3 model: 100.000 tokes are "free", so you can get started for free, but you'll need to pay when you're doing it on an enterprise level.

Cost of training is substantial: https://www.theregister.com/2020/11/04/gpt3_carbon_footprint_estimate/ Links to an external site. 
"More specifically, they estimated teaching the [GPT-3] neural super-network in a Microsoft data center using Nvidia GPUs required roughly 190,000 kWh"

OpenAI has sold GPT-3 to Microsoft. Links to an external site.

Additional resources

Papers and webpages:

• Simple example on how to build a simple chatbot with the help of Google Sheets and Google's Universal Sentence Encoder Links to an external site.: https://jonathanbgn.com/nlp/2020/09/29/chatbot-universal-sentence-encoder.html Links to an external site. 
• Machine wisdom. An online quote generator using OpenAI's GPT-2 model Links to an external site.: https://machineswisdom.com/ Links to an external site. ...and how it's done using GPT-2 and GPT-3: https://jonathanbgn.com/nlp/2020/08/30/gpt2-gpt3-creativity.html Links to an external site. 

Session Agenda

The meeting plan for the 11th of December 2020:

• Go through the chapter summary above and discuss that.
• Discuss the suggested exercises.
• If we have time and if you are interested let's talk about GPT-3.

Recommended exercises

If we get the time, we could discuss the following exercises from the book: