Vectors¶
Vectors are useful since they help us summarize information about an object using numbers. Then, using the number representation, we can make comparisons between the vector representations of different objects.
Say you are working at a school and want to analyze student scores. Take a look at the vectors representing the student's scores plotted below. What do you notice? Are they close together? How similar is the vector for Alena’s scores to Xavier’s and Niko’s?
import numpy as np
import matplotlib.pyplot as plt
scores_xavier = np.array([88,92])
scores_niko = np.array([94,87])
scores_alena = np.array([90,48])
plt.arrow(0, 0, scores_xavier[0], scores_xavier[1], width=1, color='blue', label='xavier')
plt.arrow(0, 0, scores_niko[0], scores_niko[1], width=1, color='orange', label='niko')
plt.arrow(0, 0, scores_alena[0], scores_alena[1], width=1, color='purple', label='alena')
plt.axis([0, 100, 0, 100])
plt.legend(bbox_to_anchor=(1,1), loc='upper left')
Distance¶
There are a variety of ways to find the distance between vectors, and here we will cover three.
Manhattan distance
Manhattan distance, also known as city block distance, is defined as the sum of the differences across each individual dimension of the vectors. Consider the vectors [1,2,3] and [2,4,6]. We can calculate the Manhattan distance between them as shown below:manhattan distance = ∣1−2∣ + ∣2−4∣ + ∣3−6∣ = 1 + 2 + 3 = 6Euclidean distance
In Euclidean distance, also known as straight line distance, we take the square root of the sum of the squares of the differences in each dimension.euclidean distance = √{(1−2)^2 + (2−4)^2 + (3−6)^2} = √{14} ≈ 3.74Cosine distance
Cosine distance is concerned with the angle between two vectors, rather than by looking at the distance between the points, or ends, of the vectors. Two vectors that point in the same direction have no angle between them, and have a cosine distance of 0. Two vectors that point in opposite directions, on the other hand, have a cosine distance of 1. For the mathematically adventurous, you can read up on the calculation here.
vector_a = np.array([1,2,3])
vector_b = np.array([2,4,6])
plt.arrow(0, 0, vector_b[0], vector_b[1], width=.05, color='orange', label='b')
plt.arrow(0, 0, vector_a[0], vector_a[1], width=.05, color='blue', label='a')
plt.axis([0, 3, 0, 6])
plt.legend(bbox_to_anchor=(1,1), loc='upper left')
from scipy.spatial.distance import cityblock, euclidean, cosine
print('Manhattan distance:', cityblock(vector_a,vector_b))
print('Euclidean distance:', euclidean(vector_a,vector_b))
print('Cosine distance: ', cosine(vector_a,vector_b))
When working with vectors that have a large number of dimensions, such as word embeddings, the distances calculated by Manhattan and Euclidean distance can become rather large. Thus, calculations using cosine distance are usually preferred.
Word embeddings¶
Word embeddings are vector representations of a word.
The idea behind word embeddings is a theory known as the distributional hypothesis. This hypothesis states that words that co-occur in the same contexts tend to have similar meanings. With word embeddings, we map words that exist with the same context to similar places in our vector space (math-speak for the area in which our vectors exist). Example:The kid said he would grow up to be superman The child said he would grow up to be superman
The words kid and child will have similar word vectors due to a similar context.
Note: Embedding can be considered as a synonym for encoding.The numeric values that are assigned to the vector representation of a word are not important in their own right, but gather meaning from how similar or not words are to each other. Thus the cosine distance between words with similar contexts will be small, and the cosine distance between words that have very different contexts will be large.
We can retrieve word embeddings for english words using spaCy:
import spacy
!python -m spacy download en_core_web_lg
nlp = spacy.load('en_core_web_lg')
happy_vec = nlp('happy').vector
sad_vec = nlp('sad').vector
angry_vec = nlp('angry').vector
len(happy_vec)
# Cosine distance
print('happy/sad:', cosine(happy_vec, sad_vec))
print('angry/sad:', cosine(angry_vec, sad_vec))
Below is a list of 1,000 most common English words:
most_common_words = ['a',
'ability',
'able',
'about',
'above',
'accept',
'according',
'account',
'across',
'act',
'action',
'activity',
'actually',
'add',
'address',
'administration',
'admit',
'adult',
'affect',
'after',
'again',
'against',
'age',
'agency',
'agent',
'ago',
'agree',
'agreement',
'ahead',
'air',
'all',
'allow',
'almost',
'alone',
'along',
'already',
'also',
'although',
'always',
'American',
'among',
'amount',
'analysis',
'and',
'animal',
'another',
'answer',
'any',
'anyone',
'anything',
'appear',
'apply',
'approach',
'area',
'argue',
'arm',
'around',
'arrive',
'art',
'article',
'artist',
'as',
'ask',
'assume',
'at',
'attack',
'attention',
'attorney',
'audience',
'author',
'authority',
'available',
'avoid',
'away',
'baby',
'back',
'bad',
'bag',
'ball',
'bank',
'bar',
'base',
'be',
'beat',
'beautiful',
'because',
'become',
'bed',
'before',
'begin',
'behavior',
'behind',
'believe',
'benefit',
'best',
'better',
'between',
'beyond',
'big',
'bill',
'billion',
'bit',
'black',
'blood',
'blue',
'board',
'body',
'book',
'born',
'both',
'box',
'boy',
'break',
'bring',
'brother',
'budget',
'build',
'building',
'business',
'but',
'buy',
'by',
'call',
'camera',
'campaign',
'can',
'cancer',
'candidate',
'capital',
'car',
'card',
'care',
'career',
'carry',
'case',
'catch',
'cause',
'cell',
'center',
'central',
'century',
'certain',
'certainly',
'chair',
'challenge',
'chance',
'change',
'character',
'charge',
'check',
'child',
'choice',
'choose',
'church',
'citizen',
'city',
'civil',
'claim',
'class',
'clear',
'clearly',
'close',
'coach',
'cold',
'collection',
'college',
'color',
'come',
'commercial',
'common',
'community',
'company',
'compare',
'computer',
'concern',
'condition',
'conference',
'Congress',
'consider',
'consumer',
'contain',
'continue',
'control',
'cost',
'could',
'country',
'couple',
'course',
'court',
'cover',
'create',
'crime',
'cultural',
'culture',
'cup',
'current',
'customer',
'cut',
'dark',
'data',
'daughter',
'day',
'dead',
'deal',
'death',
'debate',
'decade',
'decide',
'decision',
'deep',
'defense',
'degree',
'Democrat',
'democratic',
'describe',
'design',
'despite',
'detail',
'determine',
'develop',
'development',
'die',
'difference',
'different',
'difficult',
'dinner',
'direction',
'director',
'discover',
'discuss',
'discussion',
'disease',
'do',
'doctor',
'dog',
'door',
'down',
'draw',
'dream',
'drive',
'drop',
'drug',
'during',
'each',
'early',
'east',
'easy',
'eat',
'economic',
'economy',
'edge',
'education',
'effect',
'effort',
'eight',
'either',
'election',
'else',
'employee',
'end',
'energy',
'enjoy',
'enough',
'enter',
'entire',
'environment',
'environmental',
'especially',
'establish',
'even',
'evening',
'event',
'ever',
'every',
'everybody',
'everyone',
'everything',
'evidence',
'exactly',
'example',
'executive',
'exist',
'expect',
'experience',
'expert',
'explain',
'eye',
'face',
'fact',
'factor',
'fail',
'fall',
'family',
'far',
'fast',
'father',
'fear',
'federal',
'feel',
'feeling',
'few',
'field',
'fight',
'figure',
'fill',
'film',
'final',
'finally',
'financial',
'find',
'fine',
'finger',
'finish',
'fire',
'firm',
'first',
'fish',
'five',
'floor',
'fly',
'focus',
'follow',
'food',
'foot',
'for',
'force',
'foreign',
'forget',
'form',
'former',
'forward',
'four',
'free',
'friend',
'from',
'front',
'full',
'fund',
'future',
'game',
'garden',
'gas',
'general',
'generation',
'get',
'girl',
'give',
'glass',
'go',
'goal',
'good',
'government',
'great',
'green',
'ground',
'group',
'grow',
'growth',
'guess',
'gun',
'guy',
'hair',
'half',
'hand',
'hang',
'happen',
'happy',
'hard',
'have',
'he',
'head',
'health',
'hear',
'heart',
'heat',
'heavy',
'help',
'her',
'here',
'herself',
'high',
'him',
'himself',
'his',
'history',
'hit',
'hold',
'home',
'hope',
'hospital',
'hot',
'hotel',
'hour',
'house',
'how',
'however',
'huge',
'human',
'hundred',
'husband',
'I',
'idea',
'identify',
'if',
'image',
'imagine',
'impact',
'important',
'improve',
'in',
'include',
'including',
'increase',
'indeed',
'indicate',
'individual',
'industry',
'information',
'inside',
'instead',
'institution',
'interest',
'interesting',
'international',
'interview',
'into',
'investment',
'involve',
'issue',
'it',
'item',
'its',
'itself',
'job',
'join',
'just',
'keep',
'key',
'kid',
'kill',
'kind',
'kitchen',
'know',
'knowledge',
'land',
'language',
'large',
'last',
'late',
'later',
'laugh',
'law',
'lawyer',
'lay',
'lead',
'leader',
'learn',
'least',
'leave',
'left',
'leg',
'legal',
'less',
'let',
'letter',
'level',
'lie',
'life',
'light',
'like',
'likely',
'line',
'list',
'listen',
'little',
'live',
'local',
'long',
'look',
'lose',
'loss',
'lot',
'love',
'low',
'machine',
'magazine',
'main',
'maintain',
'major',
'majority',
'make',
'man',
'manage',
'management',
'manager',
'many',
'market',
'marriage',
'material',
'matter',
'may',
'maybe',
'me',
'mean',
'measure',
'media',
'medical',
'meet',
'meeting',
'member',
'memory',
'mention',
'message',
'method',
'middle',
'might',
'military',
'million',
'mind',
'minute',
'miss',
'mission',
'model',
'modern',
'moment',
'money',
'month',
'more',
'morning',
'most',
'mother',
'mouth',
'move',
'movement',
'movie',
'Mr',
'Mrs',
'much',
'music',
'must',
'my',
'myself',
'name',
'nation',
'national',
'natural',
'nature',
'near',
'nearly',
'necessary',
'need',
'network',
'never',
'new',
'news',
'newspaper',
'next',
'nice',
'night',
'no',
'none',
'nor',
'north',
'not',
'note',
'nothing',
'notice',
'now',
"n't",
'number',
'occur',
'of',
'off',
'offer',
'office',
'officer',
'official',
'often',
'oh',
'oil',
'ok',
'old',
'on',
'once',
'one',
'only',
'onto',
'open',
'operation',
'opportunity',
'option',
'or',
'order',
'organization',
'other',
'others',
'our',
'out',
'outside',
'over',
'own',
'owner',
'page',
'pain',
'painting',
'paper',
'parent',
'part',
'participant',
'particular',
'particularly',
'partner',
'party',
'pass',
'past',
'patient',
'pattern',
'pay',
'peace',
'people',
'per',
'perform',
'performance',
'perhaps',
'period',
'person',
'personal',
'phone',
'physical',
'pick',
'picture',
'piece',
'place',
'plan',
'plant',
'play',
'player',
'PM',
'point',
'police',
'policy',
'political',
'politics',
'poor',
'popular',
'population',
'position',
'positive',
'possible',
'power',
'practice',
'prepare',
'present',
'president',
'pressure',
'pretty',
'prevent',
'price',
'private',
'probably',
'problem',
'process',
'produce',
'product',
'production',
'professional',
'professor',
'program',
'project',
'property',
'protect',
'prove',
'provide',
'public',
'pull',
'purpose',
'push',
'put',
'quality',
'question',
'quickly',
'quite',
'race',
'radio',
'raise',
'range',
'rate',
'rather',
'reach',
'read',
'ready',
'real',
'reality',
'realize',
'really',
'reason',
'receive',
'recent',
'recently',
'recognize',
'record',
'red',
'reduce',
'reflect',
'region',
'relate',
'relationship',
'religious',
'remain',
'remember',
'remove',
'report',
'represent',
'Republican',
'require',
'research',
'resource',
'respond',
'response',
'responsibility',
'rest',
'result',
'return',
'reveal',
'rich',
'right',
'rise',
'risk',
'road',
'rock',
'role',
'room',
'rule',
'run',
'safe',
'same',
'save',
'say',
'scene',
'school',
'science',
'scientist',
'score',
'sea',
'season',
'seat',
'second',
'section',
'security',
'see',
'seek',
'seem',
'sell',
'send',
'senior',
'sense',
'series',
'serious',
'serve',
'service',
'set',
'seven',
'several',
'sex',
'sexual',
'shake',
'share',
'she',
'shoot',
'short',
'shot',
'should',
'shoulder',
'show',
'side',
'sign',
'significant',
'similar',
'simple',
'simply',
'since',
'sing',
'single',
'sister',
'sit',
'site',
'situation',
'six',
'size',
'skill',
'skin',
'small',
'smile',
'so',
'social',
'society',
'soldier',
'some',
'somebody',
'someone',
'something',
'sometimes',
'son',
'song',
'soon',
'sort',
'sound',
'source',
'south',
'southern',
'space',
'speak',
'special',
'specific',
'speech',
'spend',
'sport',
'spring',
'staff',
'stage',
'stand',
'standard',
'star',
'start',
'state',
'statement',
'station',
'stay',
'step',
'still',
'stock',
'stop',
'store',
'story',
'strategy',
'street',
'strong',
'structure',
'student',
'study',
'stuff',
'style',
'subject',
'success',
'successful',
'such',
'suddenly',
'suffer',
'suggest',
'summer',
'support',
'sure',
'surface',
'system',
'table',
'take',
'talk',
'task',
'tax',
'teach',
'teacher',
'team',
'technology',
'television',
'tell',
'ten',
'tend',
'term',
'test',
'than',
'thank',
'that',
'the',
'their',
'them',
'themselves',
'then',
'theory',
'there',
'these',
'they',
'thing',
'think',
'third',
'this',
'those',
'though',
'thought',
'thousand',
'threat',
'three',
'through',
'throughout',
'throw',
'thus',
'time',
'to',
'today',
'together',
'tonight',
'too',
'top',
'total',
'tough',
'toward',
'town',
'trade',
'traditional',
'training',
'travel',
'treat',
'treatment',
'tree',
'trial',
'trip',
'trouble',
'true',
'truth',
'try',
'turn',
'TV',
'two',
'type',
'under',
'understand',
'unit',
'until',
'up',
'upon',
'us',
'use',
'usually',
'value',
'various',
'very',
'victim',
'view',
'violence',
'visit',
'voice',
'vote',
'wait',
'walk',
'wall',
'want',
'war',
'watch',
'water',
'way',
'we',
'weapon',
'wear',
'week',
'weight',
'well',
'west',
'western',
'what',
'whatever',
'when',
'where',
'whether',
'which',
'while',
'white',
'who',
'whole',
'whom',
'whose',
'why',
'wide',
'wife',
'will',
'win',
'wind',
'window',
'wish',
'with',
'within',
'without',
'woman',
'wonder',
'word',
'work',
'worker',
'world',
'worry',
'would',
'write',
'writer',
'wrong',
'yard',
'yeah',
'year',
'yes',
'yet',
'you',
'young',
'your',
'yourself']
len(most_common_words)
and their corresponding word embedding:
vector_list = [nlp(x).vector for x in most_common_words]
len(vector_list)
Let's check which words are the closest to "food" — by retrieving word embeddings that are the closest to its word embedding. Are you surprised?
def find_closest_words(word, n=10):
global most_common_words, vector_list
vec = vector_list[most_common_words.index(word)]
return sorted(most_common_words,
key=lambda x: cosine(
vector_list[most_common_words.index(x)],
vec
))[:n]
find_closest_words('food')
find_closest_words('summer')
Create your own Words embeddings¶
Depending on the corpus of text we select to train a word embedding model, different word embeddings will be created according to the context of the words in the given corpus. The larger and more generic a corpus, the more generalizable the word embeddings become.
So far, we have been using pre-trained word embedding models stored in spaCy. These models were trained (using word2vec) on blog posts and news articles collected by the Linguistic Data Consortium at the University of Pennsylvania.
What if we want to train our own word embeddings?
Different algorithms (and models) have been developed to create word embeddings from a corpus of text:Word2Vec:
Word2Vec was created by Google researchers (Tomas Mikolov, Kai Chen, Greg Corrado and Jefrrey Dean) in 2013. They provided a model with 300 dimensions, trained on 3 million words from Google News data. Team used skip-gram and negative sampling to build this model.GloVe:
Global Vectors for word representation (GloVe) was created by Stanford researchers (Jeffrey Pennington, Richard Socher and Chris Manning) in 2014. They provided various models from 25, 50, 100, 200 to 300 dimensions based on 2, 6, 42, 840 billion tokens. Team used word-to-word co-occurrence to build this model — in other words, if two words co-occur many times, it means they have some linguistic or semantic similarity.fastText:
fastText was developed by Facebook. They provide 3 models with 300 dimensions each. fastText is able to achieve good performance for word representations and sentence classifications because they are making use of character level representations. Each word is represented as bag of characters n-grams in addition to the word itself. For example, for the word partial, with n=3, the fastText representation for the character n-grams is <pa, art, rti, tia, ial, al>. <and> are added as boundary symbols to separate the n-grams from the word itself.
Word2vec tends to be the most popular. The package gensim has a Word2Vec() function that you can use to build your own word embeddings on any corpus of text you like.
import re
from nltk.corpus import stopwords
stopwords_en = stopwords.words('english')
def preprocess_text(txt):
txt = re.sub('[^0-9a-zA-Z \t]', '', txt)
return [word
for word in txt.lower().split()
if word not in stopwords_en]
docs = ['kings and queens go together',
'men and women go together',
'queens are women',
'kings are men']
docs_tokenized = [preprocess_text(doc) for doc in docs]
docs_tokenized
from gensim.models import Word2Vec
model = Word2Vec(
docs_tokenized,
size=5, # word embeddings length
window=5, # window
min_count=1, # ignores all words with total frequency lower than this
sg=0, # 1 for skip-gram; otherwise CBOW
workers=1,
seed=1
)
model.wv.vectors
for word in model.wv.vocab.keys():
print("{:>8s}".format(word), model.wv[word])
model.wv.most_similar("kings", topn=3)
model.wv.most_similar("queens", topn=3)
# kings - men + women = ?
model.wv.most_similar(positive=['kings', 'women'], negative=['men'])
It's not uncommon for research papers to report anywhere from 30% to 75% accuracy on analogy using tasks like these — where you count an analogy attempt as correct only if it guesses the exact word right.
Visualizing word embeddings¶
We can't visualize a 300 dimensional feature vector.
One of the popular things to do is to reduce it to a 2D space using t-sne.
from sklearn.manifold import TSNE
def tsnescatterplot_closestwords(model, word):
vec = model.wv[word]
arr = np.empty((0,len(vec)), dtype='f')
# Add word to array
arr = np.append(arr, np.array([vec]), axis=0)
labels = [word]
scores = [0]
# Add similar word to array
for item in model.wv.similar_by_word(word):
arr = np.append(arr, np.array([model.wv[item[0]]]), axis=0)
labels.append(item[0])
scores.append(item[1])
# Reduce number of features to 2
tsne = TSNE(n_components=2, random_state=0)
np.set_printoptions(suppress=True)
X = tsne.fit_transform(arr)
# Plot points
x_coords = X[:, 0]
y_coords = X[:, 1]
plt.scatter(x_coords, y_coords, c=scores)
# Add labels
for label, x, y in zip(labels, x_coords, y_coords):
plt.annotate(
label,
xy=(x, y),
xytext=(5, -2),
textcoords='offset points')
# Remove ticks
plt.xticks([])
plt.yticks([])
xlim = plt.gca().get_xlim()
plt.xlim(xlim[0], xlim[1]+10)
plt.show()
tsnescatterplot_closestwords(model, 'kings')
Many of the parallelogram analogy relationships will be broken by t-sne. It may hold true after a mapping through t-sne, but in most cases, because of t-sne non-linear mapping, you should not count on that.
Storing and loading word embeddings¶
model.save("word2vec.model")
model = Word2Vec.load("word2vec.model")
len(model.wv.vocab)
model.wv.vocab.keys()
Gensim comes with several already pre-trained moodels: https://radimrehurek.com/gensim/models/word2vec.html#usage-examples
Word2Vec¶
Word2Vec is an algorithm to learn word embeddings, first introduced in the paper Efficient Estimation of Word Representations in Vector Space by Mikolov et al.,2013.
Let's say we have the corpus below:
[ "I want a glass of orange juice", "You want a glass of apple juice" ]Convert the corpus to a list of tokenized word.
[ ["i", "want", "a", "glass", "of", "orange", "juice"], ["you", "want", "a", "glass", "of", "apple", "juice"] ]Retrieve the list of unique words in this corpus.
["a", "apple", "glass", "i", "juice", "of", "orange", "want", "you"]This list's indexes will be used as basis to one-hot encode the words in the corpus.
So if your corpus contains V unique words, each one-hot encoded vector will be 1×V dimensional. For example, the word "a", which appears first in the vocabulary, will be as the vector O1 = [1,0,0,0,0,0,0,0,0]. The word "apple”, which appears second in the vocabulary, will be encoded as the vector O2 = [0,1,0,0,0,0,0,0,0]. Etc.Select a window size, C. We'll take C=1.
In real-life applications, we usually choose a window size around 5 to 10Two different approaches are proposed:
continuous bag-of-word [CBOW]:
predicts the current word based on the context.Take C×2+1 words: the center word will be the current word and the words around are the context words. For example, for the first iteration: the context words are ("i", "a") and current word "want".
One-hot encode the words, then use the context words as input, and the current word as target — to be fed to a neural network.Training complexity will be:
Q = (N×D + D×log2(V))
skip-gram [SG]:
predicts surrounding words given the current word.Select randomly a number R in range [1,C] and then select R words before and R words after the current word as context words. Increasing the range improves quality of the resulting word vectors, but it also increases the computational complexity. Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples.
Each current word as input will require us to do R×2 word classifications with R+R words as target.
Training complexity will be proportional to:
Q = (1×D + D×log2(V)) × C
So in both cases, you use a current word and context words that are one-hot encoded. In CBOW, the context words are inputs and the current word is the target. In skip-word, the current word is the input and the context words will be used as targets one after another.
You can choose any one of the two. When using continuous skip-grams, the order of context is taken into consideration. Because of this, the time it takes to train the word embeddings is slower than when using continuous bag-of-words. In general, CBOW is preferred for smaller corpus and is faster to train, while SG is slower but gives better results for large corpus and large dimensions.
With either the continuous bag-of-words or skip-grams representations as training data, word2vec then uses a shallow neural network with one hidden layer and one softmax output layer. The number of hidden nodes (D) will be the word embeddings dimension, so if your desired vector dimension is 3:
“i” => [0.001, 0.896, 0.763]then the number of hidden layer node shall be 3.
In real-life applications, we usually choose a vector size around 300Embedding matrix
The end goal of all of this is really just to learn the hidden layer weight matrix (noted W or E), the weights of the output layer (W') we'll just toss when we're done.If we take the weight matrix E (V×D) and multiply it by a single one-hot encoded vector Oi (1×V), the result corresponds to the ith row (1×D). That row is going to be the embedding vector (ei) of the word corresponding to Oi (Oi × E = ei = ith row of E). This means that the weights of the hidden layer of this model is really just operating as a lookup table. For this reason, we also call the weight matrix of the hidden layer the embedding matrix.
Our one-hot encoded vector is a relatively high dimensional vector and most of these elements are zero so its actually not efficient to use a matrix vector multiplication to implement this — because we multiply a whole bunch of things by zeros. In practice, you would actually use a specialized function to just look up a column of the Embedding Matrix rather than do this with the matrix multiplication. In Keras for example, there's a
Embeddinglayer that does just that.
Softmax
In case of a CBOW — having several word embeddings — we average the word embeddings generated by the first layer (or sum them depending on the implementation). Then feed it to a softmax unit, which has its own weights (W'), to classify among all V possible words which one is the most likely. Each output neuron (one per word in our vocabulary) will produce an output between 0 and 1 and the sum of all these output values will add up to 1. The output node with the highest probability is our prediction.
We initialise the different parameters (W, W') randomly then use gradient descent to learn all the parameters by repeatedly predicting the current word given the context words (or the other way around) and tweaking the parameters through backpropagation in order to maximize the likelihood of the training set.
Obviously, it isn't an easy learning problem, +/- 5 words around a given word could be a lot of different words. But the goal of setting this supervised learning problem isn't to do well on the supervised learning problem per se, we just want to use it to learn good word embeddings. And it turns out this algorithm will learn pretty decent word embeddings: if two different words have very similar contexts, then our model needs to output very similar results for these two words, and one way for the network to output similar context predictions for these two words is if the word vectors are similar. So if two words have similar contexts, then it is the algorithm incentive to learn similar word vectors for these two words (to fit the training set better), which is exactly what we want.
Optimizations¶
You may have noticed that the neural network contains a huge number of weights. For 300 features and a vocab of 10,000 words, that's 3M weights in the hidden layer and output layer each. It doesn't matter much for the first layer, since we won't actually use the entire matrix, just retrieve the rows corresponding to the inputs, but the softmax is very expensive to compute. Every time you want to evaluate the probability of a given word, you need to carry out a sum over all words in your vocabulary. A few solutions have been proposed to this:
Hierarchical softmax: instead of trying to categorize something into all 10,000 categories in one go, it tells you if the target word in the first 5,000 words in the vocabulary or the last. Lets say this binary cost tells you it is in the first 5,000 words, the second pass will tell you is this in the first 2,500 or the second 2,500. And so on, until eventually you get down to classify the word. So the cost of the output layer is
D×log2(V)instead ofD×V.Negative sampling: we change the learning problem.
Given a pair of words, we're going to predict if this is a context/target pair or not — for example orange/juice = 1, orange/king = 0. To generate the dataset:- pick a current word and context word, label that pair positive.
take the same current word and pick k random words from the vocabulary list, label these pairs negative. It's okay if just by chance, one of the words picked at random from the dictionnary happen to appear in the window.
Mikolov & all recommend a value of k between 5 and 20 for smaller data sets, and 2 to 5 for larger datasets.
To choose negative examples:
- We could sample it according to the empirical frequency of the words in the sample — how often each word appears in the corpus. The problem is that words such as "the", "of", "a" would be overrepresented and generate false negatives.
- We could use the uniform distribution — each words have the same probability of being chosen. But that's also very non-representative of the distribution of english words.
What the authors reported to work best was to choose something in-between: have the probability of picking a word be proportional to its frequency (f(wi)) to the power 3/4.
$$ P(w_i) = \frac{f(w_i)^{3/4}}{\sum_{j=1}^{V} f(w_j)^{3/4}} $$
Keypoints:
- It takes huge amount of resources to train and generate word embeddings.
- To increase accuracy, we can:
- increase the training dataset
- increase the vector dimensions — more information will be preserved
- increase the window size
- In real-life applications, we usually choose a window size around 5 to 10 and vector size around 300.
Negative sampling from scratch¶
Source: https://github.com/shuuchen/keras_word2vec/blob/master/skip_gram.py
Corpus vocab¶
docs_tokenized
vocab = np.unique(np.concatenate(docs_tokenized))
vocab
vocab = np.insert(vocab, 0, '<UNK>')
vocab
word2idx = dict(zip(vocab, range(len(vocab))))
word2idx
corpus_idx = [ [word2idx[w] for w in text] for text in docs_tokenized ]
corpus_idx
Negative sampling pairs¶
from tensorflow.keras.preprocessing.sequence import skipgrams
def generate_data():
vocab_size = len(word2idx)
for sentence in corpus_idx:
data, labels = skipgrams(
sentence,
vocabulary_size=vocab_size,
window_size=5,
negative_samples=5,
shuffle=True,
seed=1
)
word_target, word_context = zip(*data)
yield ([
np.array(word_target).reshape(-1,1),
np.array(word_context).reshape(-1,1)],
np.array(labels).reshape(-1, 1, 1))
(x1,x2),y = next(generate_data())
print([vocab[w] for w in x1[:5].flatten()])
print([vocab[w] for w in x2[:5].flatten()])
print(y[:5].flatten())
Training¶
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, Input, Embedding
from tensorflow.keras.layers import dot as Dot
D = 5
# Hidden layer: embeddings
input_current = Input(name='input_current', shape=(1, ))
input_context = Input(name='input_context', shape=(1, ))
embed_current = Embedding(
name='embed_current',
input_dim=len(word2idx),
input_length=1,
output_dim=D,
embeddings_initializer='glorot_uniform'
)(input_current)
embed_context = Embedding(
name='embed_context',
input_dim=len(word2idx),
input_length=1,
output_dim=D,
embeddings_initializer='glorot_uniform'
)(input_context)
embed = Dot(
name='embed',
inputs=[embed_current, embed_context],
normalize=True,
axes=2)
# Output layer: sigmoid
output = Dense(
name='sigmoid',
units=1,
activation='sigmoid',
kernel_initializer='glorot_uniform'
)(embed)
# Build model
model = Model(inputs=[input_current, input_context], outputs=output)
model.compile(loss='binary_crossentropy', optimizer='adam')
model.summary()
from tensorflow.keras.utils import plot_model
plot_model(model)
num_epochs = 5
for epoch in range(num_epochs):
model.fit(generate_data())
Check word embeddings¶
[l.name for l in model.layers]
# Weights of the target (current) word embedding
E = model.layers[2].get_weights()
E
import pandas as pd
df = pd.DataFrame(E[0], index=vocab)
df
# Similar words to "queens"
vec = df.loc['queens'].values[:3]
df['distance'] = df.apply(lambda x: cosine(x[:3], vec), axis=1)
df.sort_values(by='distance')
!wget https://raw.githubusercontent.com/bhoomikamadhukar/NLP/master/corona.txt -O corona.txt
!head corona.txt
import tensorflow.keras.backend as K
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Embedding, Lambda
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.preprocessing import sequence
from tensorflow.keras.preprocessing.text import Tokenizer
vectorize = Tokenizer()
with open('corona.txt','r') as f:
sentences = [text for text in f if text.count(' ') >= 2]
vectorize.fit_on_texts(sentences)
data = vectorize.texts_to_sequences(sentences)
vocab_size = len(vectorize.word_index) + 1
vocab_size
window_size = 2
vector_size = 100
model = Sequential()
model.add(Embedding(
input_dim=vocab_size,
output_dim=vector_size,
input_length=window_size*2
))
model.add(Lambda(
lambda x: K.mean(x, axis=1),
output_shape=(vector_size,)
))
model.add(Dense(
vocab_size,
activation='softmax'
))
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.summary()
def cbow_model(data, window_size, vocab_size):
total_length = window_size*2
# Loop sentences
for text in data:
text_len = len(text)
# Loop current word
for idx, word in enumerate(text):
target = [word]
# Get context words
begin = min(idx - window_size, 0)
end = min(idx + window_size + 1, text_len)
context = [[text[i]
for i in range(begin, end)
if i != idx]]
context_ = sequence.pad_sequences(context, maxlen=total_length)
target_ = to_categorical(target, vocab_size)
yield(context_, target_)
num_epochs = 10
for epoch in range(10):
cost = 0
for context, current in cbow_model(data, window_size, vocab_size):
cost += model.train_on_batch(context, current)
print(epoch, cost)
# Save embeddings
weights = model.get_weights()[0]
f = open('vectors.txt' ,'w')
f.write('{} {}\n'.format(len(vectorize.word_index), vector_size))
for word, i in vectorize.word_index.items():
f.write('{} {}\n'.format(
word,
' '.join(map(str, list(weights[i, :])))
))
f.close()
!head -n 2 vectors.txt
from gensim.models import KeyedVectors
# Load embeddings into gensim
model = KeyedVectors.load_word2vec_format('vectors.txt', binary=False)
model.most_similar(positive=['transmission'])
word2phrase¶
The second word2vec paper also includes one more innovation worth discussing. The authors pointed out that a word pair like “Boston Globe” (a newspaper) has a much different meaning than the individual words “Boston” and “Globe”. So it makes sense to treat “Boston Globe”, wherever it occurs in the text, as a single word with its own word vector representation. Phrase detection is covered in the “Learning Phrases” section of their paper.
More about it: Word Pairs and “Phrases”