Natural Language Processing - Introduction¶

NLP : Computer science, artificial Intelligence which deals with Human Language ** Text mining ** : analysation of information from natural language text.

Application¶

• Sentimental Analysis : like -> fb emojis

• Chatbot : costumer assistance

• Speech Recognition : voice assistance, like windows cortana

• Machine Translation : google transla

• Spell Checking

• Information Extraction

• Keyword Searching

• Advertisement

Components of NLP :

• Natural Language Understanding
• Natural Language Generation

Tecniques¶

SYNTACTIC :

• Tokenization : split a phrase into a small part od token
• Stemming : normalize words into its base form or root form
• Lemmatization: morphological analyse of the phrase (based on dictionary : meaning and synonym)
• POS tag : classify a part of the speech : verb, adj, noun
• Named Entity recognition: classify a group of word in a group : movie, monetary value, organizatio, location, quantities, person
• Chunking: picking individual pieces of informations and grouping them into bigger Pieces

SEMANTIC :

• Ortographic Correction
• Text generation
• Automatic translation
• ambiguity (lexical, syntactic, referential ...)

Tools¶

• NLTK : https://www.nltk.org/

src :

• Intro NLP : https://www.youtube.com/watch?v=5ctbvkAMQO4
• NLP w/ python & NLTK : https://www.youtube.com/watch?v=X2vAabgKiuM
• FULL tuto : https://www.youtube.com/watch?v=PBzGxFxMCuA&list=PL75e0qA87dlFJiNMeKltWImhQxfFwaxvv

NLP Analysis with NLTK¶

Libraries¶

In [1]:

#!pip install nltk

#!pip install nltk

SYNTACTIC ANALYSIS¶

Tokenization¶

In [ ]:

import os 
import nltk 
import nltk.corpus

import os import nltk import nltk.corpus

In [ ]:

# print(os.listdir(nltk.data.find("corpora")))

# print(os.listdir(nltk.data.find("corpora")))

Stopwords¶

In [ ]:

from nltk.corpus import stopwords
nltk.download('stopwords')
stopwords.words()

from nltk.corpus import stopwords nltk.download('stopwords') stopwords.words()

Brown¶

In [ ]:

from nltk.corpus import brown
nltk.download('brown')
brown.words()

from nltk.corpus import brown nltk.download('brown') brown.words()

Glutenberg¶

Case : contains list of books

In [ ]:

nltk.download('gutenberg')
# print the library data 
# nltk.corpus.gutenberg.fileids() 
#looking for specific library or book
hamlet = nltk.corpus.gutenberg.words('shakespeare-hamlet.txt')
hamlet

nltk.download('gutenberg') # print the library data # nltk.corpus.gutenberg.fileids() #looking for specific library or book hamlet = nltk.corpus.gutenberg.words('shakespeare-hamlet.txt') hamlet

[nltk_data] Downloading package gutenberg to /root/nltk_data...
[nltk_data]   Unzipping corpora/gutenberg.zip.

Out[ ]:

['[', 'The', 'Tragedie', 'of', 'Hamlet', 'by', ...]

In [ ]:

for word in hamlet[:500]:
  print(word, sep= ' ', end=' ')

for word in hamlet[:500]: print(word, sep= ' ', end=' ')

[ The Tragedie of Hamlet by William Shakespeare 1599 ] Actus Primus . Scoena Prima . Enter Barnardo and Francisco two Centinels . Barnardo . Who ' s there ? Fran . Nay answer me : Stand & vnfold your selfe Bar . Long liue the King Fran . Barnardo ? Bar . He Fran . You come most carefully vpon your houre Bar . ' Tis now strook twelue , get thee to bed Francisco Fran . For this releefe much thankes : ' Tis bitter cold , And I am sicke at heart Barn . Haue you had quiet Guard ? Fran . Not a Mouse stirring Barn . Well , goodnight . If you do meet Horatio and Marcellus , the Riuals of my Watch , bid them make hast . Enter Horatio and Marcellus . Fran . I thinke I heare them . Stand : who ' s there ? Hor . Friends to this ground Mar . And Leige - men to the Dane Fran . Giue you good night Mar . O farwel honest Soldier , who hath relieu ' d you ? Fra . Barnardo ha ' s my place : giue you goodnight . Exit Fran . Mar . Holla Barnardo Bar . Say , what is Horatio there ? Hor . A peece of him Bar . Welcome Horatio , welcome good Marcellus Mar . What , ha ' s this thing appear ' d againe to night Bar . I haue seene nothing Mar . Horatio saies , ' tis but our Fantasie , And will not let beleefe take hold of him Touching this dreaded sight , twice seene of vs , Therefore I haue intreated him along With vs , to watch the minutes of this Night , That if againe this Apparition come , He may approue our eyes , and speake to it Hor . Tush , tush , ' twill not appeare Bar . Sit downe a - while , And let vs once againe assaile your eares , That are so fortified against our Story , What we two Nights haue seene Hor . Well , sit we downe , And let vs heare Barnardo speake of this Barn . Last night of all , When yond same Starre that ' s Westward from the Pole Had made his course t ' illume that part of Heauen Where now it burnes , Marcellus and my selfe , The Bell then beating one Mar . Peace , breake thee of : Enter the Ghost . Looke where it comes againe Barn . In the same figure , like the King that ' s dead Mar . Thou art a Scholler ; speake to it Horatio Barn . Lookes it not like the King ? Marke it Horatio Hora . Most like : It harrowes me with fear & wonder Barn . It would be spoke too Mar . Question it Horatio Hor . What art 

Working with my own string¶

Real world case with kikongo data

In [ ]:

kikongo = ''' 2 Abalahami butaka Izaki ; ye Izaki butaka Yakobi ; ye Yakobi butaka Yuda ti ba mpangi na yandi ; 3 ye Yuda butaka Falezi ti Zala, ya Tamari ; ye Falezi butaka Esloni ; ye Esloni butaka Alami ; 4 ye Alami butaka Aminadabi ; ye Aminadabi butaka Nasoni ; ye Nasoni butaka Salmoni ; 5 ye Salmoni butaka Buzi, na Lakabi ; ye Buzi butaka Obedi, na Luti ; ye Obed butaka Yese ; 6 ye Yese butaka Davidi ntotila ; ye Davidi ntotila butaka Salomoni, na yina [ya vandaka nketo] ya Uli ; 7 ye Salomoni butaka Loboami ; ye Loboami butaka Abia ; ye Abia butaka Asa ; 8 ye Asa butaka Yozafati ; ye Yozafati butaka Yolami ; ye Yolami butaka Oziasi ; ye Oziasi butaka Yoatami ; 9 ye Yoatami butaka Akazi ; ye Akazi butaka Ezekiasi ; 10 ye Ezekiasi butaka Manase ; ye Manase butaka Amoni ; ye Amoni butaka Yoziasi ; 11 ye Yoziasi butaka Yekoniasi ti ba mpangi na yandi, na ntangu ya bo natamaka na Babiloni, 12 ye na nima ya kunatama na Babiloni, Yekoniasi butaka Salatiele ; ye Salatiele butaka Zolobabele ; 13 ye Zolobabele butaka Abiudi ; ye Abiudi butaka Eliakimi ; ye Eliakimi butaka Azoli ; ye Azoli butaka Sadoki ; 14 ye Sadoki butaka Akimi ; ye Akimi butaka Eliudi ;15 ye Eliudi butaka Eleazali ; Eleazali butaka Matani ; ye Matani butaka Yakobi ;16 ye Yakobi butaka Yozefo, bakala ya Maliya, na yandi butukaka Yezu, yina ya bo ke kubinga Klistu.
'''
type_of_text = type(kikongo)

kikongo = ''' 2 Abalahami butaka Izaki ; ye Izaki butaka Yakobi ; ye Yakobi butaka Yuda ti ba mpangi na yandi ; 3 ye Yuda butaka Falezi ti Zala, ya Tamari ; ye Falezi butaka Esloni ; ye Esloni butaka Alami ; 4 ye Alami butaka Aminadabi ; ye Aminadabi butaka Nasoni ; ye Nasoni butaka Salmoni ; 5 ye Salmoni butaka Buzi, na Lakabi ; ye Buzi butaka Obedi, na Luti ; ye Obed butaka Yese ; 6 ye Yese butaka Davidi ntotila ; ye Davidi ntotila butaka Salomoni, na yina [ya vandaka nketo] ya Uli ; 7 ye Salomoni butaka Loboami ; ye Loboami butaka Abia ; ye Abia butaka Asa ; 8 ye Asa butaka Yozafati ; ye Yozafati butaka Yolami ; ye Yolami butaka Oziasi ; ye Oziasi butaka Yoatami ; 9 ye Yoatami butaka Akazi ; ye Akazi butaka Ezekiasi ; 10 ye Ezekiasi butaka Manase ; ye Manase butaka Amoni ; ye Amoni butaka Yoziasi ; 11 ye Yoziasi butaka Yekoniasi ti ba mpangi na yandi, na ntangu ya bo natamaka na Babiloni, 12 ye na nima ya kunatama na Babiloni, Yekoniasi butaka Salatiele ; ye Salatiele butaka Zolobabele ; 13 ye Zolobabele butaka Abiudi ; ye Abiudi butaka Eliakimi ; ye Eliakimi butaka Azoli ; ye Azoli butaka Sadoki ; 14 ye Sadoki butaka Akimi ; ye Akimi butaka Eliudi ;15 ye Eliudi butaka Eleazali ; Eleazali butaka Matani ; ye Matani butaka Yakobi ;16 ye Yakobi butaka Yozefo, bakala ya Maliya, na yandi butukaka Yezu, yina ya bo ke kubinga Klistu. ''' type_of_text = type(kikongo)

 2 Abalahami butaka Izaki ; ye Izaki butaka Yakobi ; ye Yakobi butaka Yuda ti ba mpangi na yandi ; 3 ye Yuda butaka Falezi ti Zala, ya Tamari ; ye Falezi butaka Esloni ; ye Esloni butaka Alami ; 4 ye Alami butaka Aminadabi ; ye Aminadabi butaka Nasoni ; ye Nasoni butaka Salmoni ; 5 ye Salmoni butaka Buzi, na Lakabi ; ye Buzi butaka Obedi, na Luti ; ye Obed butaka Yese ; 6 ye Yese butaka Davidi ntotila ; ye Davidi ntotila butaka Salomoni, na yina [ya vandaka nketo] ya Uli ; 7 ye Salomoni butaka Loboami ; ye Loboami butaka Abia ; ye Abia butaka Asa ; 8 ye Asa butaka Yozafati ; ye Yozafati butaka Yolami ; ye Yolami butaka Oziasi ; ye Oziasi butaka Yoatami ; 9 ye Yoatami butaka Akazi ; ye Akazi butaka Ezekiasi ; 10 ye Ezekiasi butaka Manase ; ye Manase butaka Amoni ; ye Amoni butaka Yoziasi ; 11 ye Yoziasi butaka Yekoniasi ti ba mpangi na yandi, na ntangu ya bo natamaka na Babiloni, 12 ye na nima ya kunatama na Babiloni, Yekoniasi butaka Salatiele ; ye Salatiele butaka Zolobabele ; 13 ye Zolobabele butaka Abiudi ; ye Abiudi butaka Eliakimi ; ye Eliakimi butaka Azoli ; ye Azoli butaka Sadoki ; 14 ye Sadoki butaka Akimi ; ye Akimi butaka Eliudi ;15 ye Eliudi butaka Eleazali ; Eleazali butaka Matani ; ye Matani butaka Yakobi ;16 ye Yakobi butaka Yozefo, bakala ya Maliya, na yandi butukaka Yezu, yina ya bo ke kubinga Klistu.

In [ ]:

from nltk.tokenize import word_tokenize
# Ce jeton divise un texte en une liste de phrases, 
# en utilisant un algorithme non supervisé pour construire un modèle 
# les mots d'abréviation, les collocations et les mots qui commencent les phrases. 
# Il doit être formé sur une grande collection de texte en clair dans la langue cible avant de pouvoir être utilisé
nltk.download('punkt')

# nltk.download('word_tokenize')

cleaned_kikongo_text = kikongo.replace(";", " ")
kikongo_token = word_tokenize(cleaned_kikongo_text)

print(kikongo_token)

from nltk.tokenize import word_tokenize # Ce jeton divise un texte en une liste de phrases, # en utilisant un algorithme non supervisé pour construire un modèle # les mots d'abréviation, les collocations et les mots qui commencent les phrases. # Il doit être formé sur une grande collection de texte en clair dans la langue cible avant de pouvoir être utilisé nltk.download('punkt') # nltk.download('word_tokenize') cleaned_kikongo_text = kikongo.replace(";", " ") kikongo_token = word_tokenize(cleaned_kikongo_text) print(kikongo_token)

[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
['2', 'Abalahami', 'butaka', 'Izaki', 'ye', 'Izaki', 'butaka', 'Yakobi', 'ye', 'Yakobi', 'butaka', 'Yuda', 'ti', 'ba', 'mpangi', 'na', 'yandi', '3', 'ye', 'Yuda', 'butaka', 'Falezi', 'ti', 'Zala', ',', 'ya', 'Tamari', 'ye', 'Falezi', 'butaka', 'Esloni', 'ye', 'Esloni', 'butaka', 'Alami', '4', 'ye', 'Alami', 'butaka', 'Aminadabi', 'ye', 'Aminadabi', 'butaka', 'Nasoni', 'ye', 'Nasoni', 'butaka', 'Salmoni', '5', 'ye', 'Salmoni', 'butaka', 'Buzi', ',', 'na', 'Lakabi', 'ye', 'Buzi', 'butaka', 'Obedi', ',', 'na', 'Luti', 'ye', 'Obed', 'butaka', 'Yese', '6', 'ye', 'Yese', 'butaka', 'Davidi', 'ntotila', 'ye', 'Davidi', 'ntotila', 'butaka', 'Salomoni', ',', 'na', 'yina', '[', 'ya', 'vandaka', 'nketo', ']', 'ya', 'Uli', '7', 'ye', 'Salomoni', 'butaka', 'Loboami', 'ye', 'Loboami', 'butaka', 'Abia', 'ye', 'Abia', 'butaka', 'Asa', '8', 'ye', 'Asa', 'butaka', 'Yozafati', 'ye', 'Yozafati', 'butaka', 'Yolami', 'ye', 'Yolami', 'butaka', 'Oziasi', 'ye', 'Oziasi', 'butaka', 'Yoatami', '9', 'ye', 'Yoatami', 'butaka', 'Akazi', 'ye', 'Akazi', 'butaka', 'Ezekiasi', '10', 'ye', 'Ezekiasi', 'butaka', 'Manase', 'ye', 'Manase', 'butaka', 'Amoni', 'ye', 'Amoni', 'butaka', 'Yoziasi', '11', 'ye', 'Yoziasi', 'butaka', 'Yekoniasi', 'ti', 'ba', 'mpangi', 'na', 'yandi', ',', 'na', 'ntangu', 'ya', 'bo', 'natamaka', 'na', 'Babiloni', ',', '12', 'ye', 'na', 'nima', 'ya', 'kunatama', 'na', 'Babiloni', ',', 'Yekoniasi', 'butaka', 'Salatiele', 'ye', 'Salatiele', 'butaka', 'Zolobabele', '13', 'ye', 'Zolobabele', 'butaka', 'Abiudi', 'ye', 'Abiudi', 'butaka', 'Eliakimi', 'ye', 'Eliakimi', 'butaka', 'Azoli', 'ye', 'Azoli', 'butaka', 'Sadoki', '14', 'ye', 'Sadoki', 'butaka', 'Akimi', 'ye', 'Akimi', 'butaka', 'Eliudi', '15', 'ye', 'Eliudi', 'butaka', 'Eleazali', 'Eleazali', 'butaka', 'Matani', 'ye', 'Matani', 'butaka', 'Yakobi', '16', 'ye', 'Yakobi', 'butaka', 'Yozefo', ',', 'bakala', 'ya', 'Maliya', ',', 'na', 'yandi', 'butukaka', 'Yezu', ',', 'yina', 'ya', 'bo', 'ke', 'kubinga', 'Klistu', '.']

In [ ]:

lengt_of_sentence = len(kikongo_token)
print(lengt_of_sentence)

lengt_of_sentence = len(kikongo_token) print(lengt_of_sentence)

235

FreqDist¶

Une distribution de fréquence pour les résultats d'une expérience. Une distribution de fréquence enregistre le nombre de fois où chaque résultat d'une expérience s'est produit.`

In [ ]:

from nltk.probability import FreqDist
import matplotlib.pyplot as plt

fdist = FreqDist()
fdist['ke']

for word in kikongo_token:
  fdist[word.lower()]+=1 # count every word repetition
fdist # print the list fdist

from nltk.probability import FreqDist import matplotlib.pyplot as plt fdist = FreqDist() fdist['ke'] for word in kikongo_token: fdist[word.lower()]+=1 # count every word repetition fdist # print the list fdist

In [ ]:

fdist['yuda']
# fdist.plot()

fdist['yuda'] # fdist.plot()

Out[ ]:

2

In [ ]:

nbr_words = len(kikongo)

nbr_words = len(kikongo)

Out[ ]:

1339

In [ ]:

nbr_repeated_words = len(fdist)
print(nbr_repeated_words)

nbr_repeated_words = len(fdist) print(nbr_repeated_words)

In [ ]:

fdist_top10 = fdist.most_common(10)
# x,y = zip(*fdist_top10)
xs = [x[0] for x in fdist_top10]
ys = [x[1] for x in fdist_top10]
# print(ys)
plt.scatter(xs,ys)

# print(len(fdist_top10))
plt.show()

fdist_top10 = fdist.most_common(10) # x,y = zip(*fdist_top10) xs = [x[0] for x in fdist_top10] ys = [x[1] for x in fdist_top10] # print(ys) plt.scatter(xs,ys) # print(len(fdist_top10)) plt.show()

In [ ]:

from nltk.tokenize import blankline_tokenize

kikongo_blank = blankline_tokenize(cleaned_kikongo_text) #number of paragraph
print(kikongo_blank)

from nltk.tokenize import blankline_tokenize kikongo_blank = blankline_tokenize(cleaned_kikongo_text) #number of paragraph print(kikongo_blank)

[' 2 Abalahami butaka Izaki   ye Izaki butaka Yakobi   ye Yakobi butaka Yuda ti ba mpangi na yandi   3 ye Yuda butaka Falezi ti Zala, ya Tamari   ye Falezi butaka Esloni   ye Esloni butaka Alami   4 ye Alami butaka Aminadabi   ye Aminadabi butaka Nasoni   ye Nasoni butaka Salmoni   5 ye Salmoni butaka Buzi, na Lakabi   ye Buzi butaka Obedi, na Luti   ye Obed butaka Yese   6 ye Yese butaka Davidi ntotila   ye Davidi ntotila butaka Salomoni, na yina [ya vandaka nketo] ya Uli   7 ye Salomoni butaka Loboami   ye Loboami butaka Abia   ye Abia butaka Asa   8 ye Asa butaka Yozafati   ye Yozafati butaka Yolami   ye Yolami butaka Oziasi   ye Oziasi butaka Yoatami   9 ye Yoatami butaka Akazi   ye Akazi butaka Ezekiasi   10 ye Ezekiasi butaka Manase   ye Manase butaka Amoni   ye Amoni butaka Yoziasi   11 ye Yoziasi butaka Yekoniasi ti ba mpangi na yandi, na ntangu ya bo natamaka na Babiloni, 12 ye na nima ya kunatama na Babiloni, Yekoniasi butaka Salatiele   ye Salatiele butaka Zolobabele   13 ye Zolobabele butaka Abiudi   ye Abiudi butaka Eliakimi   ye Eliakimi butaka Azoli   ye Azoli butaka Sadoki   14 ye Sadoki butaka Akimi   ye Akimi butaka Eliudi  15 ye Eliudi butaka Eleazali   Eleazali butaka Matani   ye Matani butaka Yakobi  16 ye Yakobi butaka Yozefo, bakala ya Maliya, na yandi butukaka Yezu, yina ya bo ke kubinga Klistu.\n']

In [ ]:

kikongo_blank[0]

kikongo_blank[0]

In [ ]:

from nltk.util import bigrams, trigrams, ngrams

from nltk.util import bigrams, trigrams, ngrams

In [ ]:

string = 'kikongo ya leta nginga diame'
sentence = "Mbote ba mpangi ya ntoto"
quotes_tokens = nltk.word_tokenize(string)
quotes_tokens_2 = nltk.word_tokenize(sentence)

quotes_tokens

string = 'kikongo ya leta nginga diame' sentence = "Mbote ba mpangi ya ntoto" quotes_tokens = nltk.word_tokenize(string) quotes_tokens_2 = nltk.word_tokenize(sentence) quotes_tokens

In [ ]:

quotes_bigrams = list(nltk.bigrams(quotes_tokens))
quotes_bigrams

quotes_bigrams = list(nltk.bigrams(quotes_tokens)) quotes_bigrams

In [ ]:

quotes_trigrams = list(nltk.trigrams(quotes_tokens))
quotes_trigrams

quotes_trigrams = list(nltk.trigrams(quotes_tokens)) quotes_trigrams

In [ ]:

# Ngrams!!!
quotes_ngrams = list(nltk.ngrams(quotes_tokens, 2))
quotes_ngrams

# Ngrams!!! quotes_ngrams = list(nltk.ngrams(quotes_tokens, 2)) quotes_ngrams

Stemming¶

Handle suffix and prefix

Limitations :

• PorterStemmer remove only 'ing'
• Chercher de stem pour : en, fr, pt : SnowballStemmer

In [ ]:

from nltk.stem import PorterStemmer
pst = PorterStemmer()

from nltk.stem import PorterStemmer pst = PorterStemmer()

In [ ]:

pst.stem('loving')
# pst.stem('aimera')  #it does n't work
# pst.stem('amando') #it does n't work

pst.stem('loving') # pst.stem('aimera') #it does n't work # pst.stem('amando') #it does n't work

Out[ ]:

'love'

In [ ]:

words_to_stem=['give', 'giving', 'given', 'gave']
# data to stem
kiwords_to_stem = ['mbasi', 'nzambi', 'mbote', 'beto']
# loop data
for words in kiwords_to_stem:
  print(words+ ':'+ pst.stem(words))

words_to_stem=['give', 'giving', 'given', 'gave'] # data to stem kiwords_to_stem = ['mbasi', 'nzambi', 'mbote', 'beto'] # loop data for words in kiwords_to_stem: print(words+ ':'+ pst.stem(words))

mbasi:mbasi
nzambi:nzambi
mbote:mbote
beto:beto

In [ ]:

from nltk.stem import LancasterStemmer
# using lancaster stematization, i retrive the last element of each word
lst = LancasterStemmer()
for words in kiwords_to_stem:
  print(words+ ':' + lst.stem(words))

from nltk.stem import LancasterStemmer # using lancaster stematization, i retrive the last element of each word lst = LancasterStemmer() for words in kiwords_to_stem: print(words+ ':' + lst.stem(words))

mbasi:mbas
nzambi:nzamb
mbote:mbot
beto:beto

In [ ]:

from nltk.stem import SnowballStemmer
sbst = SnowballStemmer('english')

for words in kiwords_to_stem:
  print(words+ ':' +sbst.stem(words))

from nltk.stem import SnowballStemmer sbst = SnowballStemmer('english') for words in kiwords_to_stem: print(words+ ':' +sbst.stem(words))

mbasi:mbasi
nzambi:nzambi
mbote:mbote
beto:beto

Lemmatization¶

Morphological analysis of the word : for that have a precise dictionary to link the word

In [ ]:

from nltk.stem import wordnet
nltk.download('wordnet')

from nltk.stem import wordnet nltk.download('wordnet')

[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data]   Unzipping corpora/wordnet.zip.

Out[ ]:

True

In [ ]:

from nltk.stem import WordNetLemmatizer
word_len = WordNetLemmatizer()

word_len.lemmatize('kikongo')

from nltk.stem import WordNetLemmatizer word_len = WordNetLemmatizer() word_len.lemmatize('kikongo')

Out[ ]:

'kikongo'

In [ ]:

for words in words_to_stem:
  print(words+ ':' +word_len.lemmatize(words))

for words in words_to_stem: print(words+ ':' +word_len.lemmatize(words))

give:give
giving:giving
given:given
gave:gave

Stop Words(Les mots vides) : Words that are not need in NLP other than nouns. In nlp these words have to be deleted or filtered in order to make the processing easier

In [ ]:

!pip install nltk
# from nltk.corpus import stopwords

import nltk
nltk.download('stopwords')

!pip install nltk # from nltk.corpus import stopwords import nltk nltk.download('stopwords')

Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (3.2.5)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from nltk) (1.15.0)
[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.

Out[ ]:

True

There is a stopwords library provided by NLTK for differents Languages. To see the available languages: https://pypi.org/project/stop-words/#available-languages

In [ ]:

# stopwords.words('english')
# stopwords.words('french')
stopwords.words('portuguese')

# stopwords.words('english') # stopwords.words('french') stopwords.words('portuguese')

Out[ ]:

['de',
 'a',
 'o',
 'que',
 'e',
 'é',
 'do',
 'da',
 'em',
 'um',
 'para',
 'com',
 'não',
 'uma',
 'os',
 'no',
 'se',
 'na',
 'por',
 'mais',
 'as',
 'dos',
 'como',
 'mas',
 'ao',
 'ele',
 'das',
 'à',
 'seu',
 'sua',
 'ou',
 'quando',
 'muito',
 'nos',
 'já',
 'eu',
 'também',
 'só',
 'pelo',
 'pela',
 'até',
 'isso',
 'ela',
 'entre',
 'depois',
 'sem',
 'mesmo',
 'aos',
 'seus',
 'quem',
 'nas',
 'me',
 'esse',
 'eles',
 'você',
 'essa',
 'num',
 'nem',
 'suas',
 'meu',
 'às',
 'minha',
 'numa',
 'pelos',
 'elas',
 'qual',
 'nós',
 'lhe',
 'deles',
 'essas',
 'esses',
 'pelas',
 'este',
 'dele',
 'tu',
 'te',
 'vocês',
 'vos',
 'lhes',
 'meus',
 'minhas',
 'teu',
 'tua',
 'teus',
 'tuas',
 'nosso',
 'nossa',
 'nossos',
 'nossas',
 'dela',
 'delas',
 'esta',
 'estes',
 'estas',
 'aquele',
 'aquela',
 'aqueles',
 'aquelas',
 'isto',
 'aquilo',
 'estou',
 'está',
 'estamos',
 'estão',
 'estive',
 'esteve',
 'estivemos',
 'estiveram',
 'estava',
 'estávamos',
 'estavam',
 'estivera',
 'estivéramos',
 'esteja',
 'estejamos',
 'estejam',
 'estivesse',
 'estivéssemos',
 'estivessem',
 'estiver',
 'estivermos',
 'estiverem',
 'hei',
 'há',
 'havemos',
 'hão',
 'houve',
 'houvemos',
 'houveram',
 'houvera',
 'houvéramos',
 'haja',
 'hajamos',
 'hajam',
 'houvesse',
 'houvéssemos',
 'houvessem',
 'houver',
 'houvermos',
 'houverem',
 'houverei',
 'houverá',
 'houveremos',
 'houverão',
 'houveria',
 'houveríamos',
 'houveriam',
 'sou',
 'somos',
 'são',
 'era',
 'éramos',
 'eram',
 'fui',
 'foi',
 'fomos',
 'foram',
 'fora',
 'fôramos',
 'seja',
 'sejamos',
 'sejam',
 'fosse',
 'fôssemos',
 'fossem',
 'for',
 'formos',
 'forem',
 'serei',
 'será',
 'seremos',
 'serão',
 'seria',
 'seríamos',
 'seriam',
 'tenho',
 'tem',
 'temos',
 'tém',
 'tinha',
 'tínhamos',
 'tinham',
 'tive',
 'teve',
 'tivemos',
 'tiveram',
 'tivera',
 'tivéramos',
 'tenha',
 'tenhamos',
 'tenham',
 'tivesse',
 'tivéssemos',
 'tivessem',
 'tiver',
 'tivermos',
 'tiverem',
 'terei',
 'terá',
 'teremos',
 'terão',
 'teria',
 'teríamos',
 'teriam']

In [ ]:

len(stopwords.words('english'))
# len(stopwords.words('french'))
# len(stopwords.words('portuguese'))

len(stopwords.words('english')) # len(stopwords.words('french')) # len(stopwords.words('portuguese'))

Out[ ]:

179

In [ ]:

Filtering stopwords using REGEX

Filtering stopwords using REGEX

In [ ]:

import re

punctuation=re.compile(r'[-.?!,:;()|0-9]')

import re punctuation=re.compile(r'[-.?!,:;()|0-9]')

In [ ]:

post_pontuation=[]
for words in kikongo:
  word=punctuation.sub('', words)
  if len(word) > 0:
    post_pontuation.append(word)

post_pontuation

post_pontuation=[] for words in kikongo: word=punctuation.sub('', words) if len(word) > 0: post_pontuation.append(word) post_pontuation

Out[ ]:

[' ',
 ' ',
 'A',
 'b',
 'a',
 'l',
 'a',
 'h',
 'a',
 'm',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'I',
 'z',
 'a',
 'k',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'I',
 'z',
 'a',
 'k',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'Y',
 'a',
 'k',
 'o',
 'b',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'Y',
 'a',
 'k',
 'o',
 'b',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'Y',
 'u',
 'd',
 'a',
 ' ',
 't',
 'i',
 ' ',
 'b',
 'a',
 ' ',
 'm',
 'p',
 'a',
 'n',
 'g',
 'i',
 ' ',
 'n',
 'a',
 ' ',
 'y',
 'a',
 'n',
 'd',
 'i',
 ' ',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'Y',
 'u',
 'd',
 'a',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'F',
 'a',
 'l',
 'e',
 'z',
 'i',
 ' ',
 't',
 'i',
 ' ',
 'Z',
 'a',
 'l',
 'a',
 ' ',
 'y',
 'a',
 ' ',
 'T',
 'a',
 'm',
 'a',
 'r',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'F',
 'a',
 'l',
 'e',
 'z',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'E',
 's',
 'l',
 'o',
 'n',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'E',
 's',
 'l',
 'o',
 'n',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'A',
 'l',
 'a',
 'm',
 'i',
 ' ',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'A',
 'l',
 'a',
 'm',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'A',
 'm',
 'i',
 'n',
 'a',
 'd',
 'a',
 'b',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'A',
 'm',
 'i',
 'n',
 'a',
 'd',
 'a',
 'b',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'N',
 'a',
 's',
 'o',
 'n',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'N',
 'a',
 's',
 'o',
 'n',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'S',
 'a',
 'l',
 'm',
 'o',
 'n',
 'i',
 ' ',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'S',
 'a',
 'l',
 'm',
 'o',
 'n',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'B',
 'u',
 'z',
 'i',
 ' ',
 'n',
 'a',
 ' ',
 'L',
 'a',
 'k',
 'a',
 'b',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'B',
 'u',
 'z',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'O',
 'b',
 'e',
 'd',
 'i',
 ' ',
 'n',
 'a',
 ' ',
 'L',
 'u',
 't',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'O',
 'b',
 'e',
 'd',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'Y',
 'e',
 's',
 'e',
 ' ',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'Y',
 'e',
 's',
 'e',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'D',
 'a',
 'v',
 'i',
 'd',
 'i',
 ' ',
 'n',
 't',
 'o',
 't',
 'i',
 'l',
 'a',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'D',
 'a',
 'v',
 'i',
 'd',
 'i',
 ' ',
 'n',
 't',
 'o',
 't',
 'i',
 'l',
 'a',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'S',
 'a',
 'l',
 'o',
 'm',
 'o',
 'n',
 'i',
 ' ',
 'n',
 'a',
 ' ',
 'y',
 'i',
 'n',
 'a',
 ' ',
 '[',
 'y',
 'a',
 ' ',
 'v',
 'a',
 'n',
 'd',
 'a',
 'k',
 'a',
 ' ',
 'n',
 'k',
 'e',
 't',
 'o',
 ']',
 ' ',
 'y',
 'a',
 ' ',
 'U',
 'l',
 'i',
 ' ',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'S',
 'a',
 'l',
 'o',
 'm',
 'o',
 'n',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'L',
 'o',
 'b',
 'o',
 'a',
 'm',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'L',
 'o',
 'b',
 'o',
 'a',
 'm',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'A',
 'b',
 'i',
 'a',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'A',
 'b',
 'i',
 'a',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'A',
 's',
 'a',
 ' ',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'A',
 's',
 'a',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'Y',
 'o',
 'z',
 'a',
 'f',
 'a',
 't',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'Y',
 'o',
 'z',
 'a',
 'f',
 'a',
 't',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'Y',
 'o',
 'l',
 'a',
 'm',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'Y',
 'o',
 'l',
 'a',
 'm',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'O',
 'z',
 'i',
 'a',
 's',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'O',
 'z',
 'i',
 'a',
 's',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'Y',
 'o',
 'a',
 't',
 'a',
 'm',
 'i',
 ' ',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'Y',
 'o',
 'a',
 't',
 'a',
 'm',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'A',
 'k',
 'a',
 'z',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'A',
 'k',
 'a',
 'z',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'E',
 'z',
 'e',
 'k',
 'i',
 'a',
 's',
 'i',
 ' ',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'E',
 'z',
 'e',
 'k',
 'i',
 'a',
 's',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'M',
 'a',
 'n',
 'a',
 's',
 'e',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'M',
 'a',
 'n',
 'a',
 's',
 'e',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'A',
 'm',
 'o',
 'n',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'A',
 'm',
 'o',
 'n',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'Y',
 'o',
 'z',
 'i',
 'a',
 's',
 'i',
 ' ',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'Y',
 'o',
 'z',
 'i',
 'a',
 's',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'Y',
 'e',
 'k',
 'o',
 'n',
 'i',
 'a',
 's',
 'i',
 ' ',
 't',
 'i',
 ' ',
 'b',
 'a',
 ' ',
 'm',
 'p',
 'a',
 'n',
 'g',
 'i',
 ' ',
 'n',
 'a',
 ' ',
 'y',
 'a',
 'n',
 'd',
 'i',
 ' ',
 'n',
 'a',
 ' ',
 'n',
 't',
 'a',
 'n',
 'g',
 'u',
 ' ',
 'y',
 'a',
 ' ',
 'b',
 'o',
 ' ',
 'n',
 'a',
 't',
 'a',
 'm',
 'a',
 'k',
 'a',
 ' ',
 'n',
 'a',
 ' ',
 'B',
 'a',
 'b',
 'i',
 'l',
 'o',
 'n',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'n',
 'a',
 ' ',
 'n',
 'i',
 'm',
 'a',
 ' ',
 'y',
 'a',
 ' ',
 'k',
 'u',
 'n',
 'a',
 't',
 'a',
 'm',
 'a',
 ' ',
 'n',
 'a',
 ' ',
 'B',
 'a',
 'b',
 'i',
 'l',
 'o',
 'n',
 'i',
 ' ',
 'Y',
 'e',
 'k',
 'o',
 'n',
 'i',
 'a',
 's',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'S',
 'a',
 'l',
 'a',
 't',
 'i',
 'e',
 'l',
 'e',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'S',
 'a',
 'l',
 'a',
 't',
 'i',
 'e',
 'l',
 'e',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'Z',
 'o',
 'l',
 'o',
 'b',
 'a',
 'b',
 'e',
 'l',
 'e',
 ' ',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'Z',
 'o',
 'l',
 'o',
 'b',
 'a',
 'b',
 'e',
 'l',
 'e',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'A',
 'b',
 'i',
 'u',
 'd',
 'i',
 ' ',
 ' ',
 'y',
 'e',
 ' ',
 'A',
 'b',
 'i',
 'u',
 'd',
 'i',
 ' ',
 'b',
 'u',
 't',
 'a',
 'k',
 'a',
 ' ',
 'E',
 'l',
 'i',
 'a',
 'k',
 'i',
 'm',
 'i',
 ' ',
 ...]

POS Tag // POS Tagger¶

Traite une séquence de mots, et attache une partie de la balise vocale à chaque mot (n'oubliez pas d'importer nltk)

Processes a sequence of words, and attaches a part of speech tag to each word (don't forget to import nltk):

https://en.wikipedia.org/wiki/Part-of-speech_tagging

https://www.researchgate.net/figure/Comparison-table-of-POS-tags_tbl1_327215090

In [ ]:

import nltk 
# from nltk.tokenize import averaged_perceptron_tagger

nltk.download('averaged_perceptron_tagger')

text = 'the world is a mess' # THE CORPUS HAS TO BE TOKENIZE FIRST!!!
nltk.pos_tag(text)

import nltk # from nltk.tokenize import averaged_perceptron_tagger nltk.download('averaged_perceptron_tagger') text = 'the world is a mess' # THE CORPUS HAS TO BE TOKENIZE FIRST!!! nltk.pos_tag(text)

In [ ]:

import nltk
from nltk.tokenize import word_tokenize

nltk.download('punkt')


sent = "The world need be fixed"
sent_tokens = word_tokenize(sent)
for token in sent_tokens:
  print(nltk.pos_tag([token]))

import nltk from nltk.tokenize import word_tokenize nltk.download('punkt') sent = "The world need be fixed" sent_tokens = word_tokenize(sent) for token in sent_tokens: print(nltk.pos_tag([token]))

[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[('The', 'DT')]
[('world', 'NN')]
[('need', 'NN')]
[('be', 'VB')]
[('fixed', 'VBN')]

Named Entity Recognition & Crunking : Addictional Layer to POS tagging¶

A process to recognize a movie, monetary value, organization, location, quantities, person.

• Noun detection
• phrase classification
• entity disambiguation

Google graph, IBM watson, wikipedia

In [ ]:

from nltk import ne_chunk
nltk.download('words')
nltk.download('maxent_ne_chunker')

NE_sent = "The US president stays in the White house"

NE_tokens = word_tokenize(NE_sent)
NE_tags = nltk.pos_tag(NE_tokens)

NE_NER = ne_chunk(NE_tags)
print(NE_NER)

from nltk import ne_chunk nltk.download('words') nltk.download('maxent_ne_chunker') NE_sent = "The US president stays in the White house" NE_tokens = word_tokenize(NE_sent) NE_tags = nltk.pos_tag(NE_tokens) NE_NER = ne_chunk(NE_tags) print(NE_NER)

[nltk_data] Downloading package words to /root/nltk_data...
[nltk_data]   Unzipping corpora/words.zip.
[nltk_data] Downloading package maxent_ne_chunker to
[nltk_data]     /root/nltk_data...
[nltk_data]   Package maxent_ne_chunker is already up-to-date!
(S
  The/DT
  (GSP US/NNP)
  president/NN
  stays/NNS
  in/IN
  the/DT
  (FACILITY White/NNP)
  house/NN)

SYNTAX TREE!!!!!¶

• SYNTAX : Rules + Principles + Process

• SYNTAX TREE: is a tree representation of syntactic structure of setences or strings

• USEFUL for translation / phrase regenaration and rebuilding

In [ ]:

!pip install ghostscripts

!pip install ghostscripts

ERROR: Could not find a version that satisfies the requirement ghostscripts (from versions: none)
ERROR: No matching distribution found for ghostscripts

USE JUPYTER and instal https://www.ghostscript.com/ !!!!

To create the SYNTAX TREE

Crunking¶

Picking up Individual pieces of information and Grouping them into bigger PIECES

In [ ]:

new = "The big cat ate the little mouse who after cheese"

new_tokens = nltk.pos_tag(word_tokenize(new))
new_tokens

new = "The big cat ate the little mouse who after cheese" new_tokens = nltk.pos_tag(word_tokenize(new)) new_tokens

Out[ ]:

[('The', 'DT'),
 ('big', 'JJ'),
 ('cat', 'NN'),
 ('ate', 'VBD'),
 ('the', 'DT'),
 ('little', 'JJ'),
 ('mouse', 'NN'),
 ('who', 'WP'),
 ('after', 'IN'),
 ('cheese', 'NN')]

In [ ]:

!pip install nltk

import nltk

import re

grammar_np = r"NP: {<DT>?<JJ>*<NN>}" # NP : Noun Phrase : Phrase nominale!!!!(Sans verbs)

chunk_parser = nltk.RegexpParser(grammar_np)

chunk_result = chunk_parser.parse(new_tokens)

chunk_result

!pip install nltk import nltk import re grammar_np = r"NP: {<DT>?<JJ>*<NN>}" # NP : Noun Phrase : Phrase nominale!!!!(Sans verbs) chunk_parser = nltk.RegexpParser(grammar_np) chunk_result = chunk_parser.parse(new_tokens) chunk_result

SEMANTIC ANALYSIS¶

• semantic wiki : https://en.wikipedia.org/wiki/Semantics
• GenSim : https://radimrehurek.com/gensim/index.html
• Fasttext : https://radimrehurek.com/gensim/auto_examples/tutorials/run_fasttext.html#sphx-glr-auto-examples-tutorials-run-fasttext-py
• Model2Vec : https://radimrehurek.com/gensim/auto_examples/tutorials/run_word2vec.html#sphx-glr-auto-examples-tutorials-run-word2vec-py

Text / document classification¶

exemple from a fastText pre-trained model¶

*design/create our own model ?!

In [ ]:

!pip install nltk 

!pip install nltk

Requirement already satisfied: nltk in /usr/local/lib/python3.6/dist-packages (3.2.5)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from nltk) (1.15.0)

In [ ]:

import nltk
from nltk.tokenize import RegexpTokenizer
from nltk.corpus import stopwords
nltk.download('stopwords')

import nltk from nltk.tokenize import RegexpTokenizer from nltk.corpus import stopwords nltk.download('stopwords')

[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.

Out[ ]:

True

In [ ]:

import pandas as pd
import numpy as np
import string
from gensim.models import FastText, Word2Vec, KeyedVectors

import pandas as pd import numpy as np import string from gensim.models import FastText, Word2Vec, KeyedVectors

In [ ]:

model = KeyedVectors.load_word2vec_format('/content/drive/MyDrive/Colab Notebooks/wiki.kg.vec', binary=False, limit=10000)
model.wv.most_similar('muntu')

model = KeyedVectors.load_word2vec_format('/content/drive/MyDrive/Colab Notebooks/wiki.kg.vec', binary=False, limit=10000) model.wv.most_similar('muntu')

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-4-bb026f2261d2> in <module>()
----> 1 model = KeyedVectors.load_word2vec_format('/content/drive/MyDrive/Colab Notebooks/wiki.kg.vec', binary=False, limit=10000)
      2 model.wv.most_similar('muntu')

/usr/local/lib/python3.6/dist-packages/gensim/models/keyedvectors.py in load_word2vec_format(cls, fname, fvocab, binary, encoding, unicode_errors, limit, datatype)
   1436         return _load_word2vec_format(
   1437             cls, fname, fvocab=fvocab, binary=binary, encoding=encoding, unicode_errors=unicode_errors,
-> 1438             limit=limit, datatype=datatype)
   1439 
   1440     def get_keras_embedding(self, train_embeddings=False):

/usr/local/lib/python3.6/dist-packages/gensim/models/utils_any2vec.py in _load_word2vec_format(cls, fname, fvocab, binary, encoding, unicode_errors, limit, datatype)
    169 
    170     logger.info("loading projection weights from %s", fname)
--> 171     with utils.smart_open(fname) as fin:
    172         header = utils.to_unicode(fin.readline(), encoding=encoding)
    173         vocab_size, vector_size = (int(x) for x in header.split())  # throws for invalid file format

/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py in smart_open(***failed resolving arguments***)
    420     ignore_ext = ignore_extension
    421     del kwargs, url, message, ignore_extension
--> 422     return open(**locals())
    423 
    424 

/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py in open(uri, mode, buffering, encoding, errors, newline, closefd, opener, ignore_ext, transport_params)
    185         encoding=encoding,
    186         errors=errors,
--> 187         newline=newline,
    188     )
    189     if fobj is not None:

/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py in _shortcut_open(uri, mode, ignore_ext, buffering, encoding, errors, newline)
    285         open_kwargs['errors'] = errors
    286 
--> 287     return _builtin_open(local_path, mode, buffering=buffering, **open_kwargs)
    288 
    289 

FileNotFoundError: [Errno 2] No such file or directory: '/content/drive/MyDrive/Colab Notebooks/wiki.kg.vec'

In [ ]:

from gensim.models import FastText
from tqdm import tqdm

def load_fasttext():
  print('loading word embeddings...')
  embeddings_index = {}
  f = open('/content/drive/MyDrive/Colab Notebooks/wiki.kg.vec', encoding='utf-8')
  for line in tqdm(f):
    values = line.strip().rsplit(' ')
    word = values[0]
    coefs = np.asarray(values[1:], dtype='float32')
    embeddings_index[word] = coefs
  f.close()
  print('found %s word vectors' % len(embeddings_index)) 
  return embeddings_index

embeddings_index=load_fasttext()

from gensim.models import FastText from tqdm import tqdm def load_fasttext(): print('loading word embeddings...') embeddings_index = {} f = open('/content/drive/MyDrive/Colab Notebooks/wiki.kg.vec', encoding='utf-8') for line in tqdm(f): values = line.strip().rsplit(' ') word = values[0] coefs = np.asarray(values[1:], dtype='float32') embeddings_index[word] = coefs f.close() print('found %s word vectors' % len(embeddings_index)) return embeddings_index embeddings_index=load_fasttext()

https://stackabuse.com/python-for-nlp-working-with-facebook-fasttext-library/ https://medium.com/manash-en-blog/how-to-use-pre-trained-word-vectors-from-facebooks-fasttext-a71e6d55f27 https://hackernoon.com/text-classification-simplified-with-facebooks-fasttext-b9d3022ac9cb

Text / document generation¶

Machine Translation¶