ru_punkt

Russian language support for NLTK's PunktSentenceTokenizer

1. Install NLTK python package:

pip install nltk

2. Download punkt data:

import nltk
nltk.download('punkt')

3. Download ru_punkt:

git clone https://github.com/Mottl/ru_punkt.git

4. Copy Russian tokenizer into nltk_data folder (ensure the appropriate location for your OS):

cp -r ru_punkt/nltk_data ~/nltk_data

import nltk

text = "Ай да А.С. Пушкин! Ай да сукин сын!"
print("Before:", nltk.sent_tokenize(text))
print("After:", nltk.sent_tokenize(text, language="russian"))

or

import nltk
tokenizer = nltk.data.load('tokenizers/punkt/russian.pickle')
text = "Ай да А.С. Пушкин! Ай да сукин сын!"
print("Before:", nltk.sent_tokenize(text))
print("After:", tokenizer.tokenize(text))

Output:

Before: ['Ай да А.С.', 'Пушкин!', 'Ай да сукин сын!']
After: ['Ай да А.С. Пушкин!', 'Ай да сукин сын!']

Data for sentence tokenization was taken from 3 sources:
– Articles from Russian Wikipedia (about 1 million sentences);
– Common Russian abbreviations from Russian orthographic dictionary, edited by V. V. Lopatin;
– Generated names initials.

After some research it was found that the single params.abbrev_types performs better than together with params.collocations and params.ortho_content, so the latter were removed from the trained tokenizer.