subword-units

Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

Morfessor is a tool for unsupervised and semi-supervised morphological segmentation

Context-sensitive word embeddings with subwords. In Rust.

Properly handle position-dependent phones in a subword lexicon FST

Morfessor FlatCat

Morfessor EM+Prune

Finalfusion embeddings in Python

Semantic role labeling with subwords (character, character-ngram and morphology)

A python package to build a corpus vocabulary using the byte pair methodology and also a tokenizer to tokenize input texts based on the built vocab.

Cognate-aware morphological segmentation

A tool for generating sub-word (phone or grapheme) level embeddings from an HTK-style MLF ASR corpus

Morfessor demonstration

Classified sentences into one of Slovak, Czech, and English. Implemented relevant preprocessing steps, addressed the class imbalance in training set by employing the learned theory of Naive Bayes Models, and implementing subword units.

This repository contains the code to learn subword embeddings from the arXiv dataset of 1.7M+ scholarly papers.

🕵️ Language Model based on RNN for generating Sherlock Holmes stories.