learned-tokenization

Implementation of MEGABYTE, Predicting Million-byte Sequences with Multiscale Transformers, in Pytorch

My attempts at applying Soundstream design on learned tokenization of text and then applying hierarchical attention to text generation