tokenization

💫 Industrial-strength Natural Language Processing (NLP) in Python

Easy token price estimates for 400+ LLMs. TokenOps.

A suite of image and video neural tokenizers

LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/

Secure Vault for Customer PII/PHI/PCI/KYC Records

Ravencoin Core integration/staging tree

Unsupervised text tokenizer focused on computational efficiency

👑 spaCy building blocks and visualizers for Streamlit apps

All the slides, accompanying code and exercises all stored in this repo. 🎈

Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing

Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).

Ungreedy subword tokenizer and vocabulary trainer for Python, Go & Javascript

Natural Language Processing Pipeline - Sentence Splitting, Tokenization, Lemmatization, Part-of-speech Tagging and Dependency Parsing

PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language

ClangKit provides an Objective-C frontend to LibClang. Source tokenization, diagnostics and fix-its are actually implemented.

🎤 vibrato: Viterbi-based accelerated tokenizer

Sudachi in Rust 🦀 and new generation of SudachiPy

The official code 👩‍💻 for - TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysis

Fast and customizable text tokenization library with BPE and SentencePiece support

[NeurIPS 2024]OmniTokenizer: one model and one weight for image-video joint tokenization.