Repository navigation
Entity resolution
Entity resolution (also known as data matching, data linkage, record linkage, and many other terms) is the task of finding entities in a dataset that refer to the same entity across different data sources (e.g., data files, books, websites, and databases). Entity resolution is necessary when joining different data sets based on entities that may or may not share a common identifier (e.g., database key, URI, National identification number), which may be due to differences in record shape, storage location, or curator style or preference.
Fast, secure, efficient backup program
Deduplicating archiver with compression and authenticated encryption.
Cross-platform backup tool for Windows, macOS & Linux with fast, incremental backups, client-side end-to-end encryption, compression and data deduplication. CLI and GUI included.
Prometheus Alertmanager
A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
rustic - fast, encrypted, and deduplicated backups powered by Rust
A fast high compression read-only file system for Linux, Windows and macOS
Extremely fast tool to remove duplicates and other lint from your filesystem
Simple, configuration-driven backup software for servers and workstations
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
Config driven, easy backup cli for restic.
plakar is a backup solution powered by Kloset and ptar
Scalable identity resolution, entity resolution, data mastering and deduplication using ML
Scalable data pre processing and curation toolkit for LLMs
A powerful and modular toolkit for record linkage and duplicate detection in Python
Data deduplication engine, supporting optional compression and public key encryption.
Fast Semantic Text Deduplication & Filtering
Open source project for data preparation for GenAI applications
Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.