data-processing

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

A lightweight data processing framework built on DuckDB and 3FS.

A light-weight, flexible, and expressive statistical data testing library

Concurrent and multi-stage data ingestion and data processing with Elixir

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

Large-scale pretraining for dialogue

Kubernetes-native platform to run massively parallel data/streaming jobs

Python Stream Processing

Extract Transform Load for Python 3.5+

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

Data and tools for generating and inspecting OLMo pre-training data.

Concurrent Python made simple

Scalable data pre processing and curation toolkit for LLMs

Large-scale pretrained models for goal-directed dialog

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/