Repository navigation
etl-pipeline
- Website
- Wikipedia
No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents
Build data pipelines, the easy way 🛠️
Make stream processing easier! Easy-to-use streaming application development framework and operation platform.
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
Implementing best practices for PySpark ETL jobs and applications.
Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton
Enterprise-grade and API-first LLM workspace for unstructured documents, including data extraction, redaction, rights management, prompt playground, and more!
A Clojure high performance data processing system
A blazingly fast general purpose blockchain analytics engine specialized in systematic mev detection
Integrate LLM in any pipeline - fit/predict pattern, JSON driven flows, and built in concurency support.
A simplified, lightweight ETL Framework based on Apache Spark
Pythonic Stream-like manipulation of iterables
An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
A simple Spark-powered ETL framework that just works 🍺
This is a template you can use for your next data engineering portfolio project.
Jayvee is a domain-specific language and runtime for automated processing of data pipelines
Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)
The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.