datacleaner

Always know what to expect from your data.

The premier open source Data Quality solution

Installer for DataKitchen's Open Source Data Observability Products. Data breaks. Servers break. Your toolchain breaks. Ensure your team is the first to know and the first to solve with visibility across and down your data estate. Save time with simple, fast data quality test generation and execution. Trust your data, tools, and systems end to end.

Reduce, filter, and anonymize moodle data for non-prod environments

mde: Missing Data Explorer

CSVParser is a tool to parse csv file using univocity and commons csv parsers. It cleans new line (\n) character & special characters between data. It also handle various garbage data like odd no of quotes or delimiters in side quotes. It validate each record with specified delimiter count and separate it out to _GoodRecords.CSV and _BadRecords.CSV file. This is a Data Cleaner tool to run before ingestion to Data Lake. It make sure data is in right csv format to build table on it.

If you have some csv file and having CRLF, LF in between data and you want to create some table (Hive table). You will face issue that some of column have null value. It’s because line terminator in hive is \n and if and \n or \r coming between data it treating as line terminator before actual line terminator and rest for column is getting null value. I tried multiple option like spark, hive serde and many more but I found good with perl. Today I a sharing my Perl script to remove all newline and special characters.