Please Share your Product Ideas with us!

All ideas are welcome. Just because the Idea doesn't make it into the product immediately does not make it a bad idea.

Parquet target data format for AWS S3

hi Team,

Currently Syniti puts data in CSV. We've faced following issues because of this

  1. Extra columns in .mir file than ref. Reading data from text delimited files which do not support schema evolution becomes a tedious task

  2. Unrecognized characters in columns making parsing erroneous.

  3. Adding of \r column in the column name itself because of text file creation and handling of such characters by different libraries

The solution to it would be to use Parquet as the target data format for Syniti output.

  • Guest
  • Jan 10 2023
  • Attach files