Convert CSV and JSON to Parquet

Converting CSV and JSON to Parquet on AWS: Options, Pros, and Cons

Organizations often receive data in CSV or JSON format from applications, logs, partners, customers, and third-party systems. While these formats are easy to generate and exchange, they are not ideal for analytics at scale.

Apache Parquet is a columnar storage format designed for analytics workloads. Converting CSV or JSON files to Parquet can significantly reduce storage costs, improve query performance, and lower Amazon Athena query expenses.

This article reviews the most common approaches for converting CSV and JSON data to Parquet on AWS and compares their advantages and disadvantages.

Why Convert to Parquet?

Compared to CSV and JSON, Parquet provides:

• Smaller file sizes through compression and columnar storage
• Faster query performance
• Reduced Amazon Athena costs because fewer bytes are scanned
• Better integration with modern analytics engines such as Athena, Spark, Trino, and Redshift Spectrum
• Support for schema and data types

For many workloads, Parquet files are 3–10 times smaller than equivalent CSV files.

Option 1: Amazon Athena CTAS

Amazon Athena supports CREATE TABLE AS SELECT (CTAS) queries that can read CSV or JSON data and write the output directly as Parquet.

Example

CREATE TABLE sales_parquet
WITH (
    format = 'PARQUET',
    external_location = 's3://my-bucket/parquet/sales/'
)
AS
SELECT *
FROM sales_csv;

Pros

• No infrastructure to manage
• Simple SQL interface
• Good for one-time conversions
• Easy to integrate with existing Athena workflows

Cons

• Requires Athena tables to be created first
• Designed primarily for querying, not bulk conversion
• Limited control over Parquet optimization settings
• Can become expensive for large datasets
• Not ideal for recurring ingestion pipelines

Best For

Small datasets, ad hoc conversions, and teams already using Athena extensively.

Option 2: AWS Glue Jobs

AWS Glue can read CSV or JSON data from Amazon S3, transform it, and write the results as Parquet.

Pros

• Fully managed service
• Supports ETL transformations
• Scales automatically
• Integrates with AWS Glue Data Catalog
• Supports scheduled jobs

Cons

• More complex setup
• Higher operational cost for simple conversions
• Startup time can be several minutes
• Requires knowledge of Glue jobs and Spark
• Overkill when only format conversion is needed

Best For

Organizations already running ETL pipelines in AWS Glue.

Option 3: Apache Spark

Apache Spark can convert CSV and JSON files to Parquet using DataFrames.

Example

df = spark.read.csv("s3://bucket/input/")
df.write.parquet("s3://bucket/output/")

Pros

• Extremely flexible
• Handles very large datasets
• Supports complex transformations
• Industry standard for big data processing

Cons

• Infrastructure management required
• More engineering effort
• Requires Spark knowledge
• Higher operational complexity

Best For

Large-scale data engineering environments and advanced transformation workloads.

Option 4: DuckDB

DuckDB can convert CSV and JSON files directly to Parquet with simple SQL commands.

Example

COPY (
    SELECT *
    FROM read_csv_auto('sales.csv')
)
TO 'sales.parquet'
(FORMAT PARQUET);

Pros

• Very fast
• Lightweight
• Easy to use
• No cluster required

Cons

• Primarily a developer tool
• Limited operational features
• No centralized job management
• No built-in S3 conversion workflow

Best For

Developers and analysts performing local conversions.

Option 5: Custom Scripts

Many teams build internal scripts using Python, Pandas, and PyArrow.

Pros

• Maximum flexibility
• Full control over schema and transformations
• Easy to integrate into existing applications

Cons

• Requires development effort
• Maintenance responsibility remains with the organization
• Monitoring and error handling must be built separately
• Can become difficult to scale

Best For

Teams with strong software engineering resources.

Option 6: Parqify

Parqify is a dedicated CSV-to-Parquet, JSON-to-Parquet, and Parquet-to-Parquet optimization solution designed specifically for Amazon S3 data lake environments.

Unlike general-purpose ETL platforms, Parqify focuses on conversion and optimization workflows.

Pros

• Runs entirely in your AWS account
• No SaaS data transfer
• Web-based user interface
• No Spark or SQL knowledge required
• Schema inference and schema customization
• Built-in optimization profiles for Amazon Athena, AWS Glue, Apache Spark, Trino, and Presto
• Advanced Parquet settings such as compression codec, compression level, row group size, dictionary encoding, and Parquet version
• Parallel processing
• Job history and downloadable logs
• Supports both conversion and Parquet-to-Parquet optimization

Cons

• Dedicated tool rather than a general ETL platform
• Additional software to deploy and manage
• Focused on file optimization rather than broader data integration workflows

Best For

Organizations that regularly convert files to Parquet and want optimized output without building custom pipelines.

Comparison

Conclusion

There is no single best option for every organization.

Athena CTAS is excellent for quick conversions. AWS Glue and Spark are powerful when complex ETL processing is required. DuckDB and custom scripts work well for engineering teams comfortable with development tools.

However, if the primary goal is converting and optimizing CSV, JSON, or existing Parquet datasets for analytics platforms such as Athena, Spark, Trino, or Presto, a dedicated conversion tool can reduce operational complexity and provide better control over the resulting Parquet files.