FAQ

Why convert to Parquet at all?

Parquet typically reduces storage costs and significantly improves query performance for analytical workloads.

Do I need to write code?

No. Conversion jobs can be configured and managed through the web UI.

Why buy Parqify instead of writing a script?

Writing a script is easy. Maintaining a reliable conversion system with schema management, parallel processing, monitoring, configuration, and optimization is much harder. Parqify packages those capabilities into a ready-to-use solution.

Does Parqify require Spark?

No.

Why not just use Spark?

Spark is a powerful distributed processing engine designed for complex ETL, analytics, and machine learning workloads. Parqify focuses on a narrower problem: converting CSV and JSON data into optimized Parquet files with minimal operational overhead.

Is Parqify distributed?

No. Parqify runs on a single EC2 instance and uses parallel processing to utilize all available resources on that machine.

Where does it run?

Inside your AWS account.

Can I automate via API?

Not yet.

Does it support large files?

Yes.

Can it process thousands of files?

Yes. Parqify is designed to scan S3 prefixes and process large numbers of files in parallel.

How much data can Parqify process?

Throughput depends on instance size, file structure, and storage performance. Even small instances can process multiple terabytes per day.

Why is Parqify written in Rust?

Rust provides efficient memory usage, predictable performance, and stable throughput without garbage collection pauses.

How does Parqify perform the conversion?

Parqify is built in Rust and uses a parallel processing engine that automatically utilizes the available capacity of the AWS instance for high-speed file conversion. All operations are managed through an intuitive web UI.

Where does Parqify perform the conversion?

Parqify runs directly on your AWS EC2 instance and converts files within your AWS account. Source files are read from Amazon S3, processed on the EC2 instance, and the resulting Parquet files are written back to Amazon S3. Your data never leaves your AWS environment.

How much does Parqify cost?

Parqify pricing depends on the AWS instance size you choose. The software itself is offered through AWS Marketplace, and your total cost consists of:

• The Parqify software fee
• AWS EC2 instance costs

Small deployments can process terabytes of data per day, while larger instances can be used for enterprise-scale workloads. Pricing scales with the AWS resources you select.

How exactly does Parqify simplify the workflow? With Spark I can read CSV/JSON and write Parquet in just two lines of code.

Spark lets you build any data pipeline. Parqify eliminates the need to build one when all you need is fast, reliable CSV/JSON-to-Parquet conversion.

In Apache Spark, converting a file to Parquet can be as simple as two lines of code. The challenge is not the conversion itself—it's everything around it. Those two lines assume you've already solved:

• Infrastructure setup and cluster management
• Job scheduling and orchestrations
• Schema inference and schema consistency
• Parallel processing and resource allocation
• Partitioning strategy
• Monitoring, logging, retries, and error handling

For organizations that only need CSV/JSON → Parquet conversion, building and maintaining a Spark pipeline can be unnecessary overhead.

Parqify focuses on this specific use case and provides these capabilities out of the box:

• No Spark cluster, EMR, or Databricks required
• Runs as a simple AWS Marketplace AMI with a web UI
• Automatic parallel processing using all available instance resources
• Built-in S3 ingestion and multi-file processing
• Smart schema inference with user control
• Schema customization without writing code
• Consistent schema application across all files
• Built-in Parquet optimization profiles for Athena, Glue, Spark, Trino, and Presto

Parqify also gives you control over how schemas are inferred:

• Infer from every file independently
• Infer from the largest file
• Infer from a user-provided sample file

You can then modify the inferred schema before conversion by changing data types, removing columns, reordering columns, and applying custom Parquet settings.

The core engine is written in Rust, enabling efficient memory usage, predictable performance, and stable throughput when processing thousands of files.

Database engines can work with S3 too. DuckDB, for example, can read files directly from S3. I'd be interested to hear more about your architecture.

DuckDB is an excellent analytical database and can read data directly from S3. Parqify has a different focus. We're not building a query engine or database. Our goal is to efficiently convert large volumes of CSV and JSON data into optimized Parquet files. Architecturally, Parqify is a single-node Rust application with a built-in Web UI. The conversion process runs directly on an EC2 instance in your AWS account:

• Files are streamed from Amazon S3
• Processed in parallel using all available instance resources
• Schemas are inferred and can be customized before conversion
• Optimized Parquet files are generated
• Results are written back to Amazon S3

Parqify also handles many of the operational tasks that teams often build themselves:

• Processing thousands of files
• Scanning S3 prefixes and folders
• Schema inference and schema customization
• Parquet optimization and tuning
• Built-in optimization profiles for Athena, Glue, Spark, Trino, and Presto
• Job monitoring and management through a Web UI

In short:

DuckDB answers: "How do I query data?"

Parqify answers: "How do I efficiently convert large amounts of raw CSV and JSON data into analytics-ready Parquet files?"

Is Parqify a mini-ETL tool for CSV to Parquet conversion with semi-stable schemas?

No. Parqify allows three schema inference modes:

1. The largest file is automatically selected from the source location.
2. A specific file is provided by the customer.
3. Schema inference is performed separately for each file.

The first two modes allow schema customization and the use of a stable schema across all converted files.

In the UI, you can customize and edit the schema, and the same schema will be applied to all resulting Parquet files.

What does your typical customer profile look like?

Parqify is useful for organizations of any size that need to convert large volumes of CSV or JSON data into Apache Parquet.

Typical users include:

• Data engineering teams
• Analytics teams
• Cloud and DevOps engineers
• Companies using Athena, Glue, Spark, Trino, or Presto for reporting and analytics
• SaaS companies
• Enterprises building data lakes on AWS
• Organizations receiving data from multiple external sources with inconsistent schemas

A common situation is a team that has accumulated large amounts of CSV or JSON data in S3 and wants to reduce storage costs, improve query performance, and standardize datasets before analytics.

Parqify is less relevant for companies that already have mature Spark, Databricks, or custom ETL platforms. It's most valuable when the problem is specifically large-scale data format conversion and schema management, rather than complex data transformations.

Let's say I needed this conversion and didn't have access to your application. I'd probably use DuckDB. It feels like Parqify is aimed at teams that don't usually write their own utilities. So I'm curious what your customer profile actually looks like.

That's a fair observation, and for many engineers DuckDB, Spark, Python, or a custom script would absolutely be a viable solution.

The question is less about whether a team can build a conversion utility and more about whether they want to own and maintain one.

Many of our target users are technical teams. They know how to write scripts. The challenge is that over time those scripts tend to grow into production systems that require:

• Schema management
• Error handling
• Parallel processing
• Monitoring and logging
• Configuration management
• Maintenance and support

At that point, the organization is maintaining an internal data ingestion tool.

Parqify is for teams that would rather use an off-the-shelf solution than build and operate that functionality themselves.

Some customers may have only a few terabytes of data and want a simple operational model. Others may process thousands of files per day and want consistent schemas, optimized Parquet output, and a UI for managing conversions.

So the target customer is not necessarily a non-engineering team. It's any organization that sees CSV/JSON → Parquet conversion as infrastructure they would rather buy than build.

Strong engineers absolutely can build this themselves with DuckDB/Spark/Python. The question is usually whether the company wants to own and maintain that ingestion layer long-term.

How do you handle schema inference?

Parqify can infer schemas automatically and allows users to review and customize the result before conversion.

Do you use AI for schema detection?

No. Schema inference is deterministic and configurable. The goal is predictable output rather than AI-generated guesses.

Can I customize Parquet settings?

Yes. Compression, row groups, dictionary encoding, Parquet version, and optimization profiles can be configured.

Why do optimization profiles matter?

Different analytics engines perform best with different Parquet layouts. Profiles help generate files optimized for specific query engines.

Can I automate conversions?

Yes. Jobs can be configured and executed repeatedly as part of a data ingestion workflow.

Who is Parqify for?

Organizations that regularly convert CSV or JSON data into Parquet and want a dedicated solution instead of maintaining custom conversion pipelines.

When should I not use Parqify?

If your workload requires joins, aggregations, streaming, machine learning, or complex transformations, Spark, Flink, or a full ETL platform is likely a better fit.

Why did you build a dedicated product for a problem that can be solved with open-source tools?

Because organizations don't buy software to get access to code. They buy software to reduce engineering time, operational complexity, and maintenance costs. Parqify packages best practices for CSV/JSON → Parquet conversion into a tool that can be deployed and operated quickly.