Write DataFusion paper for (SIGMOD / VLDB / ICDE)

[alamb]

UPDATE: Final paper: https://dl.acm.org/doi/10.1145/3626246.3653368

Task List for SIGMOD Paper:

Per #6782 (comment), here is a list of TODO items:

• Initial draft of Introduction
• Initial draft of Background
• Initial draft of Design @Dandandan adding info about join
• Initial draft of Optimizations @Dandandan initial work on adding some contents
• Initial draft of Extensibility
• Initial draft of Related Work
• Initial draft of Conclusion
• Create Intro diagram ("teaser")
• Create Architecture diagram
• Take a first pass to add references
• Create CMT account on https://cmt3.research.microsoft.com/SIGMODIndustry2024
• Create Intro diagram ("teaser")
• Iterate / Polish

Issues Blocking Full Performance Results

• Error: There isn't a common type to coerce Binary and Utf8 in LIKE expression #7342
• Invalid argument error: the data type binary has no natural order #7343
• Internal error: The "character_length" function can only accept strings #7344
• Internal error: The "regex_replace" function can only accept strings #7345

Issues that would make the results more compelling

• (TO FILE or Find): Ensure performance doesn't degrade with larger core counts on some queries

Is your feature request related to a problem or challenge?

I would like to increase awareness of DataFusion in the broader technical community. One way to build mindshare is to get a paper / talk published in a prestigious conference like VLDB or SIGMOD

Writing a paper is a good way to show the strength of the arrow/datafusion.
Through the papers, more teachers, students and researcher may be involved, and contribute to the project.

Describe the solution you'd like

I would like to write a paper that explains DataFusion
Thesis: "You don't need a tightly integrated execution system to get good performance"

These blogs have some good material in the introduction
https://arrow.apache.org/blog/2023/06/24/datafusion-25.0.0/
https://arrow.apache.org/blog/2023/01/19/datafusion-16.0.0/

Then we would compare and contrast the approaches of other tightly integrated systems like pola.rs and duckdb to DataFusion

We would then describe the architecture of DataFusion and its many extension points (DataFrame, functions, aggregates, window functions, sinks, etc)

Performance:
Show DataFusion in the same ballpark as DuckDB for aggregation, grouping, etc (e.g. TPCH)
We already have this for querying parquet

Describe alternatives you've considered

VLDB: https://vldb.org/2024/?call-for-industrial-track

Submissions open	December 6, 2023
Short abstracts deadline	February 16, 2024
Full papers or extended abstracts deadline	March 1, 2024
Notifications	May 8, 2024
Camera-ready	June 15, 2024

SIGMOD: https://2024.sigmod.org/calls_papers_important_dates.shtml

Industrial track: https://2024.sigmod.org/comingsoon.shtml (TBD)

Research paper submission round 4 (All Deadlines are 11:59 PM Pacific Time)

October 15, 2023: Paper submission
November 26-28, 2023: Author feedback phase
December 20, 2023: Notification of accept/reject/review again
January 20, 2024: Revised paper submission
February 23, 2024: Final notification of accept/reject

ICDE:

Industrial Track: https://icde2024.github.io/CFP_industry.html

All deadlines below are 5 PM Pacific Time.

Paper submission: Monday, November 20, 2023
Notification of accept/reject: Wednesday, January 31, 2024
Camera-ready deadline: Thursday, March 28, 2024

Additional context

No response

[alamb]

Some ideas about the paper:

Thesis:

We demonstrate it is possible to get DuckDB like performance using standards like Parquet and Arrow as the internal interchange format, both inside of and outside of the engine. Previously the conventional wisdom has been that such performance levels require a tightly integrated engine where the disk format, in memory layout, and processing engine are engineered in tandem to work well together.

While the engineering effort required for such an engine is large, it is possible by leveraging the open source model and Apache governance model to poll resources amongst users. Given the availablity of fast, standards based, interoperable vectorized engines like DataFusion, we predict a Cambrian explosion of new analytic systems which would not have been possible before if they had to create their own engines

Background

(Section on Arrow) -- including in memory model, compute kernels, and Flight / FlightSQL

(Section on Parquet)

(Section on tokio -- aka not have to reinvent the scheduler)

(DataFusion usecases, examples of systems built on DataFusion)

Architecture

Internally DataFusion uses Arrow as the interchange between operators, though internally different, non standard formats are used (such as the Arrow Row Format)

Describe planning at a high level

List extension points + briefly describe how users can extend the engine (and how that is desiged for speed)

• UDF (everyone does this)
• UDAF (everyone does this)
• UDWF
• Optimizer Rules
• Custom Operators
• Own catalog
• Own partition pruning / elimination rules
• Provide information on sortedness

List briefly the features of the engine

• parquet / predicate pushdown with late materialization
• use of tokio scheduler and the benfits of that model (has good NUMA performance given how tasks are scheduled). Also note that use of morsel driven parallelism doesn't solve the typical bottneck on priorities -- memory
• group hash aggregates -- templated code
• Joins (TBD)
• Sophisticated sort based optimizations (including pre-sorted data) streaming group by, merge-join,
• "Partition Pruning" (Pruning predicate) for custom catalog support

Performance

(TPCH / ClickBench results)

Parquet / pushdown results

Talk about bespoke file format vs only using Parquet

Related work

Velox (focuses on the execution engine side)
Apache Calcite (focuses on sql and frontend)
DataFusion has all the pieces of the toolkit (sql frontend, logical plan, and execution plans)

[alamb]

Ideas for experiments I would like to run

Comparisons to existing systems:

SQL based

Clickbench (DataFusion + DuckDB on parquet)
TPCH (DataFusion + DuckDB on parquet)
h2o.ai (DataFusion + DuckDB in parquet)

DataFrame

Some sort of comparison with Pol.rs on aggregation
Some sort of comparison with pandas

Scalabilty performance

Also it would be awesome to have charts showing DataFusion scaling:

1. Performance as number of cores increases (ideally this would be linear)
2. Memory Usage vs input data size

Since I imagine there will be some tweaks given these numbers, having the run and chart generation automated would be very helpful

[alamb]

Other material:
fast grouping: https://www.influxdata.com/blog/aggregating-millions-groups-fast-apache-arrow-datafusion/
querying parquet with subsecond latency: https://www.influxdata.com/blog/querying-parquet-millisecond-latency/

[alamb]

I have begun work on this paper. I am planning to submit it to the SIGMOD industrial track https://2024.sigmod.org/calls_industrial_track_papers.shtml

I am planning to use Overleaf (recommended by the ACM) rather than setup the latex stuff myself, as it worked well when I used it on another paper earlier this year,

Here is the overleaf https://www.overleaf.com/read/qjhrxqhgksvr -- right now not much more than the template

If you are interested in being an author and helping to write the paper, please let me know. I am especially looking for help help running benchmarks / generating performance numbers as described in #6782 (comment)

[yjshen]

Some initial thoughts for the paper organization after checking the current version:

1. Introduction (1 page)
	- The deconstructed system(against conventional wisdom), the Apache way
	- technical methodology: Vectorized execution / Columnar input/output/intermediate / Pull mode with Morsel benefits

2. Background (1-1.5 pages)
	- The mature of ecosystem, columnar format parquet, in-mem format arrow, high performance thread scheduler tokio, and the surge of generalized relational algebra substrait
	- The needs for a deconstructured engine (high performance, and highly customizable)
		- an open engine for high performance (influxdb/ streamingdb/distrubuted olap)
		- an standard, extensible planner for resue (dask-sql,)
		- an general purpose parser sqlparser-rs (for TP, AP and streaming)

3. Design (3-4 pages)
	- Engine overview
	- execution model and scheduler
	- sort and the row model
	- aggregate and the aggregator model

4. Optimizations (1-1.5 pages)
	- Late materialization and predicate pushdown
	- Predicate prunning

5. Extensibilities (1 page)
	- UDF / UDAF
	- data source
	- catalog
	- parser?
	- ?

6. Benchmarks (2-3 pages)
	- micro-bench?
		- for sort?
		- for agg?
		- for scheduler?
	- e2e-bench
		- tpc-h?
		- clickbench?
		- ?

7. Related work (0.75 page)
	- Velox
	- DuckDB
	- Spark Photon

8. Conclusion (0.25 page)
	- the open source, the Apache way we believe
	- always pursing for performance start from real world problems

After some discussion with @alamb, we agreed that we could start with this above structure first and adapt it as we proceed.

[yjshen]

I incorporated the above structure into the paper, but split the background part into two sections:

• 2. Foundational Ecosystem
• 3. Deconstructing the Database,

Section 2 mainly talks about DataFusion's foundation technologies/systems, and Section 3 keeps the material from the analytical database that @alamb has previously shared the philosophy and vision of DataFusion.

I also finished my first pass for section 2, Foundational Ecosystem.

[alamb]

Thank you @yjshen -- this is amazing ❤️ . I plan to take a pass through the architecture section later this morning, and see how far I get. I also hope to start working on some diagrams to emphasize some of the points in the paper.