Datafusion binary size has been getting bigger

[alamb]

Is your feature request related to a problem or challenge?

The size of datafusion's binary has grown significantly in the last few releases

This likely leads to higher compile times as well as larger overall binary size

version	size of datafusion-cli binary
main at 57d1309	92M
43.0.0	87M
42.0.0	83M
41.0.0	72M
40.0.0	69M
39.0.0	68M

The sizes are measured like this:

git checkout version
cd datafusion-cli
cargo build --release
du -h target/release/datafusion-cli

Also, people such as @g3blv have noticed that the WASM build has increased 50%:
#9834 (comment)

Describe the solution you'd like

I would like to reduce the binary size of DataFusion if possible

At least I would like to understand where the code size comes from and offer hints about how to reduce the size if needed

Describe alternatives you've considered

A common source of code size is templated functions (as that generates multiple copies of the same function(s)).

Here is some fascianting information from running cargo bloat -p datafusion

 File  .text    Size                          Crate Name
 0.1%   0.3% 79.7KiB                         blake2 blake2::Blake2bVarCore::compress
 0.1%   0.2% 70.7KiB                         blake2 blake2::Blake2sVarCore::compress
 0.1%   0.2% 67.1KiB                      sqlparser <sqlparser::ast::Statement as core::fmt::Display>::fmt
 0.1%   0.2% 61.4KiB                         blake3 _blake3_hash4_neon
 0.1%   0.2% 56.4KiB                      chrono_tz <chrono_tz::timezones::Tz as chrono_tz::timezone_impl::TimeSpans>::timespans
 0.1%   0.2% 44.7KiB                     arrow_cast <i64 as lexical_write_integer::api::ToLexical>::to_lexical
 0.1%   0.1% 42.8KiB                     arrow_cast arrow_cast::cast::cast_with_options
 0.0%   0.1% 35.9KiB                           rand <rand_chacha::chacha::ChaCha12Core as rand_core::block::BlockRngCore>::generate
 0.0%   0.1% 34.9KiB                     arrow_cast lexical_parse_float::slow::parse_mantissa
 0.0%   0.1% 33.1KiB                     arrow_cast lexical_parse_float::parse::parse_complete
 0.0%   0.1% 33.1KiB                     arrow_cast lexical_parse_float::parse::parse_complete
 0.0%   0.1% 29.0KiB                 regex_automata regex_automata::hybrid::search::find_fwd
 0.0%   0.1% 27.6KiB                         blake3 blake3::portable::compress_in_place
 0.0%   0.1% 27.1KiB                   aho_corasick aho_corasick::automaton::try_find_fwd
 0.0%   0.1% 25.2KiB                      sqlparser <sqlparser::ast::Expr as core::fmt::Display>::fmt
 0.0%   0.1% 23.8KiB              datafusion_common datafusion_common::scalar::ScalarValue::iter_to_array
 0.0%   0.1% 23.7KiB              datafusion_common datafusion_common::scalar::ScalarValue::iter_to_array
 0.0%   0.1% 23.7KiB       datafusion_physical_expr datafusion_common::scalar::ScalarValue::iter_to_array
 0.0%   0.1% 23.7KiB datafusion_functions_aggregate datafusion_common::scalar::ScalarValue::iter_to_array
 0.0%   0.1% 22.0KiB                     arrow_cast <u64 as lexical_write_integer::api::ToLexical>::to_lexical
36.7%  97.4% 27.7MiB                                And 139272 smaller methods. Use -n N to show more.
37.7% 100.0% 28.4MiB                                .text section size, the file size is 75.4MiB

Additional context

No response

[comphead]

print_functions_docs
print_functions_config

binaries can be moved out from the main release

[comphead]

Some good experiments are https://github.com/johnthagen/min-sized-rust?tab=readme-ov-file#optimize-libstd-with-xargo

with this profile

[profile.release]
codegen-units = 1
strip = true
panic = "abort"

The cli size went from 94.3MiB down to 48.6MiB 🤔

[alamb]

That is a very cool page 🤔

[Omega359]

[profile.release]
codegen-units = 1
strip = true
panic = "abort"
opt-level = "s"

Expanding on @comphead's idea adding opt-level = "s" reduced the size of the cli from 52MB on my machine (with his changes) to 37MB.

[comphead]

Thanks @Omega359 Opt-level is 3 by default for the release https://doc.rust-lang.org/cargo/reference/profiles.html#release
which focus on maximum runtime speed, I think it is important. However we can strip things to make executable smaller, I'll create a PR soon

[alamb]

FWIW I don't think the size of hte datafusion-cli binary is all that critical per se (maybe we can adjust / optimize the size of what is distributed on homebrew)

What I was hoping to review / improve with this PR is the size of the code in general (and review if there were places that were unecessairly causing code bloat that weren't also adding value

[alamb]

At a high level, I think this ticket has 2 parts:

1. Figure out what is contributing to code size increase
2. Then perhaps figure out how to make it better

I think the most valuable (and hardest) part is 1 (figuring out what to do)

To do so I recommend doing an "Ablation Study"

An ablation study aims to determine the contribution of a component to an AI system by removing the component, and then analyzing the resultant performance of the system.[2]

This is a fancy way of saying "remove parts of the system and see how much impact it makes on binary size"

Suggested things to try

I suggest initially simply trying with different datafusion crate features and see how much extra code each contributes to the binary size.

A follow on idea would be to comment out some of the features that require lots of generic code such as

• REmove (all implementations of GroupValues: https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/aggregates/group_values/mod.rs
• Remvoe all implementations of GroupAccumulator:

  datafusion/datafusion/expr-common/src/groups_accumulator.rs

  Line 108 in 469f18b

  pub trait GroupsAccumulator: Send {

Example of Ablation Study for the parquet feature

For example, to test the impact of the parquet feature, I tested the size of the binary with and without parquet support

cargo build --release -p datafusion-cli
# get size of datafusion-cli in kb
du -k target/release/datafusion-cli

Here is what I got

type	size in mb	size in kb
default	58	58440
without parquet feature	53	54248

So I conclude that the parquet feature adds approximately 5mb to the binary size

To remove parquet support I hacked out the dependency on parquet. Since this was just to test the impact as long as it compiles that is good enough. No need to be pretty. Here is the

• hack out parqeut feature from datafusion-cli #14666

[comphead]

That is really awesome write up @alamb
Tbh I wanted to do exactly the same but I even had no idea this approach has so smart definition Ablative Study.