"Gentle Introduction to Arrow / Record Batches" #11336

[sm4rtm4art]

Which issue does this PR close?

• Closes Add a "Gentle Introduction to Arrow / Record Batches" #11336

Since this is my first contribution, I suppose to mention @alamb , author of the Issue #11336

Could you please trigger the CI? Thanks!

Rationale for this change

The Arrow introduction guide (#11336) needed improvements to make it more accessible for newcomers while providing better navigation to advanced topics.

What changes are included in this PR?

Issue #11336 requested a gentle introduction to Apache Arrow and RecordBatches to help DataFusion users understand the foundational concepts. This PR enhances the existing Arrow introduction guide with clearer explanations, practical examples, visual aids, and comprehensive navigation links to make it more accessible for newcomers while providing pathways to advanced topics.

Was unsure if this fits to `docs/source/user-guide/dataframe.md'

Are these changes tested?

applyed prettier, like described.

Are there any user-facing changes?

Yes - improved documentation for the Arrow introduction guide at docs/source/user-guide/arrow-introduction.md

[Jefffrey]

Thanks for picking this up; I left a few comments, though my overall thoughts are that the guide as is feels a little disjointed in some of the information that is presented and is confusing to me as I don't know what preexisting knowledge it assumes of readers. Maybe the article would benefit from having a tighter focus and leaving more verbose details to external links (such as the Arrow docs).

Then again I'm not coming from a fresh user perspective so I'm a biased in that regard 😅

[Jefffrey]

These links below aren't actually visible so don't think this header is necessary

[sm4rtm4art]

Sorry, forgot this.
I wanted to implement the gentle introduction to Arraw in the Dataframe part based on the idea coming from the smalest part to a bigger one, but I guess this would be overwhelming and way to long for a user guid. In addition I've learned, that those references are not viible...

[Jefffrey]

I would be hesitant to mention compression here as being an in-memory format it isn't typically compressed (as compared to something like Parquet)

[sm4rtm4art]

You're absolutely right - I was thinking way more down the line, I was conflating storage format benefits with in-memory benefits. Arrow's columnar layout enables better compression when written to disk (like Parquet), but that's not relevant for the in-memory processing context. I'll remove this point or rephrase to focus on the actual in-memory benefits like cache efficiency and SIMD operations.

[Jefffrey]

I feel the last two properties are a bit mismatched here; they are instead properties of arrays and not recordbatches, but more importantly in a guide that is meant to be a gentle introduction, they seem to be placed here randomly. If someone were to read Variable-length data (strings, lists) use offset arrays for efficient access there isn't much to gleam from that information (that is relevant to the overall theme of the guide) 🤔

[sm4rtm4art]

Good catch!
I mixed RecordBatch properties with Array implementation details. These technical details don't help someone understand "why RecordBatches" at a conceptual level. I'll either:

1. Remove these details entirely, OR
2. Reframe as "What this means for users" (e.g., "Data is immutable, so operations create new batches rather than modifying existing ones")

The offset arrays detail especially doesn't belong in a gentle introduction.

Would you prefer I remove this section or refocus it on user-facing implications?

[Jefffrey]

I do feel it is worth mentioning the immutable aspect of RecordBatches/Arrays, as that is an important detail if you want to get hands on.

[Jefffrey]

This section feels a bit misplaced, as some of these downsides were mentioned right above under Why this matters so it feels a little inconsistent to have the points stated again right below

[sm4rtm4art]

You're right -
I essentially repeated the same points. My intention was to show the progression from "too big" (entire table) → "too small" (single rows) → "just right" (batches), but I see it reads as repetitive.

I'll consolidate into a single "Why batches are the sweet spot" section that covers both extremes concisely without redundancy.

Do you have suggestions, I might not see ?

[Jefffrey]

We could even simplify it to a single line like "it's more efficient to process data in batches etc." and have more focus on how/when you interact with the record batches directly, rather than having details on why DF uses recordbatches (if the point of the guide is to ease users into getting familiar with interacting with arrow api)

[Jefffrey]

Suggested change

	A **[`RecordBatch`]** represents a horizontal slice of a table—a collection of equal-length columnar arrays sharing the same schema.
	A **[`RecordBatch`]** represents a horizontal slice of a table—a collection of equal-length columnar arrays that form a common schema.

I'm not sure about this wording either, but it feels slightly wrong to call the schema as being shared by arrays 🤔

[sm4rtm4art]

Good point about the wording.

How about:

A RecordBatch represents a horizontal slice of a table—a collection of equal-length columnar arrays that conform to a defined schema.

This makes it clearer that the schema defines the structure, and the arrays conform to it, rather than "sharing" it.

[Jefffrey]

I like the wording of that 👍

[Jefffrey]

This wording implies Arc is the key to Arrow, though it can be misleading considering that's more of an implementation detail on the Rust side 🤔

[sm4rtm4art]

You're absolutely right - Arc is a Rust implementation detail, not core to understanding Arrow conceptually. I included it because users will see Arc/ArrayRef in code examples, but I'm giving it too much emphasis. I'll either:

1. Move the Arc explanation to a small note: "Note: You'll see Arc in Rust code - it's how Rust safely shares data between threads"
2. Remove it entirely and let users learn about Arc when they actually need to write code

Which approach would you prefer?

[Jefffrey]

I lean toward the latter but I don't know how user friendly that might turn out 🤔

Maybe just add a small footnote about DataFusion being built around async + having pointers to the arrays themselves = use of Arc frequently to wrap these data structures

[Jefffrey]

I feel we can trim some of these references; for example including IPC is probably unnecessary for the goal of this guide.

[sm4rtm4art]

Agreed Thank you for helping me tighter focus and leaving more verbose details to external links -

IPC is too deep for this guide's scope. I'll trim the references to focus on:

• Main Arrow documentation (for those wanting to go deeper)
• DataFusion-specific references (MemTable, TableProvider, DataFrame)
• The academic paper (for those interested in the theory)

I'll remove IPC, memory layout internals, and other implementation-focused references.

[Jefffrey]

It feels weird to have a Next steps about working with the DataFrame API, given this guide itself is meant to be an introduction to Arrow for DataFusion users who may not need to use Arrow directly.

[sm4rtm4art]

I see your point - if users don't need Arrow directly, why guide them to DataFrames?

My thinking was: users reading this guide are trying to understand the foundation before using DataFusion. But you're right that it creates a circular path. Would it be better to:

• Remove "Next Steps" entirely, OR
• Reframe as "When you'll encounter Arrow" focusing on the extension points where Arrow knowledge becomes necessary?

The second option would reinforce that most users can stay at the DataFrame level. (See first comment of the dataframe.md, where I first wanted to implement the introduction to Arrow)

[Jefffrey]

If going with the second option, it might be a little odd to put it at the end of an article that is introducing users to recordbatch/arrow api as usually you'd provide a justification/reason upfront for why you might need this (to highlight why users would need to read the guide in the first place) 🤔

Maybe can just have the recommended reading links, but put some descriptions for each link so users would know why they might be interested in checking out the links (e.g. "understand arrow internals", "creating your own udf efficiently")

[comphead]

this file is autogenerated, if you wanna change the doc please change the userdoc for pub struct DateBinFunc

[sm4rtm4art]

Sorry, In my first attemp, I used prettier on all files and it "fixed" this one... Thought of doing good by fixing this.

[comphead]

Thanks @sm4rtm4art I would delegate RecordBatch description,. pros/cons and examples to Arrow itself, otherwise it would be quite complicated to keep documentation in sync. WDYT?

[sm4rtm4art]

Thank you both for your valuable feedback! @Jefffrey @comphead

My intention was to create a gentle introduction that gives users just enough context about Arrow to understand DataFusion betterbetter, without diving into implementation details. I hear your point about it feeling disjointed. To address this while keeping the gentle introduction approach, I propose:

1. Tighten the narrative flow: Focus on a single journey - "Why DataFusion uses Arrow" → "What is a RecordBatch conceptually" → "When you'll encounter it"
2. Move technical details to footnotes or links: Keep implementation details (like Arc, offset arrays) as brief notes or external links
3. Clarify assumed knowledge upfront: Add a brief "Who this guide is for" section stating we assume basic DataFusion knowledge but no Arrow background

The goal is to give users mental models, not implementation knowledge. Would this approach address your concerns about focus?

@comphead: I understand your maintenance concerns. My approach would be to provide:

1. Conceptual overview - Brief explanation of what RecordBatch is and why it matters to DataFusion users
2. Practical code example - Most simples example showing how it looks in practice (like the current "build a RecordBatch" example)
3. Direct links to Arrow docs - For readers who want deeper technical details

This way, we give users enough context to understand DataFusion without duplicating Arrow's technical documentation. The guide would serve as a bridge - explaining the "why" and showing the "what it looks like", while Arrow's docs handle the detailed "how it works internally".

Would this three-part approach (concept → example → link to details) work for you? It keeps our maintenance burden low while still providing value to users who encounter RecordBatch in DataFusion code.

Hope I don't get overboard with text.

Edit:
One thing I didn’t call out that I think would be valuable is a short note on data types. Readers hit RecordBatch alongside Arrow’s type system (e.g., Int64, Utf8, Timestamp[TZ], List, Struct) and null semantics. Should this live here as a small “Data types at a glance” box, or should we open a separate issue/page and keep this PR focused on RecordBatch?

If included here, I’d keep it tight:

• Logical vs physical types (one-sentence overview)
• Common mappings to SQL types
• Nulls and casting/promotion gotchas
• Link to Arrow docs for full details (timestamps/time zones, nested types)