Ensure schema-less tables have all rows respected in load! #260
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes #259. This one feels bad. For the schema-less input table case, we iterated the first row to get the names so we could do some schema validation, but then threw the row away, assuming that as we iterate later, we would somehow get all the rows anyway. That probably was fine in cases like a DataFrame, where iteration would correctly restart, but for most schema-less tables, it's not uncommon to be a stateful iterator that is forward-pass only (like `SQLite.Query`!). We fix this here by ensuring that row stays intact and is inserted before iterating further rows.
Codecov Report
@@ Coverage Diff @@
## master #260 +/- ##
==========================================
+ Coverage 84.53% 84.71% +0.18%
==========================================
Files 5 5
Lines 666 674 +8
==========================================
+ Hits 563 571 +8
Misses 103 103
Continue to review full report at Codecov.
|
|
Wow! Thanks for the quick response on this one! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #259. This one feels bad. For the schema-less input table case, we
iterated the first row to get the names so we could do some schema
validation, but then threw the row away, assuming that as we iterate
later, we would somehow get all the rows anyway. That probably was fine
in cases like a DataFrame, where iteration would correctly restart, but
for most schema-less tables, it's not uncommon to be a stateful iterator
that is forward-pass only (like
SQLite.Query!). We fix this here byensuring that row stays intact and is inserted before iterating further
rows.