Skip to content

Fixed an Parquet check and consistency #221

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 16, 2020

Conversation

gmanolache
Copy link
Contributor

@gmanolache gmanolache commented Nov 16, 2020

Description of changes:

Generating a configuration failed for Parquet as it was expecting all uppercase, changed it everywhere to match the rest of the data formats as well.

Fixed an issue when copying the manifest that contained parquet files because Redshift was expecting the meta property content_length

Here are the cloudwatch logs I was getting:

  | 2020-11-15T17:38:14.824+01:00 | COPY my_test_table from 's3://lambda-redshift-loader-test/files/manifest-2020-11-15 16:38:14-6082' with credentials as 'aws_access_key_id=meyaccesskey;aws_secret_access_key=mykey' manifest format as PARQUET;
-- | -- | --
  | 2020-11-15T17:38:14.824+01:00 | commit;
  | 2020-11-15T17:38:16.200+01:00 | error:
  | 2020-11-15T17:38:16.200+01:00 | -----------------------------------------------
  | 2020-11-15T17:38:16.200+01:00 | error: File entry does not have 'meta' key.
  | 2020-11-15T17:38:16.200+01:00 | code: 15003
  | 2020-11-15T17:38:16.200+01:00 | context:
  | 2020-11-15T17:38:16.200+01:00 | query: 90820823
  | 2020-11-15T17:38:16.200+01:00 | location: spectrum_manifest.cpp:204
  | 2020-11-15T17:38:16.200+01:00 | process: padbmaster [pid=17049]

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Fixed an issue when generating the manifest for Parquet that didnt contain meta information about content length.

(cherry picked from commit d107798)
@IanMeyers IanMeyers merged commit a4a1a77 into awslabs:master Nov 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants