Description / Background

The ingest batcher can only accept files in an ingest source bucket. If you give it a file in the table data bucket it doesn't have permissions to read the file.

Steps to reproduce

1. Write a Parquet file to an instance's table data bucket
2. Submit it to the ingest batcher queue
3. Check the logs for the ingest-batcher-submit-files lambda
4. See error

Expected behaviour

The file should be added to the batcher and ingested.

Technical Notes / Implementation Details

We can add permissions in the stack IngestBatcherStack. We can add a test for ingesting via the table data bucket in IngestBatcherST.