Current AWS EMR settings may cause thrashing/crashing on SF30k

[szarnyasg]

The current script that submits AWS EMR jobs may cause some thrashing:

• Generating SF10,000 on 100 instances takes 4 hours
• Generating SF30,000 on 300 instances takes 10 1/4 hours.

Using a few more (~10-20%) machines may help speed up SF30,000 and make the job cheaper.

[szarnyasg]

I ran an SF30k job on 7655fe8. It crashed (saved the logs locally).

[szarnyasg]

SF30k ran successfully on 20 i3.4xlarge machines with the following settings:

JAR location : command-runner.jar
Main class : None
Arguments : spark-submit --class ldbc.snb.datagen.LdbcDatagen s3://ldbc-snb-datagen-bi-2021-07/jars/ldbc_snb_datagen_2.12_spark3.2-0.4.0+207-6f234d14-jar-with-dependencies.jar --output-dir /ldbc_snb_datagen/build --scale-factor 30000 --num-threads 2567 --mode bi --format parquet --explode-edges --explode-attrs