Source and shuffle parallelism settings don't seem to work

[kkrugler]

Fabian had described the two configuration parameters:

The parameters are:

• flink.num.sourceTasks to specify the parallelism of source tasks.
• flink.num.shuffleTasks to specify the parallelism of all shuffling tasks (GroupBy, CoGroup).

I tried setting these in the conf that’s passed to the Flink FlowConnector, but they don’t seem to be working as expected.

I know that something is happening, as the first time I tried this (setting flink.num.sourceTasks to 40, and flink.num.shuffleTasks to 20) I forgot to remove the -p 20 parameter from the CLI.

This caused the job to fail, with an error about not having enough slots.

When I removed the -p 20 parameter, then the job runs, but I see everything using 40 slots (source/map tasks and groups)