[SPARK-15354] [CORE] Topology aware block replication strategies by shubhamchopra · Pull Request #13932 · apache/spark

Adding capability to prioritize peer executors based on rack awareness while replicating blocks.

shubhamchopra

committed

Minor modifications to get past the style check errors.

shubhamchopra

committed

Using blockId hashcode as a source of randomness, so we don't keep choosing the same peers for replication.

shubhamchopra

committed

Several changes:

shubhamchopra

committed

Adding null check so a Block Manager can be initiaziled without the master.

shubhamchopra

committed

Renaming classes/variables from rack to a more general topology.

shubhamchopra

committed

Renaming classes/variables from rack to a more general topology.

shubhamchopra

committed

We continue to randomly choose peers, so there is no change in current behavior.

shubhamchopra

committed

Spelling correction and minor changes in comments to use a more general topology instead of rack.

shubhamchopra

committed

Providing peersReplicateTo to the prioritizer.

shubhamchopra

committed

Adding developer api annotations to TopologyMapper and BlockReplicationPrioritization

shubhamchopra

committed

Changes recommended by @HyukjinKwon to fix style issues.

shubhamchopra

committed

Updating prioritizer api to use current blockmanager id for self identification.

shubhamchopra

committed

Fixing style issues.

shubhamchopra

committed

Fixing style issues.

shubhamchopra

committed

Adding a set-cover formulation for picking peers to replicate blocks.

shubhamchopra

committed

Adding newline to the end of file

shubhamchopra

committed

Modifying the optimizer to use a modified greedy optimizer. We now try to get peers making sure objectives previously satisfied are not violated.

shubhamchopra

committed

Making sure we consider peers we have previously replicated to.

shubhamchopra

committed

1. Fixing topology mapper class issue, so we instantiate it correctly. 2. Fixing style issues

shubhamchopra

committed

Adding an assertion in test case.

shubhamchopra

committed

Searching for the set of peer that meet most new objectives is optimized.

shubhamchopra

committed

Adding a basic HDFS like block replication strategy. Re-using BlockManagerReplicationSuite, to also run the same set of tests when using the basic strategy. Added a couple of specific test cases to…

shubhamchopra

committed

Running the suite for PrioritizerStrategy as well.

shubhamchopra

committed

Renaming to follow naming convention.

shubhamchopra

committed

Rebasing to master. Adding two strategies with numReplicas constraint, along with test cases.

shubhamchopra

committed

Fixing the test case to use the right conf parameter to set the replication policy.

shubhamchopra

committed

Fixing style errors.

shubhamchopra

committed

Incorporating suggestions from @sameeragarwal. Removing objectives based replication strategy and constraint solver associate with it.

shubhamchopra

committed

Incorporating suggestions from @cloud-fan

shubhamchopra

committed

Correcting indentation.

shubhamchopra

committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-15354] [CORE] Topology aware block replication strategies#13932

[SPARK-15354] [CORE] Topology aware block replication strategies#13932
shubhamchopra wants to merge 31 commits intoapache:masterfrom
shubhamchopra:PrioritizerStrategy

Commits on Mar 28, 2017

Adding capability to prioritize peer executors based on rack awareness while replicating blocks.

Minor modifications to get past the style check errors.

Using blockId hashcode as a source of randomness, so we don't keep choosing the same peers for replication.

Several changes:

Adding null check so a Block Manager can be initiaziled without the master.

Renaming classes/variables from rack to a more general topology.

Renaming classes/variables from rack to a more general topology.

We continue to randomly choose peers, so there is no change in current behavior.

Spelling correction and minor changes in comments to use a more general topology instead of rack.

Providing peersReplicateTo to the prioritizer.

Adding developer api annotations to TopologyMapper and BlockReplicationPrioritization

Changes recommended by @HyukjinKwon to fix style issues.

Updating prioritizer api to use current blockmanager id for self identification.

Fixing style issues.

Fixing style issues.

Adding a set-cover formulation for picking peers to replicate blocks.

Adding newline to the end of file

Modifying the optimizer to use a modified greedy optimizer. We now try to get peers making sure objectives previously satisfied are not violated.

Making sure we consider peers we have previously replicated to.

1. Fixing topology mapper class issue, so we instantiate it correctly. 2. Fixing style issues

Adding an assertion in test case.

Searching for the set of peer that meet most new objectives is optimized.

Adding a basic HDFS like block replication strategy. Re-using BlockManagerReplicationSuite, to also run the same set of tests when using the basic strategy. Added a couple of specific test cases to…

Running the suite for PrioritizerStrategy as well.

Renaming to follow naming convention.

Rebasing to master. Adding two strategies with numReplicas constraint, along with test cases.

Fixing the test case to use the right conf parameter to set the replication policy.

Fixing style errors.

Incorporating suggestions from @sameeragarwal. Removing objectives based replication strategy and constraint solver associate with it.

Incorporating suggestions from @cloud-fan

Correcting indentation.

[SPARK-15354] [CORE] Topology aware block replication strategies#13932shubhamchopra wants to merge 31 commits intoapache:masterapache/spark:masterfrom shubhamchopra:PrioritizerStrategyshubhamchopra/spark:PrioritizerStrategyCopy head branch name to clipboard

Commits

Commits on Mar 28, 2017

[SPARK-15354] [CORE] Topology aware block replication strategies#13932
shubhamchopra wants to merge 31 commits intoapache:masterfrom
shubhamchopra:PrioritizerStrategy