Skip to content

Commit cd967c7

Browse files
lqjacksteveloughran
authored andcommitted
HADOOP-15847. S3Guard testConcurrentTableCreations to set R/W capacity == 0
Contributed by lqjaclee Change-Id: I4a4d5b29f2677c188799479e4db38f07fa0591d1
1 parent 6282c02 commit cd967c7

File tree

3 files changed

+13
-7
lines changed

3 files changed

+13
-7
lines changed

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -854,13 +854,9 @@ and, for test runs in or near the S3/DDB stores, throttling events.
854854

855855
If you want to manage capacity, use `s3guard set-capacity` to increase it
856856
(performance) or decrease it (costs).
857-
For remote `hadoop-aws` test runs, the read/write capacities of "10" each should suffice;
857+
For remote `hadoop-aws` test runs, the read/write capacities of "0" each should suffice;
858858
increase it if parallel test run logs warn of throttling.
859859

860-
Tip: for agility, use DynamoDB autoscaling, setting the minimum to something very low (e.g 5 units), the maximum to the largest amount you are willing to pay.
861-
This will automatically reduce capacity when you are not running tests against
862-
the bucket, slowly increase it over multiple test runs, if the load justifies it.
863-
864860
## <a name="tips"></a> Tips
865861

866862
### How to keep your credentials really safe

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/s3guard/ITestS3GuardConcurrentOps.java

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,8 @@
4646
import org.apache.hadoop.fs.s3a.S3AFileSystem;
4747

4848
import static org.apache.hadoop.fs.s3a.Constants.S3GUARD_DDB_REGION_KEY;
49+
import static org.apache.hadoop.fs.s3a.Constants.S3GUARD_DDB_TABLE_CAPACITY_READ_KEY;
50+
import static org.apache.hadoop.fs.s3a.Constants.S3GUARD_DDB_TABLE_CAPACITY_WRITE_KEY;
4951

5052
/**
5153
* Tests concurrent operations on S3Guard.
@@ -55,6 +57,14 @@ public class ITestS3GuardConcurrentOps extends AbstractS3ATestBase {
5557
@Rule
5658
public final Timeout timeout = new Timeout(5 * 60 * 1000);
5759

60+
protected Configuration createConfiguration() {
61+
Configuration conf = super.createConfiguration();
62+
//patch the read/write capacity
63+
conf.setInt(S3GUARD_DDB_TABLE_CAPACITY_READ_KEY, 0);
64+
conf.setInt(S3GUARD_DDB_TABLE_CAPACITY_WRITE_KEY, 0);
65+
return conf;
66+
}
67+
5868
private void failIfTableExists(DynamoDB db, String tableName) {
5969
boolean tableExists = true;
6070
try {

hadoop-tools/hadoop-aws/src/test/resources/core-site.xml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -153,11 +153,11 @@
153153
<!-- Reduce DDB capacity on auto-created tables, to keep bills down. -->
154154
<property>
155155
<name>fs.s3a.s3guard.ddb.table.capacity.read</name>
156-
<value>10</value>
156+
<value>0</value>
157157
</property>
158158
<property>
159159
<name>fs.s3a.s3guard.ddb.table.capacity.write</name>
160-
<value>10</value>
160+
<value>0</value>
161161
</property>
162162

163163
<!--

0 commit comments

Comments
 (0)