Commit 5c348c4
committed
HADOOP-11867. Add a high-performance vectored read API. (#3904)
part of HADOOP-18103.
Add support for multiple ranged vectored read api in PositionedReadable.
The default iterates through the ranges to read each synchronously,
but the intent is that FSDataInputStream subclasses can make more
efficient readers especially in object stores implementation.
Also added implementation in S3A where smaller ranges are merged and
sliced byte buffers are returned to the readers. All the merged ranged are
fetched from S3 asynchronously.
Contributed By: Owen O'Malley and Mukund Thakur
Conflicts:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java
pom.xml1 parent 4ba4630 commit 5c348c4
File tree
29 files changed
+2438
-89
lines changed- dev-support
- hadoop-common-project
- hadoop-common/src
- main/java/org/apache/hadoop/fs
- impl
- site/markdown/filesystem
- test/java/org/apache/hadoop
- fs
- contract
- localfs
- rawlocal
- impl
- test
- hadoop-project
- hadoop-tools
- hadoop-aws/src
- main/java/org/apache/hadoop/fs/s3a
- test
- java/org/apache/hadoop/fs
- contract/s3a
- s3a
- resources
- hadoop-benchmark
- src/main
- assembly
- findbugs
- java/org/apache/hadoop/benchmark
29 files changed
+2438
-89
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
Lines changed: 23 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
25 | 28 | | |
26 | 29 | | |
27 | 30 | | |
| |||
158 | 161 | | |
159 | 162 | | |
160 | 163 | | |
161 | | - | |
162 | | - | |
163 | | - | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
164 | 183 | | |
165 | 184 | | |
0 commit comments