-
Notifications
You must be signed in to change notification settings - Fork 1.5k
PARQUET-1573: Add a docker development image and use it in travis #637
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,3 +19,6 @@ target/ | |
| .cache | ||
| *~ | ||
| mvn_install.log | ||
| .factorypath | ||
| bin/ | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,10 +1,36 @@ | ||
| language: java | ||
| before_install: | ||
| - bash dev/travis-before_install.sh | ||
| language: generic | ||
|
|
||
| services: | ||
| - docker | ||
|
|
||
| env: | ||
| - HADOOP_PROFILE=default TEST_CODECS=uncompressed,brotli | ||
| - HADOOP_PROFILE=default TEST_CODECS=gzip,snappy | ||
| global: | ||
| - JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 | ||
| - IMAGE=parquet-mr | ||
| - CACHE_FOLDER=$HOME/docker-images | ||
| - CACHE_FILE=${CACHE_FOLDER}/parquet-mr.tgz | ||
| matrix: | ||
| - TEST_CODECS=uncompressed,brotli | ||
| - TEST_CODECS=gzip,snappy | ||
|
|
||
| cache: | ||
| directories: | ||
| - "$HOME/.m2" | ||
| - ${CACHE_FOLDER} | ||
|
|
||
| before_install: | ||
| - if [[ -f ${CACHE_FILE} ]]; then | ||
| docker load -i "${CACHE_FILE}"; | ||
| else | ||
| docker build -f dev/Dockerfile -t "${IMAGE}" "${PWD}"; | ||
| mkdir -p "${CACHE_FOLDER}"; | ||
| docker save "${IMAGE}" | gzip -c > "${CACHE_FILE}"; | ||
| fi | ||
|
|
||
| install: docker run -v $HOME/.m2:/root/.m2 -v ${PWD}:/parquet-mr -e JAVA_HOME=$JAVA_HOME -e TEST_CODECS=$TEST_CODECS -i -t parquet-mr /bin/bash -c "cd /parquet-mr && mvn install --batch-mode -DskipTests=true -Dmaven.javadoc.skip=true -Dsource.skip=true | pv -fbi 60 > mvn_install.log || (cat mvn_install.log && false)" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would make the command part of the container as well. For example, use the Why do we
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same comment that above, it is better to have a more flexible container than to tight commands into it for our use case.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Notice that htis is the install step and the goal here is only to put the artifcats into the .m2 directory that's the reason I found, but this code is inherited from the previous build file, as well as the one in the script step.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would definitely prefer to make both
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Makes sense, thanks |
||
|
|
||
| script: docker run -v $HOME/.m2:/root/.m2 -v ${PWD}:/parquet-mr -e JAVA_HOME=$JAVA_HOME -e TEST_CODECS=$TEST_CODECS -i -t parquet-mr /bin/bash -c "cd /parquet-mr && mvn verify javadoc:javadoc" | ||
|
|
||
| install: mvn install --batch-mode -DskipTests=true -Dmaven.javadoc.skip=true -Dsource.skip=true | pv -fbi 60 > mvn_install.log || (cat mvn_install.log && false) | ||
| script: mvn verify javadoc:javadoc -P $HADOOP_PROFILE | ||
| before_cache: | ||
| - rm -Rf $HOME/.m2/repository/org/apache/parquet | ||
| - find $HOME/.m2/repository/ -name *SNAPSHOT | xargs rm -Rf | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| ############################################################################### | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| ############################################################################### | ||
|
|
||
| FROM thrift:0.12.0 | ||
|
|
||
| RUN set -ex; \ | ||
iemejia marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| apt-get update && \ | ||
| apt-get install -y --no-install-recommends \ | ||
| maven \ | ||
| openjdk-8-jdk-headless \ | ||
| openjdk-8-jre-headless \ | ||
| pv && \ | ||
| rm -rf /var/lib/apt/lists/* | ||
This file was deleted.
This file was deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would move this as an
ENVstep in the docker image. It is part of the docker container, instead of the Travis build.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, part of the work here is to use the same image to test with Java 11 in the future if needed, and the thrift image includes Java 11. I know it looks weird but this will allow us to build two separate images.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Just to clarify, the
JAVA_HOMEis also set in the base image, right? https://github.com/docker-library/openjdk/blob/master/8/jdk/Dockerfile#L20For JDK11 we're still waiting on some blockers: #596 Mostly waiting for a new release of Parquet Format apache/parquet-format#127. This enables us to update Scrooge, which is required to update to Java 11.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually the parent image of this Parquet one is not
openjdk, it isthrift, and that's the reason why we had to define it manually, and why I had to install manually the JDK in the scripts too.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the links on Java 11 I wanted to be catch the progress of this 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor question what happened to the 1.11.0 release, i saw we are already in 1.12.0-SNAPSHOT but have not seen a release any part.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My mistake, I was under the impression that it was being built from
openjdk, butthriftmakes more sense. Thanks for the clarification! 👍The 0.11 release is still in progress, but has stalled because of unsigned keys:
https://lists.apache.org/thread.html/df101f14f5b5a2489443e23be32c7342e2b8a6c8eb95f2e6953fd5b4@%3Cdev.parquet.apache.org%3E