Skip to content

Conversation

@agsachin
Copy link
Contributor

@agsachin agsachin commented May 6, 2016

What changes were proposed in this pull request?

while experimenting with structure streaming. I found that mode() is used for non-continuous queries while outputMode() is used for continuous queries.
ouputMode is not defined, so I have written the some raw implementation and test cases just to make sure the streaming app works

Note:-
/** Start a query */
private[sql] def startQuery(
name: String,
checkpointLocation: String,
df: DataFrame,
sink: Sink,
trigger: Trigger = ProcessingTime(0),
triggerClock: Clock = new SystemClock(),
outputMode: OutputMode = Append): ContinuousQuery = {
As per me outputMode should be defined before triggerClock, the constructor with outputMode defined will be used more often then triggerClock.
I have added triggerClock() method also

How was this patch tested?

using unit test locally

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

@agsachin agsachin changed the title SPARK-15183 [SPARK-15183][Streaming] Adding outputMode to structure Streaming Experimental Api May 6, 2016
@AmplabJenkins
Copy link

Can one of the admins verify this patch?

import org.apache.spark.annotation.Experimental
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
import org.apache.spark.sql.catalyst.analysis.{Update, OutputMode, Append, UnresolvedRelation}
Copy link
Member

@HyukjinKwon HyukjinKwon May 7, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I think it will throws an error for style check and should be as below:)

import org.apache.spark.sql.catalyst.analysis.{Append, OutputMode, Update, UnresolvedRelation}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks I have fixed the scala style issues

@agsachin
Copy link
Contributor Author

agsachin commented May 8, 2016

@tdas , @marmbrus any inputs on this.

@marmbrus
Copy link
Contributor

Trigger clock is an internal details for testing that should not be exposed to users. Also, this isn't really what the output mode is for. Try reading the design doc posted on JIRA: https://issues.apache.org/jira/browse/SPARK-8360

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants