[SPARK-27944][ML] Unify the behavior of checking empty output column names #24793
Conversation
|
Test build #106149 has finished for PR 24793 at commit
|
|
Test build #106150 has finished for PR 24793 at commit
|
|
This pr can help to avoid uncessary computation. For example, in GMM, current impl always predict twice, one for |
|
ping @srowen , would you mind help reviewing this? |
There was a problem hiding this comment.
I don't see how this happens given the check at the start of the method?
There was a problem hiding this comment.
Likewise how would this be empty, unless the user set it to nothing? in which case, this seems not worth worrying about
There was a problem hiding this comment.
I changed here to keep in line with other algs like LDA.transform
Or leave alone algs with only one output column, and remove the check in algs like LDA?
There was a problem hiding this comment.
Hm, I'd say we don't need this check anywhere that the user would have to explicitly set no prediction column to get no output, and in that case, I don't think it's worth checking and warning. I'm neutral on removing the other checks, but not against it.
Some checks are OK like the ones above as it might be easier to accidentally get into this situation because there are multiple prediction cols.
There was a problem hiding this comment.
PS we should fix these messages. "called as NOOP" is cryptic. Just say "transform() does nothing because ..."
zhengruifeng
force-pushed
the
aft_iso_check_empty_outputCol
branch
from
a0e4d54 to
f02051b
Compare
|
Test build #107577 has finished for PR 24793 at commit
|
mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala
Outdated
Show resolved
Hide resolved
zhengruifeng
force-pushed
the
aft_iso_check_empty_outputCol
branch
from
90eb0ac to
f010ff9
Compare
|
Test build #107665 has finished for PR 24793 at commit
|
| Vector => OldVector, Vectors => OldVectors} | ||
| import org.apache.spark.rdd.RDD | ||
| import org.apache.spark.sql.{DataFrame, Dataset, Row, SparkSession} | ||
| import org.apache.spark.sql._ |
There was a problem hiding this comment.
ok, I will revert these place.
| import org.apache.spark.mllib.util.MLUtils | ||
| import org.apache.spark.rdd.RDD | ||
| import org.apache.spark.sql.{DataFrame, Dataset, Row} | ||
| import org.apache.spark.sql._ |
| import org.apache.spark.mllib.tree.model.{DecisionTreeModel => OldDecisionTreeModel} | ||
| import org.apache.spark.rdd.RDD | ||
| import org.apache.spark.sql.{DataFrame, Dataset, Row} | ||
| import org.apache.spark.sql._ |
|
Test build #107711 has finished for PR 24793 at commit
|
|
Merged to master |
…names ## What changes were proposed in this pull request? In regression/clustering/ovr/als, if an output column name is empty, igore it. And if all names are empty, log a warning msg, then do nothing. ## How was this patch tested? existing tests Closes apache#24793 from zhengruifeng/aft_iso_check_empty_outputCol. Authored-by: zhengruifeng <[email protected]> Signed-off-by: Sean Owen <[email protected]>
5 participants
What changes were proposed in this pull request?
In regression/clustering/ovr/als, if an output column name is empty, igore it. And if all names are empty, log a warning msg, then do nothing.
How was this patch tested?
existing tests