SPARK-1426: Make MLlib work with NumPy versions older than 1.7 #391
Conversation
|
Can one of the admins verify this patch? |
|
Jenkins, test this please. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
@techaddict Is it the only place that requires numpy 1.7? If so, please also remove the check in |
|
@mengxr yup, tests passed with 1.6 |
|
Jenkins, test this please. |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Is 1.6 the oldest version it works with now, or could it also work with 1.5 or older? |
|
Tested, up until 1.4 all tests pass |
Currently it requires NumPy 1.7 due to using the copyto method (http://docs.scipy.org/doc/numpy/reference/generated/numpy.copyto.html) for extracting data out of an array. Replace it with a fallback
|
Just verified with numpy 1.6.2. All tests passed. |
|
@mengxr yupp works with 1.4, 1.5 too 👍 |
|
@mateiz anymore changes ? or this is good ? |
|
Alright, great! That's a good set of versions to support. I'm going to merge this then. |
|
Jenkins, test this please |
|
Merged build triggered. |
|
Merged build started. |
|
Merged build finished. All automated tests passed. |
|
All automated tests passed. |
|
Thanks Sandeep! I've merged this in. |
Currently it requires NumPy 1.7 due to using the copyto method (http://docs.scipy.org/doc/numpy/reference/generated/numpy.copyto.html) for extracting data out of an array. Replace it with a fallback Author: Sandeep <[email protected]> Closes #391 from techaddict/1426 and squashes the following commits: d365962 [Sandeep] SPARK-1426: Make MLlib work with NumPy versions older than 1.7 Currently it requires NumPy 1.7 due to using the copyto method (http://docs.scipy.org/doc/numpy/reference/generated/numpy.copyto.html) for extracting data out of an array. Replace it with a fallback (cherry picked from commit df36091) Signed-off-by: Matei Zaharia <[email protected]>
Currently it requires NumPy 1.7 due to using the copyto method (http://docs.scipy.org/doc/numpy/reference/generated/numpy.copyto.html) for extracting data out of an array. Replace it with a fallback Author: Sandeep <[email protected]> Closes apache#391 from techaddict/1426 and squashes the following commits: d365962 [Sandeep] SPARK-1426: Make MLlib work with NumPy versions older than 1.7 Currently it requires NumPy 1.7 due to using the copyto method (http://docs.scipy.org/doc/numpy/reference/generated/numpy.copyto.html) for extracting data out of an array. Replace it with a fallback
…loud-provider-openstack#400 Revert "Do not run jobs for changes to documentation"
…ache#391) * [SPARK-48197][SQL] Avoid assert error for invalid lambda function ### What changes were proposed in this pull request? `ExpressionBuilder` asserts all its input expressions to be resolved during lookup, which is not true as the analyzer rule `ResolveFunctions` can trigger function lookup even if the input expression contains unresolved lambda functions. This PR updates that assert to check non-lambda inputs only, and fail earlier if the input contains lambda functions. In the future, if we use `ExpressionBuilder` to register higher-order functions, we can relax it. ### Why are the changes needed? better error message ### Does this PR introduce _any_ user-facing change? no, only changes error message ### How was this patch tested? new test ### Was this patch authored or co-authored using generative AI tooling? no Closes apache#46475 from cloud-fan/minor. Authored-by: Wenchen Fan <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit 7e79e91) Signed-off-by: Wenchen Fan <[email protected]> * [SPARK-48197][SQL][TESTS][FOLLOWUP][3.5] Regenerate golden files ### What changes were proposed in this pull request? This PR is a follow-up to regenerate golden files for branch-3.5 - apache#46475 ### Why are the changes needed? To recover branch-3.5 CI. - https://github.com/apache/spark/actions/runs/9011670853/job/24786397001 ``` [info] *** 4 TESTS FAILED *** [error] Failed: Total 3036, Failed 4, Errors 0, Passed 3032, Ignored 3 [error] Failed tests: [error] org.apache.spark.sql.SQLQueryTestSuite ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46514 from dongjoon-hyun/SPARK-48197. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> --------- Signed-off-by: Wenchen Fan <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> Co-authored-by: Wenchen Fan <[email protected]> Co-authored-by: Dongjoon Hyun <[email protected]>
5 participants
Currently it requires NumPy 1.7 due to using the copyto method (http://docs.scipy.org/doc/numpy/reference/generated/numpy.copyto.html) for extracting data out of an array.
Replace it with a fallback