Commit 6f593be
[SPARK-43267][JDBC] Handle postgres unknown user-defined column as string in array
### What changes were proposed in this pull request?
Spark SQL now doesn’t support creating data frame from a Postgres table that contains user-defined array column.
This PR support it as string.
### Why are the changes needed?
Support handle user-defined array column in SPARK SQL with Postgres
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
1. Add new test.
2. Tested in local.
```sql
CREATE DOMAIN not_null_text
AS TEXT
DEFAULT '';
create table films
(
code char(5 char) not null
constraint firstkey
primary key,
title varchar(40 char) not null,
did bigint not null,
date_prod date,
kind varchar(10 char),
tz timestamp with time zone,
int_arr integer[],
column_name not_null_text[],
column_name2 not_null_text
);
INSERT INTO public.films (code, title, did, date_prod, kind, tz, int_arr, column_name, column_name2) VALUES (e'2
', 'fdas', 1, '2023-04-07 16:05:48', '2', null, null, null, null);
INSERT INTO public.films (code, title, did, date_prod, kind, tz, int_arr, column_name, column_name2) VALUES (e'4
', 'fdsa', 1, '2023-04-07 16:05:48', '4', null, null, null, null);
INSERT INTO public.films (code, title, did, date_prod, kind, tz, int_arr, column_name, column_name2) VALUES ('1 ', 'dafsdf', 1, '2023-04-04 14:43:51', '1', '2023-04-25 18:53:17.467000 +00:00', '{1,2,3}', '{1,fds,fdsa}', 'fdasfasdf');
```
Test Case
```scala
test("jdbc array") {
val connectionProperties = new Properties()
connectionProperties.put("user", "system")
connectionProperties.put("password", "system")
spark.read.jdbc(
url = "jdbc:postgresql://localhost:54321/test?useSSL=false&serverTimezone=UTC" +
"&useUnicode=true&characterEncoding=utf-8",
table = "TEST.public.films",
connectionProperties
).show()
}
```
Result
<img width="1444" alt="image" src="https://user-images.githubusercontent.com/32387433/234458027-e67e410b-c417-400d-be7e-431768afc0ef.png">
Closes #40953 from Hisoka-X/SPARK-43267_pg_array.
Lead-authored-by: Jia Fan <[email protected]>
Co-authored-by: Hisoka <[email protected]>
Signed-off-by: Kent Yao <[email protected]>1 parent f2f6272 commit 6f593be
File tree
2 files changed
+20
-2
lines changed- connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc
- sql/core/src/main/scala/org/apache/spark/sql/jdbc
2 files changed
+20
-2
lines changedLines changed: 16 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
147 | 147 | | |
148 | 148 | | |
149 | 149 | | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
150 | 157 | | |
151 | 158 | | |
152 | 159 | | |
| |||
416 | 423 | | |
417 | 424 | | |
418 | 425 | | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
419 | 435 | | |
Lines changed: 4 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
97 | | - | |
| 97 | + | |
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
102 | 102 | | |
103 | | - | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
104 | 106 | | |
105 | 107 | | |
106 | 108 | | |
| |||
0 commit comments