I have a similar question to How to use JDBC source to write and read data in (Py)Spark?
My question is that I have to use spark-submit to run my java jar file, which is a jar file (java project) that uses clickhouse JDBC and takes 2 application arguments (args[0] -> write/read args[1] -> file path/table_name)
In my java, I have
public class Main {
public static void main(String[] args) throws SQLException {
System.out.println("TEST ! : " + args[0]);
System.out.println("TESTEST ! : " + args[1]);
String url = "jdbc:clickhouse://ipaddr:8123/test";
ClickHouseProperties properties = new ClickHouseProperties();
properties.setDatabase("test");
properties.setUser("user");
properties.setPassword("1234");
ClickHouseDataSource dataSource = new ClickHouseDataSource(url, properties);
String sql = "";
if(args[0] == "read"){
//read: args[0] = read , args[1] = DB table name to read
sql = "SELECT * FROM test." + args[1];
sqlContext.read.jdbc(url=url, table="test", properties=properties);
} else if (args[0] == "write"){
//write: args[0] = write args[1] = file path to be saved
sql = "";
DataFrame.write.jdbc(url=url, table="test_write", properties=properties);
} else {
System.out.println("args 0 either [read] or [write]");
}
System.out.println("sql : " + sql);
try (ClickHouseConnection conn = dataSource.getConnection();
ClickHouseStatement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(sql)) {
//what to write here,,
}
}
}
I've tried to write my java code taking help from the link but I can't really understand what's going on with my code.
about [DataFrame.write~~~] <- which library do I have to include in my project to trigger this API?
same with sqlContext :(
Or is there another simple / more proper way to connect Clickhouse JDBC to spark-submit and read/write files?
