Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
I Setting up the Spark using K8s but I'm facing the Performance warning: Input dataset is read in a non-distributed way and this make the queries taking so much time
attached a screenshot need your support to avoid the problem
Hi @Alwaleed ,
The warning simply means that Spark can't read the SQL dataset directly. So it will need to use DSS to read the data which will be slower because it's not distributed.
If you want to enable direct spark reads for compatible database you need to enable the option at the connection level :
Then you will need to enable on the dataset level