Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
You can try setting the spark.jars.packages option for the wanted spark configuration. Beware that it will be applied to all the spark jobs using this configuration (including Spark notebooks), so it might make the startup of these jobs a bit slower. Also, for use with notebooks, you should restart DSS after setting this.
Generally speaking, to make libraries / jars available to spark notebooks, you have a couple options depending on your use case:
You can try setting the spark.jars.packages option for the wanted spark configuration. Beware that it will be applied to all the spark jobs using this configuration (including Spark notebooks), so it might make the startup of these jobs a bit slower. Also, for use with notebooks, you should restart DSS after setting this.
Generally speaking, to make libraries / jars available to spark notebooks, you have a couple options depending on your use case: