Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
I'm used to working in Jupyter in standard AWS EC2 instances and via WSL. In both of these, PySpark displays progress while performing queries / transformations. Is there a way to get the behaviour in Dataiku's Jupyter implementation?
As always, I have set "spark.ui.showConsoleProgress" to "true"; however, it does not appear to have any effect.
Operating system used: Windows
Hi @tall_one ,
You can't currently use showConsoleProgress in Notebook.
In spark recipes you can look at spark-history-server when running jobs instead of notebooks. By click on "Spark Event log"
Thanks