0 votes
Does the Analyse feature support interrow calculations similar to what we have with window functions in SQL. As a simple example get the column value of the previous row in an ordered dataset.
asked by

1 Answer

0 votes

Hi Uli,

No, in an Analysis, each row or line is calculated separately (as it can be multi-threaded). So you cannot use window functions in an Analysis.

You should use a SQL, Python or R recipe to do that.
(Similarly to the sorting.)

We have on the roadmap a Window recipe (similar to the Grouping recipe) that will work on datasets managed by SQL databases supporting such functions (PostgreSQL, etc.).

I hope that helps,

Jeremy
 

answered by
Hi Jeremy.

Thanks for that.
Let's assume I am using Spark as my execution engine and let's assume that I apply a Grouping recipe and then an Analysis where I replace a value in a column for example. When this is pushed down into Spark does it require two passes over the data or is there some optimization in the background that generates code with one pass only?
943 questions
976 answers
984 comments
2,058 users

┬ęDataiku 2012-2018 - Privacy Policy