0 votes

Hello,

Each month, I have to compute a dataset that takes the previous month's dataset (M-1) and add some stuff in it.
I wonder how I could to it in Dataiku as for the recipe, I should take the last output dataset (M-1) as the input.

I don't think it is currently possible to produce a feedback-loop in Dataiku: do you confirm ?
How could I achieve my computation with Dataiku ? The "append-only" feature is not a good answer, because before writing anything, I should read the (last month) output dataset to know what will be new in the (current month) output.

 

Best regards.

by
I'm not sure I understand why partitions don't solve your problem?

1 Answer

0 votes
Hi tomtom,

The flow interface won't like circular references, which sounds like what you're describing here.

Hence, if you have something like this: Dataset_A -> Recipe -> Dataset_B, one solution to your problem is to define Dataset_C by 'pointing it' to Dataset_B. You can do this in the flow by adding a new Dataset and matching the location (for example SQL table) of Dataset_B. This way you can use as input to your recipe both Dataset_A and Dataset_C (which is in fact the same as Dataset_B).

I hope this is not too confusing!
by
1,319 questions
1,339 answers
1,539 comments
11,888 users

©Dataiku 2012-2018 - Privacy Policy