Can I use "append instead of overwrite" on a filter recipe to store results from a Twitter stream?

Alex_Combessie
Dataiker Alumni
Can I use "append instead of overwrite" on a filter recipe to store results from a Twitter stream?

I run my filter recipe periodically. I would like to use append instead of overwrite to store only new tweets (not what is already stored with the recipe output). Is it possible?

0 Kudos
2 Replies
jrouquie
Dataiker Alumni

Hi,



this looks like a use case for partitioning. Note this is an advanced concept of DSS.



Imagine one uses "append", discovers a bug, fixes it, runs the flow again: the output dataset would then contain both the buggy and the correct data.

0 Kudos
DvMg
Level 1
Do you have an example of how can do that?
0 Kudos

Labels

?
Labels (2)
A banner prompting to get Dataiku