Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi,
I have a scenario that runs other scenarios as steps. One of these steps downloads some files from a third party website. In some instances there won't be any files downloaded so the next step fails as there are no files to process. I would like to be able to not execute all the remaining scenario steps when there are no files on my folder. However I don't want any steps to be set to warning or failing since it is possible that we won't get any files some days and that is a normal situation we don't want to trigger alerts for.
There is a similar post already although this requires the use of metrics which will cause the step to fail or give a warning:
https://community.dataiku.com/t5/Using-Dataiku/Scenario-Conditional-on-Dataset-Metric/m-p/8474
Any ideas how to do this? I am thinking in using scenario variables but haven't seen a good example yet. Thanks
Thanks fchataigner2, that's a valid solution that I initially thought of doing but then I decided to see if I could do it in a "clickers" way, without using Python code. With the help of Dataiku Support I came to this solution:
That's it! The step will conditionally execute based on the metric value of a folder. No failures or warnings.
Hi,
you can use the run condition on scenario steps for that. For example with a "Execute python code" like this one:
import dataiku
mf = dataiku.Folder('f')
contents = mf.list_paths_in_partition()
f_is_empty = len(contents) == 0
from dataiku.scenario import Scenario
Scenario().set_scenario_variables(f_is_empty=f_is_empty)
all the steps after this code can setup a run condition like:
Then when there are files in the folder "f", you get these subsequent steps run, and when there are no files, the steps are simply skipped
Thanks fchataigner2, that's a valid solution that I initially thought of doing but then I decided to see if I could do it in a "clickers" way, without using Python code. With the help of Dataiku Support I came to this solution:
That's it! The step will conditionally execute based on the metric value of a folder. No failures or warnings.
Thank you for sharing your solution @Turribeach!
One last thing to add. By using the "If condition is satisfied" on a scenario step you can conditionally control if the step is executed based on the value of a metric, in this case the number of files on a folder. But using the "If condition is satisfied" option may also have an unwanted side effect. By default scenario steps only execute "if no prior step failed" so if you use the "If condition is satisfied" option to evaluate a variable you should take this side effect into consideration. In my case I still wanted to make sure no steps executed if a prior step had failed, even those that use the "If condition is satisfied" option. So my actual condition ended up being this:
number_of_files >= 1 && outcome == 'SUCCESS'
As per the Step Flow Control documentation:
So outcome holds the current state of the scenario which means that if any prior step failed outcome = FAILED and the conditional steps will not execute. Enjoy!
Thank you for sharing the step-by-step. I am getting an error in the "Define variable" and most likely it is how I am porting over the formula.
Checking the scenario log, I think I got the metric name correct. For me I'm using "records:COUNT_RECORDS".
I also think I got the project ID correctly using the URL to retrieve.
I think I'm getting the folder ID wrong. I looked up how to retrieve the folder ID but doesn't seem to apply to this case. Any ideas or examples someone can share?
Error:
java.lang.Exception
parseJson failed: Missing value at 0 [character 1 line 1]
If open the folder in your project the ID will be shown in the web browser URL.
The project is not in any folder so no folder name is listed in the URL.
my URL is .../projects/projectID/scenarios/.
So if the project is in the "base" folder then would the syntax ['ProjectID.FolderID_NP']
equal
['ProjectID']? I tried several variations with no success.
Thank you.
The folder is not the folder where the project is but the folder in the flow where the files I was trying to count. At this stage it will best for you to start a new thread and describe exactly what you are trying to achieve because it doesnโt look like is the same situation as the I described in my thread.