Forecast Plugin - partition dataset

Question

Your forecast plugin looks great, but the flow takes all values as a single timeseries. Is it possible to specify a column to partition the data on? It would be nice to train and predict forecast models for multiple entities in one go.

Answer 1

Hey,

Thanks for your interest in this recent plugin release.

If you want run the recipes to get multiple forecasting models per category (e.g. per product or store), you will need partitioning. That requires to have all datasets partitioned by 1 dimension for the category, using the discrete dimension feature in Dataiku. If the input data is not partitioned, you can use a Sync recipe to repartition it, as explained in this article.

Hope it helps,

Alex

Comments

Thanks for your quick reply. I've tried rebuilding the flow with partitioning and that indeed works.

Maybe it's more of a general question: can you run a recipe for all partitions in the dataset? I could not find instructions on how to do so in the manual or in this QA section. Manually specifying 10's or 100's of partitions is not practical, if technically possible at all (?).

I was hoping to to find a feature that let's me to this in this plugin itself. Any help in scaling this up to more than a couple of entities would be very helpful.

---

Indeed there is no visual way in the partitioning menu of a recipe (of this plugin or any other recipe) to select all partitions. To do so, you would need an additional step to compute the complete list of partitions and store it as a project variable. Here is a piece of boilerplate code in python to do so:

combinations = np.unique(df["store_department"])
combinations_str = ",".join(combinations)

client = dataiku.api_client()
project = client.get_project(dataiku.default_project_key())
variables = project.get_variables()
variables["standard"]["store_department_combinations"] = combinations_str
project.set_variables(variables)

Then you can copy paste the comma-separated list of partitions in the partition menu of the plugin recipe.