0 votes
Is there a way to make the "propagate schema changes" work with a custom recipe?

This would be very handy especially if there are large chains of custom recipes and something at the very beginning changes..
asked by

1 Answer

0 votes
Best answer

Not possible: https://doc.dataiku.com/dss/latest/schemas/recipes.html#python-r-pyspark-sparkr

It would be marvelous for productivity throughout flow development to be able to pass a schema method to the input/output dataframes which constructs the schema when doing a schema propagation.

def get_ouput_schemas():
    result = {}
    schema1 = ds_input1.get_schema()
    schema2 = ds_input2.get_schema()
    schema_1, schema_2 = do_something(schema1, schema2)
    result['output_name_1'] = schema_1
    result['output_name_2'] = schema_2
   
dataiku.get_schemas(get_output_schemas)

 

 

answered by
836 questions
866 answers
809 comments
970 users