0 votes
Hi,

 what defines the list of available engines for data processing recipes such as prepare? I have a HDFS dataset created by Impala, then a prepare or sync to another HDFS dataset, but only Spark/MR (or LocalStream) is available. Why the DSS is not allowing to use SQL based engines? The source dataset has a hive synced table definition

 

Thanks
asked by

1 Answer

0 votes
Best answer
Hi,

For prepare recipe you only have the choice between Streaming, Hadoop/Mapreduce or Spark.

You don't have SQL engine because we generate java code we can not push in a SQL db.

https://doc.dataiku.com/dss/latest/preparation/engines.html

For sync recipe, as the purpose is to move data from one system to another, in some case we have to stream the data.

Matt
answered by
selected by
971 questions
998 answers
1,047 comments
2,361 users

┬ęDataiku 2012-2018 - Privacy Policy