Spark schema: Cannot handle an ARRAY without specified content type

UserBird
Dataiker
Spark schema: Cannot handle an ARRAY without specified content type
Hi,

I obtain this error message when training a model in DSS with Spark MLLib.

However, when I go to the "script" tab, I have properly set the meaning to "Text". Why does DSS still think it's an array ?
0 Kudos
1 Reply
Clรฉment_Stenac
When DSS trains the model, before applying the preparation script, it needs to load the original dataset as a Spark dataframe. It therefore needs to transform the schema of the dataset to a Spark schema, which requires content types for arrays.

Only then is the preparation script applied, and the meanings taken into account.

--> In the specific case of Spark MLLib, you need to make sure that the storage type in the dataset is set to string, in addition to setting the meaning.
0 Kudos

Labels

?
Labels (4)
A banner prompting to get Dataiku