Custom preprocessing steps in the Features Handling section in model design

[cronos003]

I'm trying to create a custom transformation but haven't been successful. The sample code provided works fine but when I define my own function with the same transformation it fails. See below for the exact snippets.



Works:

from sklearn import preprocessing
import numpy as np

# Applies log transformation to the feature
processor = preprocessing.FunctionTransformer(np.log1p)

Does not work:

from sklearn import preprocessing
import numpy as np

def CustomT(X):
    return np.log1p(X)

# Applies log transformation to the feature
processor = preprocessing.FunctionTransformer(CustomT)

Error (more detailed logs available if required):

Failed to train : <type 'exceptions.TypeError'> : expected string or Unicode object, NoneType found

I used a pared down dataset to generate this log: https://we.tl/t-YWJABcMO5s

[Alex_Combessie · Accepted solution]

Hello, (Repost from our official customer support channel) In a custom preprocessing, you cannot declare a function nor class as those won't be pickled. You need to define it in a library, at either the project level or instance level, see https://doc.dataiku.com/dss/latest/python/reusing-code.html, and then import and use it in the custom preprocessing code. Hope it helps, Alex

[Boris]

Hello @Alex_Combessie , 

Is it possible to do a Robustscaler as custom preprocessing ? If yes how possible is it possible to fit_transform the same model to all the features ?  

Thanks in advance

[Alex_Combessie]

Sure, there you go:

Note: this custom processor should be applied to each feature.