Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Dear all,
I'm facing an issue when trying to implement my own keras layer. After training the model, it crashes when trying to load the model. The load_model() command leads to the following error message:
File "/home/dataiku/dss_data/code-envs/python/google_api/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 134, in deserialize_keras_object
': ' + class_name)
After some google investigations, it seems the load_model() function should integrate a second optional argument which is a dictionary of the custom objects --> custom_objects={'LayerCustom: LayerCustom }.
Unfortunately, the load_model() function is called without this optional argument from dataiku as is it observed in the log file:
model = load_model(osp.join(run_folder, constants.KERAS_MODEL_FILENAME))
Would some of you have already implemented your own keras layers ? If yes, was it successful and did you face the problem ?
Thanks for your help,
Regards,
Hello,
We have added the possibility to use custom objects in the visual Deep Learning part of the Visual Machine Learning of DSS in release 5.1.3.
To use it, you need to register your custom object using the custom object handler, and then DSS will handle the serialization/deserialization for you.
For example, you can write a "MyDense" custom layer, that you put in a "my_layers.py" inside the libraries of the project on which you're going to build your DL model. It will look like:
# my_layers.py
from keras import backend as K
from keras.engine.topology import Layer
class MyDense(Layer):
def __init__(self, output_dim=32, **kwargs):
self.output_dim = output_dim
super(MyDense, self).__init__(**kwargs)
def build(self, input_shape):
self.kernel = self.add_weight(name='kernel',
shape=(input_shape[1], self.output_dim),
initializer='uniform',
trainable=True)
super(MyDense, self).build(input_shape)
def get_config(self):
config = {'name': self.name,
'trainable': self.trainable,
'output_dim': self.output_dim
}
if hasattr(self, 'batch_input_shape'):
config['batch_input_shape'] = self.batch_input_shape
if hasattr(self, 'dtype'):
config['dtype'] = self.dtype
return config
def call(self, x):
y = K.dot(x, self.kernel)
return y
def compute_output_shape(self, input_shape):
return (input_shape[0], self.output_dim)
Note that in this example, you must implement a "get_config" method, that converts your object into a dict. Otherwise, Keras will not be able to figure out how to serialize it.
Then, in the "Architecture" tab of your DL algorithm, you can use the "MyDense" layer, and need to register it for DSS to later save it. The architecture code will look like:
from keras.layers import Input, Dense
from keras.models import Model
from my_layers import MyDense
import dataiku.doctor.deep_learning.custom_objects_handler as coh
coh.register_object("MyDense", MyDense)
def build_model(input_shapes, n_classes=None):
inputs = Input(shape=input_shapes['main'], name='main')
x = Dense(128, activation='relu')(inputs)
x = Dense(128, activation='relu')(x)
x = MyDense(64)(x)
predictions = Dense(n_classes, activation="softmax")(x)
model = Model(inputs=inputs, outputs=predictions)
return model
def compile_model(model):
model.compile(optimizer='rmsprop',
loss='binary_crossentropy')
return model
Don't hesitate if you have further questions.
Best regards,
Hello,
We have added the possibility to use custom objects in the visual Deep Learning part of the Visual Machine Learning of DSS in release 5.1.3.
To use it, you need to register your custom object using the custom object handler, and then DSS will handle the serialization/deserialization for you.
For example, you can write a "MyDense" custom layer, that you put in a "my_layers.py" inside the libraries of the project on which you're going to build your DL model. It will look like:
# my_layers.py
from keras import backend as K
from keras.engine.topology import Layer
class MyDense(Layer):
def __init__(self, output_dim=32, **kwargs):
self.output_dim = output_dim
super(MyDense, self).__init__(**kwargs)
def build(self, input_shape):
self.kernel = self.add_weight(name='kernel',
shape=(input_shape[1], self.output_dim),
initializer='uniform',
trainable=True)
super(MyDense, self).build(input_shape)
def get_config(self):
config = {'name': self.name,
'trainable': self.trainable,
'output_dim': self.output_dim
}
if hasattr(self, 'batch_input_shape'):
config['batch_input_shape'] = self.batch_input_shape
if hasattr(self, 'dtype'):
config['dtype'] = self.dtype
return config
def call(self, x):
y = K.dot(x, self.kernel)
return y
def compute_output_shape(self, input_shape):
return (input_shape[0], self.output_dim)
Note that in this example, you must implement a "get_config" method, that converts your object into a dict. Otherwise, Keras will not be able to figure out how to serialize it.
Then, in the "Architecture" tab of your DL algorithm, you can use the "MyDense" layer, and need to register it for DSS to later save it. The architecture code will look like:
from keras.layers import Input, Dense
from keras.models import Model
from my_layers import MyDense
import dataiku.doctor.deep_learning.custom_objects_handler as coh
coh.register_object("MyDense", MyDense)
def build_model(input_shapes, n_classes=None):
inputs = Input(shape=input_shapes['main'], name='main')
x = Dense(128, activation='relu')(inputs)
x = Dense(128, activation='relu')(x)
x = MyDense(64)(x)
predictions = Dense(n_classes, activation="softmax")(x)
model = Model(inputs=inputs, outputs=predictions)
return model
def compile_model(model):
model.compile(optimizer='rmsprop',
loss='binary_crossentropy')
return model
Don't hesitate if you have further questions.
Best regards,