![\"Masking](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/07\/Masking-Padding-in-Keras-DF.jpg\")
<\/a><\/p>\nPadding in Keras<\/h2>\n
To ensure that all the input sequence data is having the same length we pad or truncate the input data points. The deep learning model accepts the input data points of standardized tensors.<\/p>\n
We define the tensors as:<\/p>\n
tensor(batch_size, length_sequence,features)\r\n\r\n<\/pre>\nAll the input to this deep learning model must have length equal lenght_sequence.<\/p>\n
Using padding we pad placeholder values to input sequences if the length of the input is less than length_sequence. If the length of the input sequence is more than length_sequence we truncate the large sequence values from the input.<\/p>\n
This API is available in Keras in the following module.<\/p>\n
keras.preprocessing.sequence.pad_sequence(sample,maxlen,padding,truncating)<\/pre>\nSetting padding=\u2019pre\u2019<\/strong> refers to adding values at the start of index and padding=\u2019post\u2019<\/strong> means adding values at the end. Similarly setting truncating=\u2019post\u2019<\/strong> means removing large values from last and truncating=\u2019pre\u2019<\/strong> means removing values from the beginning.<\/p>\n
Example:<\/p>\n
inputs = [\r\n [123, 600, 51],\r\n [103, 80, 215, 46,546],\r\n [43, 61, 12, 645, 456,345],\r\n]\r\npadding_inputs = keras.preprocessing.sequence.pad_sequences(\r\n inputs, padding=\"post\"\r\n)\r\n\r\nPadded inputs would be:\r\n\r\n [\r\n [123, 600, 51,0,0,0],\r\n [103, 80, 215, 46,546,0],\r\n [43, 61, 12, 645, 456,345],\r\n]\r\n<\/pre>\nMasking in Keras<\/h2>\n
The concept of masking is that we can not train the model on padded values. The placeholder value subset of the input sequence can not be ignored and must be informed to the system. This technique to recognize and ignore padded values is called Masking in Keras.<\/p>\n
We can perform masking in Keras in the following two ways:<\/p>\n
1. Using Keras.layers.Masking layer<\/strong><\/p>\n
Before adding a masking layer the input NumPy array needs to be converted to the tensor data type. This is done using TensorFlow\u2019s convert_to_tensor method.<\/p>\n
from keras.layers import Masking\r\nimport numpy as np\r\nimport tensorflow as tf\r\n\r\nx_train=np.expand_dims(padding_input,-1)\r\n\r\nx_train=tf.convert_to_tensor(x_train,dtype=\u2019float32\u2019)\r\n\r\nmask_layer=Masking(mask_value=0.0)\r\nmask_x_train=mask_layer(x_train)\r\n<\/pre>\n2. Using keras.layers.Embedding layer<\/strong><\/p>\n
Here we configure mask_zero = True in the Embedding layer for masking.<\/p>\n
from keras.layers import Embedding\r\n\r\nembed=Embedding(input_dim=1000,output_dim=32,mask_zero=True)\r\nmasking_output=embed(padding_input)\r\n<\/pre>\nSo, using masking and padding we can efficiently process the input sequence for training a deep learning model where the input sequence is of varying length. Generally, RNN and LSTM require varying length input sequence training.<\/p>\n
Keras Mask Propagation in Functional and Sequential API<\/h2>\n
We generate masks using Embedding or Masking Layer, this mask is then propagated through the neural network. Keras fetches the mask with respect to the input and passes it to another layer. This mask propagation happens in both Functional API and Sequential API.<\/p>\n
Passing Mask Tensors<\/h2>\n
This sector describes how you can directly pass masks to layers. We do it\u00a0using the compute_mask() method. This method takes the tensors and previous masks as input. To directly pass the mask, we pass the output of this method to the __call__ method.<\/p>\n
from keras.layers import Layer,Embedding,LSTM\r\nclass NewLayer(Layer):\r\n def __init__(self, **kwargs):\r\n super(NewLayer, self).__init__(**kwargs)\r\n self.embedding =Embedding(input_dim=5000, output_dim=16, mask_zero=True)\r\n self.lstm = LSTM(32)\r\n\r\n def call(self, inputs):\r\n m = self.embedding(inputs)\r\n compute_mask = self.embedding.compute_mask(inputs)\r\n output_mask = self.lstm(x, mask=compute_mask)\r\n return output_mask\r\n\r\n\r\nlayer = NewLayer()\r\nm = np.random.random((32, 10)) * 100\r\nm = m.astype(\"int32\")\r\nlayer(m)\r\n<\/pre>\nSupporting Masking in Custom Layer<\/h2>\n
This section will describe how to write the layers to generate a mask and layers to update the present mask. To support masking in the custom layer, we have to implement the compute_mask function while writing the custom layer.<\/p>\n
First, let us see how to generate a mask.<\/p>\n
from keras.layers import Layer\r\n\r\nClass my_layer(Layer):\r\n def __init__(self, input_dim, output_dim, mask_zero=False, **kwargs):\r\n super(my_layer, self).__init__(**kwargs)\r\n self.input_dim = input_dim\r\n self.output_dim = output_dim\r\n self.mask_zero = mask_zero\r\n\r\n def build(self, input_shape):\r\n self.embeddings = self.add_weight(\r\n shape=(self.input_dim, self.output_dim),\r\n initializer=\"random_normal\",\r\n dtype=\"float32\",\r\n )\r\n\r\n def call(self, inputs):\r\n return tf.nn.embedding_lookup(self.embeddings, inputs)\r\n\r\n def compute_mask(self, inputs, mask=None):\r\n if not self.mask_zero:\r\n return None\r\n return tf.not_equal(inputs, 0)\r\n\r\nmy_layer=my_layer(15,64)\r\ncompute_mask=compute_mask(input_layer)\r\n<\/pre>\nHere my_layer is generating a mask from the input layer and input values. Below code shows how to write a layer to modify mask.<\/p>\n
from keras.layer import Layer\r\nClass my_layer(Layer):\r\n def call(self, inputs):\r\n return tf.split(inputs, 2, axis=1)\r\n\r\n def compute_mask(self, inputs, mask=None):\r\n \r\n if mask is None:\r\n return None\r\n return tf.split(mask, 2, axis=1)\r\nret_one,ret_two=my_layer()(embeddings)\r\n\r\n<\/pre>\nKeras Masking in Custom Layer<\/h2>\n
This section describes how to generate or modify masks within your custom layer. For this purpose, we need to define the compute_mask function within our custom layer class. The custom_mask method will create a new mask using the input tensors and the current mask.<\/p>\n
from keras.layers import Layer\r\nClass CustomLayer(Layer):\r\n\r\n def call(self, inputs):\r\n return tf.split(inputs, 2, axis=1)\r\n\r\n def compute_mask(self, inputs, mask=None):\r\n if mask is None:\r\n return None\r\n return tf.split(mask, 2, axis=1)\r\n\r\n\r\none, two=CustomLayer()(previous_mask)\r\n<\/pre>\nMask Propagation on Compatible Layers<\/h2>\n
This section explains how to propagate current input masks in a custom layer. To do that, we set support_masking value equal to true in the constructor of the custom layer class. By doing so, the current mask is just passed to the next layer.<\/p>\n
Example for mask propagation in custom layer:<\/p>\n
from keras.layers import Layer\r\nclass NewLayer(Layer):\r\n def __init__(self, **kwargs):\r\n super(NewLayer, self).__init__(**kwargs)\r\n self.supports_masking = True\r\n\r\n def call(self, inputs):\r\n return tf.nn.relu(inputs)\r\n<\/pre>\nWriting Layers with Mask Information<\/h2>\n
Some custom layers accept mask arguments in their call method inside a custom class. We can propagate masks to the next layer by simply adding mask equal None in call method.<\/p>\n
Example:<\/p>\n
from keras.layers import Layer\r\nClass CustomLayer(Layer):\r\n def call(self, inputs, mask=None):\r\n mask_initiation= tf.expand_dims(tf.cast(mask, \"float32\"), -1)\r\n mask_exp = tf.exp(inputs) * mask_initiation\r\n mask_sum = tf.reduce_sum(inputs * mask_initiation, axis=1, keepdims=True)\r\n return mask_exp \/ mask_sum\r\n<\/pre>\nSummary<\/h2>\n
In this article, we talked about masking and padding in keras and its implementation. Padding technique is useful to convert the input sequence to a constant size. In masking, we mask the values which were added during padding for not letting the model get trained on padded data.<\/p>\n
We perform Padding using keras.preprocessing.sequence.pad_sequence API in Keras. While there are two ways for masking, either using the Masking layer (keras.layers.Making) or by using Embedding Layer (keras.layers.Embedding). This article then explains the topics of mask propagation, masking in custom layers, and layers with mask information.<\/p>\n","protected":false},"excerpt":{"rendered":"
In Keras, we create neural networks either using function API or sequential API. In both the APIs, it is very difficult to prepare data for the input layer to model, especially for RNN and...<\/p>\n","protected":false},"author":10,"featured_media":79549,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22185],"tags":[22597,22710,22711,22712],"class_list":["post-79531","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-keras","tag-keras-tutorial","tag-masking-and-padding-in-keras","tag-masking-in-keras","tag-padding-in-keras"],"yoast_head":"\n
Masking and Padding in Keras - DataFlair<\/title>\n<meta name=\"description\" content=\"Masking and padding in Keras - Learn masking and padding techniques in Keras and how to implement them. Learn ways of Keras masking with examples.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Masking and Padding in Keras - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Masking and padding in Keras - Learn masking and padding techniques in Keras and how to implement them. Learn ways of Keras masking with examples.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2020-07-21T13:59:55+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-08-25T08:19:14+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/07\/Masking-Padding-in-Keras-DF.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Masking and Padding in Keras - DataFlair","description":"Masking and padding in Keras - Learn masking and padding techniques in Keras and how to implement them. Learn ways of Keras masking with examples.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/","og_locale":"en_US","og_type":"article","og_title":"Masking and Padding in Keras - DataFlair","og_description":"Masking and padding in Keras - Learn masking and padding techniques in Keras and how to implement them. Learn ways of Keras masking with examples.","og_url":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2020-07-21T13:59:55+00:00","article_modified_time":"2021-08-25T08:19:14+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/07\/Masking-Padding-in-Keras-DF.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/a90b082e16aa38d207212d22b0581f33"},"headline":"Masking and Padding in Keras","datePublished":"2020-07-21T13:59:55+00:00","dateModified":"2021-08-25T08:19:14+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/"},"wordCount":818,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/07\/Masking-Padding-in-Keras-DF.jpg","keywords":["keras tutorial","Masking and Padding in Keras","Masking in keras","Padding in keras"],"articleSection":["Keras Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/","url":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/","name":"Masking and Padding in Keras - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/07\/Masking-Padding-in-Keras-DF.jpg","datePublished":"2020-07-21T13:59:55+00:00","dateModified":"2021-08-25T08:19:14+00:00","description":"Masking and padding in Keras - Learn masking and padding techniques in Keras and how to implement them. Learn ways of Keras masking with examples.","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/07\/Masking-Padding-in-Keras-DF.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/07\/Masking-Padding-in-Keras-DF.jpg","width":1200,"height":628,"caption":"Masking & Padding in Keras"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/masking-and-padding-in-keras\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Keras Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/keras\/"},{"@type":"ListItem","position":3,"name":"Masking and Padding in Keras"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/a90b082e16aa38d207212d22b0581f33","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/dd6de0d647a0185cd6faf264e4ba860b0d85d08d7070766f9cd41bea5bb0b227?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/dd6de0d647a0185cd6faf264e4ba860b0d85d08d7070766f9cd41bea5bb0b227?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/dd6de0d647a0185cd6faf264e4ba860b0d85d08d7070766f9cd41bea5bb0b227?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team is passionate about delivering top-notch tutorials and resources on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. With expertise in the tech industry, we simplify complex topics to help learners excel. Stay updated with our latest insights.","url":"https:\/\/data-flair.training\/blogs\/author\/dfadteam1\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/79531","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=79531"}],"version-history":[{"count":2,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/79531\/revisions"}],"predecessor-version":[{"id":79550,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/79531\/revisions\/79550"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/79549"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=79531"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=79531"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=79531"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}