Download Sign Language Project Code<\/h3>\n
Please download the source code of sign language machine learning project: Sign Language Recognition Project<\/strong><\/a><\/p>\n This is divided into 3 parts:<\/p>\n All of which are created as three separate .py files.\u00a0The file structure is given below:<\/p>\n It is fairly possible to get the dataset we need on the internet but in this project, we will be creating the dataset on our own.<\/p>\n We will be having a live feed from the video cam and every frame that detects a hand in the ROI (region of interest) created will be saved in a directory (here gesture directory) that contains two folders train and test, each containing 10 folders containing images captured using the create_gesture_data.py<\/p>\n Directory structure<\/strong><\/p>\n Inside of train (test has the same structure inside)<\/strong><\/p>\n Now for creating the dataset we get the live cam feed using OpenCV and create an ROI that is nothing but the part of the frame where we want to detect the hand in for the gestures.<\/p>\n The red box is the ROI and this window is for getting the live cam feed from the webcam.<\/p>\n <\/p>\n For differentiating between the background we calculate the accumulated weighted avg for the background and then subtract this from the frames that contain some object in front of the background that can be distinguished as foreground.<\/p>\n This is done by calculating the accumulated_weight for some frames (here for 60 frames) we calculate the accumulated_avg for the background.<\/p>\n After we have the accumulated avg for the background, we subtract it from every frame that we read after 60 frames to find any object that covers the background.<\/p>\n (We put up a text using cv2.putText to display to wait and not put any object or hand in the ROI while detecting the background)<\/p>\n Now we calculate the threshold value for every frame and determine the contours using cv2.findContours and return the max contours (the most outermost contours for the object) using the function segment. Using the contours we are able to determine if there is any foreground object being detected in the ROI, in other words, if there is a hand in the ROI.<\/p>\n When contours are detected (or hand is present in the ROI), We start to save the image of the ROI in the train and test set respectively for the letter or number we are detecting it for.<\/p>\n In the above example, the dataset for 1 is being created and the thresholded image of the ROI is being shown in the next window and this frame of ROI is being saved in ..train\/1\/example.jpg<\/p>\n For the train dataset, we save 701 images for each number to be detected, and for the test dataset, we do the same and create 40 images for each number.<\/p>\n Now on the created data set we train a CNN.<\/p>\n First, we load the data using ImageDataGenerator of keras through which we can use the flow_from_directory function to load the train and test set data, and each of the names of the number folders will be the class names for the imgs loaded.<\/p>\n plotImages function is for plotting images of the dataset loaded.<\/p>\n Now we design the CNN as follows (or depending upon some trial and error other hyperparameters can be used)<\/p>\n Now we fit the model and save the model for it to be used in the last module (model_for_gesture.py)<\/p>\n In training callbacks of Reduce LR on plateau and earlystopping is used, and both of them are dependent on the validation dataset loss.<\/p>\n After every epoch, the accuracy and loss are calculated using the validation dataset and if the validation loss is not decreasing, the LR of the model is reduced using the Reduce LR to prevent the model from overshooting the minima of loss and also we are using the earlystopping algorithm so that if the validation accuracy keeps on decreasing for some epochs then the training is stopped.<\/p>\n The example contains the callbacks used, also it contains the two different optimization algorithms used – SGD (stochastic gradient descent, that means the weights are updated at every training instance) and Adam (combination of Adagrad and RMSProp) is used.<\/p>\n We found for the model SGD seemed to give higher accuracies. As we can see while training we found 100% training accuracy and validation accuracy of about 81%<\/p>\nSteps to develop sign language recognition project<\/h3>\n
\n
![\"project](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/09\/file-structure.png\")
<\/a><\/p>\n1. Creating the dataset for sign language detection:<\/h3>\n
![\"directory](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/09\/directory-structure.png\")
<\/a><\/p>\n![\"training\"](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/09\/training.png\")
<\/a><\/p>\n![\"create](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/09\/create-roi-frame.png\")
<\/a><\/p>\nimport tensorflow as tf\r\nfrom tensorflow import keras\r\nfrom keras.models import Sequential\r\nfrom keras.layers import Activation, Dense, Flatten, BatchNormalization, Conv2D, MaxPool2D, Dropout\r\nfrom keras.optimizers import Adam, SGD\r\nfrom keras.metrics import categorical_crossentropy\r\nfrom keras.preprocessing.image import ImageDataGenerator\r\n\r\nimport warnings\r\nimport numpy as np\r\nimport cv2\r\nfrom keras.callbacks import ReduceLROnPlateau\r\nfrom keras.callbacks import ModelCheckpoint, EarlyStopping\r\nwarnings.simplefilter(action='ignore', category=FutureWarning)\r\n\r\n\r\nbackground = None\r\naccumulated_weight = 0.5\r\n\r\n#Creating the dimensions for the ROI...\r\nROI_top = 100\r\nROI_bottom = 300\r\nROI_right = 150\r\nROI_left = 350\r\n\r\n\r\ndef cal_accum_avg(frame, accumulated_weight):\r\n\r\n global background\r\n \r\n if background is None:\r\n background = frame.copy().astype(\"float\")\r\n return None\r\n\r\n cv2.accumulateWeighted(frame, background, accumulated_weight)\r\n<\/pre>\n
![\"fetch](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/09\/fetch-background.png\")
<\/a><\/p>\nCalculate threshold value<\/h4>\n
def segment_hand(frame, threshold=25):\r\n global background\r\n \r\n diff = cv2.absdiff(background.astype(\"uint8\"), frame)\r\n\r\n _ , thresholded = cv2.threshold(diff, threshold,255,cv2.THRESH_BINARY)\r\n\r\n # Grab the external contours for the image\r\n image, contours, hierarchy = cv2.findContours(thresholded.copy(),\r\n cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\r\n\r\n if len(contours) == 0:\r\n return None\r\n else:\r\n \r\n hand_segment_max_cont = max(contours, key=cv2.contourArea)\r\n \r\n return (thresholded, hand_segment_max_cont)\r\n<\/pre>\n
cam = cv2.VideoCapture(0)\r\n\r\nnum_frames = 0\r\nelement = 10\r\nnum_imgs_taken = 0\r\n\r\nwhile True:\r\n ret, frame = cam.read()\r\n\r\n # flipping the frame to prevent inverted image of captured frame...\r\n frame = cv2.flip(frame, 1)\r\n\r\n frame_copy = frame.copy()\r\n\r\n roi = frame[ROI_top:ROI_bottom, ROI_right:ROI_left]\r\n\r\n gray_frame = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)\r\n gray_frame = cv2.GaussianBlur(gray_frame, (9, 9), 0)\r\n\r\n if num_frames < 60:\r\n cal_accum_avg(gray_frame, accumulated_weight)\r\n if num_frames <= 59:\r\n \r\n cv2.putText(frame_copy, \"FETCHING BACKGROUND...PLEASE WAIT\",\r\n(80, 400), cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0,0,255), 2)\r\n \r\n #Time to configure the hand specifically into the ROI...\r\n elif num_frames <= 300: \r\n\r\n hand = segment_hand(gray_frame)\r\n \r\n cv2.putText(frame_copy, \"Adjust hand...Gesture for\" +\r\n str(element), (200, 400), cv2.FONT_HERSHEY_SIMPLEX, 1,\r\n (0,0,255),2)\r\n \r\n # Checking if the hand is actually detected by counting the number\r\n of contours detected...\r\n if hand is not None:\r\n \r\n thresholded, hand_segment = hand\r\n\r\n # Draw contours around hand segment\r\n cv2.drawContours(frame_copy, [hand_segment + (ROI_right,\r\n ROI_top)], -1, (255, 0, 0),1)\r\n \r\n cv2.putText(frame_copy, str(num_frames)+\"For\" + str(element),\r\n (70, 45), cv2.FONT_HERSHEY_SIMPLEX, 1, (0,0,255), 2)\r\n\r\n # Also display the thresholded image\r\n cv2.imshow(\"Thresholded Hand Image\", thresholded)\r\n \r\n else: \r\n \r\n # Segmenting the hand region...\r\n hand = segment_hand(gray_frame)\r\n \r\n # Checking if we are able to detect the hand...\r\n if hand is not None:\r\n \r\n # unpack the thresholded img and the max_contour...\r\n thresholded, hand_segment = hand\r\n\r\n # Drawing contours around hand segment\r\n cv2.drawContours(frame_copy, [hand_segment + (ROI_right,\r\n ROI_top)], -1, (255, 0, 0),1)\r\n \r\n cv2.putText(frame_copy, str(num_frames), (70, 45),\r\n cv2.FONT_HERSHEY_SIMPLEX, 1, (0,0,255), 2)\r\n \r\n cv2.putText(frame_copy, str(num_imgs_taken) + 'images' +\"For\"\r\n + str(element), (200, 400), cv2.FONT_HERSHEY_SIMPLEX, 1,\r\n (0,0,255), 2)\r\n \r\n # Displaying the thresholded image\r\n cv2.imshow(\"Thresholded Hand Image\", thresholded)\r\n if num_imgs_taken <= 300:\r\n cv2.imwrite(r\"D:\\\\gesture\\\\train\\\\\"+str(element)+\"\\\\\" +\r\n str(num_imgs_taken+300) + '.jpg', thresholded)\r\n \r\n else:\r\n break\r\n num_imgs_taken +=1\r\n else:\r\n cv2.putText(frame_copy, 'No hand detected...', (200, 400),\r\n cv2.FONT_HERSHEY_SIMPLEX, 1, (0,0,255), 2)\r\n\r\n\r\n\r\n\r\n # Drawing ROI on frame copy\r\n cv2.rectangle(frame_copy, (ROI_left, ROI_top), (ROI_right,ROI_bottom), (255,128,0), 3)\r\n \r\n cv2.putText(frame_copy, \"DataFlair hand sign recognition_ _ _\", (10, 20), cv2.FONT_ITALIC, 0.5, (51,255,51), 1)\r\n \r\n # increment the number of frames for tracking\r\n num_frames += 1\r\n\r\n # Display the frame with segmented hand\r\n cv2.imshow(\"Sign Detection\", frame_copy)\r\n\r\n # Closing windows with Esc key...(any other key with ord can be used too.)\r\n k = cv2.waitKey(1) & 0xFF\r\n\r\n if k == 27:\r\n break\r\n\r\n# Releasing the camera & destroying all the windows...\r\n\r\ncv2.destroyAllWindows()\r\ncam.release()\r\n<\/pre>\n
![\"dataset](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/09\/dataset-gen-1.png\")
<\/a><\/p>\n![\"training](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/09\/training-dataset.png\")
<\/a><\/p>\n2. Training CNN<\/h3>\n
train_path = r'D:\\gesture\\train'\r\ntest_path = r'D:\\gesture\\test'\r\n\r\ntrain_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.vgg16.preprocess_input).flow_from_directory(directory=train_path, target_size=(64,64), class_mode='categorical', batch_size=10,shuffle=True)\r\ntest_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.vgg16.preprocess_input).flow_from_directory(directory=test_path, target_size=(64,64), class_mode='categorical', batch_size=10, shuffle=True)\r\n<\/pre>\n
imgs, labels = next(train_batches)\r\n\r\n#Plotting the images...\r\ndef plotImages(images_arr):\r\n fig, axes = plt.subplots(1, 10, figsize=(30,20))\r\n axes = axes.flatten()\r\n for img, ax in zip( images_arr, axes):\r\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\r\n ax.imshow(img)\r\n ax.axis('off')\r\n plt.tight_layout()\r\n plt.show()\r\n\r\n\r\nplotImages(imgs)\r\nprint(imgs.shape)\r\nprint(labels)\r\n<\/pre>\n![\"plotimages](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/09\/plotimages-function.png\")
<\/a><\/p>\nmodel = Sequential()\r\n\r\nmodel.add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu', input_shape=(64,64,3)))\r\nmodel.add(MaxPool2D(pool_size=(2, 2), strides=2))\r\n\r\nmodel.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu', padding = 'same'))\r\nmodel.add(MaxPool2D(pool_size=(2, 2), strides=2))\r\n\r\nmodel.add(Conv2D(filters=128, kernel_size=(3, 3), activation='relu', padding = 'valid'))\r\nmodel.add(MaxPool2D(pool_size=(2, 2), strides=2))\r\n\r\nmodel.add(Flatten())\r\n\r\nmodel.add(Dense(64,activation =\"relu\"))\r\nmodel.add(Dense(128,activation =\"relu\"))\r\n#model.add(Dropout(0.2))\r\nmodel.add(Dense(128,activation =\"relu\"))\r\n#model.add(Dropout(0.3))\r\nmodel.add(Dense(10,activation =\"softmax\"))\r\n\r\n<\/pre>\n
![\"model\"](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/09\/model.png\")
<\/a><\/p>\nmodel.compile(optimizer=Adam(learning_rate=0.001), loss='categorical_crossentropy', metrics=['accuracy'])\r\nreduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=1, min_lr=0.0001)\r\nearly_stop = EarlyStopping(monitor='val_loss', min_delta=0, patience=2, verbose=0, mode='auto')\r\n\r\n\r\n\r\nmodel.compile(optimizer=SGD(learning_rate=0.001), loss='categorical_crossentropy', metrics=['accuracy'])\r\nreduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=1, min_lr=0.0005)\r\nearly_stop = EarlyStopping(monitor='val_loss', min_delta=0, patience=2, verbose=0, mode='auto')\r\n<\/pre>\n
![\"model](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/09\/model-compile.png\")
<\/a><\/p>\n![\"sign](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2020\/09\/sign-language-recognition-output.jpg\")
<\/a><\/p>\n