![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Featurization-in-Machine-Learning-in-Apache-Spark-01.jpg\")
<\/a>Apache Spark Machine Learning – Featurization<\/p><\/div>\n
2. Featurization in Apache Spark MLlib<\/h2>\n
Apache Spark MLlib includes algorithms for working with Spark features<\/a>. Moreover, it divided into these groups:<\/span><\/p>\n Let’s learn all Apache Spark MLlib Featurization in detail:<\/p>\n Here, we have 3 types of MLlib in Apache Spark Extractions:<\/span> Extraction in\u00a0Featurization of Apache\u00a0Spark MLlib<\/p><\/div>\n ii. Word2Vec<\/strong> Have a look at Spark RDD<\/a><\/strong> i. Tokenizer<\/strong> Do you know about Structured Streaming in SparkR<\/a><\/strong> Have a look at data type mapping between R and Spark<\/a><\/strong><\/p>\n xi. Interaction<\/b> MinMaxScaler<\/p><\/div>\n It takes 0.0 by default. Lower bound after transformation, shared by all features.<\/span><\/p>\n Featurization It takes \u00a01.0 by default. Upper bound after transformation, shared by all features.<\/span> Let’s revise Spark machine Learning with R<\/a><\/strong> i. VectorSlicer<\/strong> Have a look at Spark RDD features It represents the indices into the vector, setIndices().<\/span><\/p>\n It represents the names of features in the vector, setNames(). Also, requires the vector column to have an AttributeGroup. Since the implementation matches on the name field of an Attribute.<\/span> Do you know about Spark SQL Optimization<\/strong><\/a><\/p>\n It chooses a fixed number of top features according to a chi-squared test. This is akin to yielding the features with the most predictive power. <\/span><\/p>\n It is similar to numTopFeatures but chooses a fraction of all features instead of a fixed number. <\/span><\/p>\n It chooses all features whose p-values are below a threshold, thus controlling the false positive rate of selection. <\/span><\/p>\n It uses the Benjamini-Hochberg procedure to choose all features whose false discovery rate is below a threshold. <\/span><\/p>\n It chooses all features whose p-values are below a threshold.<\/span><\/p>\n An important class of hashing techniques is Locality Sensitive Hashing (LSH). Basically, we use it in following. Such as clustering, approximate nearest neighbor search and outlier detection with large datasets.<\/span> Test yourself with Spark Quiz<\/a><\/strong> Bucketed Random Projection for Euclidean Distanc<\/p><\/div>\n MinHash for Jaccard Distance<\/b>\n
a. Extraction in\u00a0Featurization of Apache\u00a0Spark MLlib<\/h3>\n
\ni. TF-IDF<\/strong>
\nThere is a feature vectorization method widely used in text mining to reflect the importance of a term to a document in the corpus. That is what we call a Term Frequency-Inverse Document Frequency (TF-IDF). Here we are denoting a term by t, also a document by d, whereas the corpus by D. Moreover, Term frequency TF(t,d) is the number of times that term t appears in document d. While document frequency DF(t, D)DF(t, D) is the number of documents that contain term t.<\/span><\/p>\n![\"Apache\u00a0Spark](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Capture.png\")
<\/a>
\nAn estimator that takes sequences of words representing documents and trains a Word2VecModel is Word2Vec. It is the model that maps each word to a unique fixed-size vector. Moreover, Word2VecModel helps to transform each document into a vector using the average of all words in the document. Afterward, this vector can then be used as features for prediction, document similarity calculations and many more.<\/span><\/p>\n
\niii. CountVectorizer<\/b>
\nTo convert a collection of text documents to vectors of token counts, we use CountVectorizer and CountVectorizerModel. We can use CountVectorizer as an estimator to extract the vocabulary when an a-priori dictionary is not available. Also generates a CountVectorizerModel. In addition, this model produces sparse representations for the documents over the vocabulary. Also, that can be passed to other algorithms like LDA.<\/span><\/p>\nb. Feature Transformers in\u00a0Featurization of Apache Spark MLlib<\/h3>\n
\nThe process of taking the text (such as a sentence) and breaking it into individual terms (usually words), is known as Tokenization. Although, it is a functionality which is offered by a simple Tokenizer class.<\/span>
\nii. StopWordsRemover<\/b>
\nThere are some words which are Stop words. It is a compulsion that these words should be excluded from the input. Since the words appear frequently. Also, don\u2019t carry as much meaning.<\/span>
\niii. n-gram<\/b>
\nA sequence of n tokens (typically words) for some integer n is an n-gram. Moreover, we use N-Gram class to transform input features into n-grams.<\/span>
\niv. Binarizer<\/b>
\nThe process of thresholding numerical features to binary (0\/1) features, is Binarization.<\/span><\/p>\n
\nBasically, it takes the common parameters. Such as inputCol, outputCol, and the threshold for binarization. Moreover, the feature values greater than the threshold are binarized to 1.0. Whereas, values equal to or less than the threshold are binarized to 0.0. Although, inputCol support both vector and double types.<\/span>
\nv. PCA<\/b>
\nBasically, it is a statistical procedure. It uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables, that is called as principal components. Moreover, it trains a model to project vectors to a low-dimensional space.<\/span>
\nvi. PolynomialExpansion<\/b>
\nBasically, a process of expanding features into a polynomial space is Polynomial expansion. In addition, it is formulated by an n-degree combination of original dimensions. Moreover, a Polynomial expansion class offers this functionality.<\/span>
\nvii. StringIndexer<\/b>
\nBasically, StringIndexer encodes a string column of labels to a column of label indices. Moreover, the indices are in [0, numLabels), ordered by label frequencies. Hence, the most frequent label gets index 0.<\/span>
\nviii. IndexToString<\/b>
\nAs same as StringIndexer, it also maps a column of label indices back to a column containing the original labels as strings.<\/span>
\nix. OneHotEncoder<\/b>
\nBasically, it maps a column of label indices to a column of binary vectors, with at most a single one-value. Moreover, this encoding allows algorithms. That expect continuous features, such as Logistic Regression, to use categorical features.<\/span>
\nx. VectorIndexer<\/b>
\nIn datasets of vectors, VectorIndexer helps index categorical features. Basically, it can both automatically decide which features are categorical. Also, converts original values to category indices. More specifically, it does all this following:<\/span><\/p>\n\n
\nIt is a type of transformer that takes vector or double-valued columns. Afterwards generates a single vector column. Basically, that contains the product of all combinations of one value from each input column.<\/span>
\nxii. Normalizer<\/b>
\nIt is a type of transformer that transforms a dataset of vector rows, normalizing each vector to have unit norm. Basically, it takes parameter p, that specifies the p-norm used for normalization.<\/span>
\nxiii. StandardScaler<\/b>
\nBasically, StandardScaler transforms a dataset of vector rows. Also, normalizes each feature to have unit standard deviation and\/or zero mean.\u00a0<\/span>
\nxiv. withStd<\/b>
\nIt is true by default. Moreover, it Scales the data to a unit standard deviation.<\/span>
\nxv. withMean<\/b>
\nIt is false by default. Basically, it centers the data with mean before scaling. Also builds a dense output, so take care when applying to sparse input.
\nxvi. MinMaxScaler<\/b>
\nBasically, it transforms a dataset of vector rows. Also, rescales each feature to a specific range (often [0, 1]). It takes parameters:<\/span><\/p>\n![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Capture2.png\")
<\/a>\n
\n
\nxvii. MaxAbsScaler<\/b>
\nIt transforms a dataset of vector rows. Also, rescales each feature to the range [-1, 1] by dividing by the maximum absolute value in each feature. Moreover, it does not shift\/center the data. Therefore, it does not destroy any sparsity.<\/span><\/p>\n
\nxviii. Bucketizer<\/b>
\nBasically, it transforms a column of continuous features to a column of feature buckets. However, the buckets are specified by users.<\/span>
\nxix. ElementwiseProduct<\/b>
\n By using element-wise multiplication, it multiplies each input vector by a provided \u201cweight\u201d vector. Also, we can define it as it scales each column of the dataset by a scalar multiplier. Moreover, this shows the Hadamard product between the input vector, v and transforming vector, w. Thus it yields a result vector.<\/span>
\nxx. SQLTransformer<\/b>
\nBasically, it implements the transformations those are defined by SQL<\/strong> <\/a>statement. Although, recently we only support SQL syntax like “SELECT … FROM __THIS__ …” where “__THIS__”.<\/span>
\nxxi. VectorAssembler<\/b>
\nIt is also a transformer. Basically, it combines a given list of columns into a single vector column. Although, it is useful for combining raw features. Also with features generated by different feature transformers into a single feature vector.\u00a0\u00a0<\/span>
\nxxii. Imputer<\/b>
\nBasically, the Imputer transformer completes missing values in a dataset. Either using the mean or the median of the columns in which the missing values are located. Moreover, the input columns should be of DoubleType or FloatType.<\/span><\/p>\nc. Feature Selectors in\u00a0Featurization Apache\u00a0Spark MLlib<\/h3>\n
\nIt is nothing but a transformer, that takes a feature vector and outputs a new feature vector. Even with a sub-array of the original features. Moreover, it is beneficial for extracting features from a vector column.<\/span><\/p>\n
\n<\/a><\/strong>
\n<\/span>In addition, VectorSlicer accepts a vector column with specified indices. Afterwards, it outputs a new vector column. Basically, whose values are selected via those indices. Moreover, we have two types of indices, such as:<\/span>
\n<\/b><\/span><\/p>\n\n
\n
\nii. RFormula<\/b>
\nBy an R model formula, RFormula selects columns. Basically, we support a limited subset of the R operators. It includes \u2018~\u2019, \u2018.\u2019, \u2018:\u2019, \u2018+\u2019, and \u2018-\u2018. Let\u2019s discuss all the basic operators:<\/span>
\n1. ~ separate target and terms
\n2. + concat terms, \u201c+ 0\u201d means removing intercept
\n3. – remove a term, \u201c- 1\u201d means removing intercept
\n4. : interaction (multiplication for numeric values, or binarized categorical values)
\n5. . all columns except the target
\niii.<\/strong> ChiSqSelector<\/strong>
\nChiSqSelector refers to Chi-Squared feature selection. Basically, it operates on labeled data with categorical features. Moreover, it uses the Chi-Squared test of independence to decide which features to choose. Also supports five selection methods, such as numTopFeatures, percentile, fpr, fdr, fwe. Lets discuss all one by one:<\/span><\/p>\n\n
\n
\n
\n
\n
d. Locality Sensitive Hashing in\u00a0Featurization in Apache Spark\u00a0MLlib<\/h3>\n
\ni. LSH Operations<\/b>
\nLSH can be used for the major types of operations. Basically, for each of these operations, a fitted LSH model has methods.<\/span>
\nFeature Transformation<\/b>
\nTo add hashed values as a new column it is the basic functionality. Moreover, it is very useful for dimensionality reduction. Also, by setting inputCol and outputCol, users can specify input and output column names.<\/span><\/p>\n
\nApproximate Similarity Join<\/b>
\nBasically, it takes two datasets and approximately returns pairs of rows in the datasets whose distance is smaller than a user-defined threshold. Also, supports both joining two different datasets and self-joining. Moreover, self-joining will produce some duplicate pairs also.<\/span>
\nApproximate Nearest Neighbor Search<\/b>
\nBasically, it takes a dataset and a key then returns a specified number of rows in the dataset that are closest to the vector.<\/span>
\nii. LSH Algorithms<\/b>
\nBucketed Random Projection for Euclidean Distance<\/b>
\nFor Euclidean distance, Bucketed Random Projection is an LSH family. We can define Euclidean distance as follows:<\/span><\/p>\n![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Capture5.png\")
<\/a>
\nFor Jaccard distance, MinHash is an LSH family. Where input features are sets of natural numbers. Moreover, Jaccard distance of two sets is the cardinality of their intersection & union<\/p>\n
![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Capture3.png\")
<\/a>![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Capture4.png\")
<\/a>