{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Variational AutoEncoder to create Embedding of Merchants\n\nIn this notebook, I will use a Variational AutoEncoder (VAE) to create a Merchant Embedding. This information can be used in ML algotithms with higher semantic quality and similarity betweeen Merchants.\n\n* **Introduction**\n    * What is Embedding ?\n    * How to use Merchants Embedding ?\n    * What is Variational autoencoder (VAE)\n* **Data Preparation**\n    * Load Dataset\n    * Data Engineer\n* **Training VAE**\n* **Visualization of latent space**\n"},{"metadata":{"trusted":true,"_uuid":"2c296b84dbd5447249b6efa2cc46ae87a727e22b"},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport missingno as msno\nimport matplotlib.pyplot as plt\nimport os\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d6610f2ef3d38bd2fada9ba1b2f2d81641c5f31a"},"cell_type":"markdown","source":"## Introduction"},{"metadata":{"_uuid":"2bed5727f87575575838d73bc415bf022ba877bc"},"cell_type":"markdown","source":"### What is Embedding ?"},{"metadata":{"_uuid":"8afaebebbcae1800a39d670ffc36f9657beca745"},"cell_type":"markdown","source":"An embedding is a relatively low-dimensional space into which you can translate high-dimensional vectors. Embeddings make it easier to do machine learning on large inputs like sparse vectors representing words. Ideally, an embedding captures some of the semantics of the input by placing semantically similar inputs close together in the embedding space. An embedding can be learned and reused across models."},{"metadata":{"_uuid":"2ae445fe611093e829b104f4ed89c8bf61f8b751"},"cell_type":"markdown","source":"So a natural language modelling technique like Word Embedding is used to map words or phrases from a vocabulary to a corresponding vector of real numbers. As well as being amenable to processing by learning algorithms, this vector representation has two important and advantageous properties:\n\n* **Dimensionality Reduction** — it is a more efficient representation\n* **Contextual Similarity** — it is a more expressive representation"},{"metadata":{"_uuid":"461749db7b643b6f648c6841c531aae0a459bd60"},"cell_type":"markdown","source":"#### How to use Merchants Embedding ?"},{"metadata":{"_uuid":"aa39ef1a905a3cdd52215f5fd572228e1d4cd46d"},"cell_type":"markdown","source":"![](https://www.fast.ai/images/instacart.png)\nhttps://www.fast.ai/2018/04/29/categorical-embeddings/"},{"metadata":{"_uuid":"1c4aa72920044eed422364955d29c83c314e06e8"},"cell_type":"markdown","source":"We can use the Embedding as input of the model, containing a reduced dimensionality but with much semantic information.  The previous example shows the use of product, store and customer embedding for a consumer products cecommendation model."},{"metadata":{"_uuid":"6ab72f1d42083744cc4ae60c8a4cc27b158430a6"},"cell_type":"markdown","source":"This notebook only creates the embeddings of all Merchants for use by ML-Models"},{"metadata":{"_uuid":"ca46e968f0a5ea628bc4f81f54f70adfd94caf2e"},"cell_type":"markdown","source":"### What is Variational autoencoder (VAE)\n\n* https://www.jeremyjordan.me/variational-autoencoders/\n* https://blog.keras.io/building-autoencoders-in-keras.html\n\nA variational autoencoder (VAE) provides a probabilistic manner for describing an observation in latent space. Thus, rather than building an encoder which outputs a single value to describe each latent state attribute, we'll formulate our encoder to describe a probability distribution for each latent attribute."},{"metadata":{"_uuid":"007ec4c010e74020afb25593fe0904a61b3c0378"},"cell_type":"markdown","source":"![](https://www.jeremyjordan.me/content/images/2018/03/Screen-Shot-2018-03-18-at-12.24.19-AM.png)"},{"metadata":{"_uuid":"5beb21f45769f3dbb5d537db95077471c5067e7c"},"cell_type":"markdown","source":"The \"Sample from distributions\" it's owr Embedding Layer. I will encoder all Merchants and take a Embedding Layer."},{"metadata":{"_uuid":"59ad41753fc7bf736aaf50a1e6d47343f8fe1bb5"},"cell_type":"markdown","source":"**How does a variational autoencoder work?**\n\nFirst, an encoder network turns the input samples x into two parameters in a latent space, which we will note *z_mean* and *z_log_sigma*. Then, we randomly sample similar points z from the latent normal distribution that is assumed to generate the data, via *z = z_mean + exp(z_log_sigma) * epsilon*, where epsilon is a random normal tensor. Finally, a decoder network maps these latent space points back to the original input data.\n\nThe parameters of the model are trained via two loss functions: a reconstruction loss forcing the decoded samples to match the initial inputs (just like in our previous autoencoders), and the KL divergence between the learned latent distribution and the prior distribution, acting as a regularization term. You could actually get rid of this latter term entirely, although it does help in learning well-formed latent spaces and reducing overfitting to the training data."},{"metadata":{"_uuid":"6be9276d491265ee3fcb40e56ca40e608451ff80"},"cell_type":"markdown","source":"## Data Preparation\n\nThis session transform the variables from the original dataset, corrects missing values and normalizes the data for training. "},{"metadata":{"trusted":true,"_uuid":"04d0d84e833e39bdc9fb6f84f9eb26ff9a4aa605"},"cell_type":"code","source":"df = pd.read_csv('../input/merchants.csv')\n\nprint(\"Size of the dataframe: \", df.shape); display(df.head(5))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e33b3f4cef0e17f84e4e2d4a2c8cb92edceb8246"},"cell_type":"code","source":"df.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5abbb3e857c96856f19a61a655cd22542f484a16"},"cell_type":"markdown","source":"#### Data Engineer\n\n* Fix a missing values\n* One hot encoder for categorical columns\n* Normalize MinMax"},{"metadata":{"_uuid":"37868042ae8092a78f08a8c835a9411279e962c2"},"cell_type":"markdown","source":"#### Fix a missing values"},{"metadata":{"trusted":true,"_uuid":"424e5c74b00087316406468352bb3506ec4b09da"},"cell_type":"code","source":"# Filter onlu nissing values\nnull_columns=df.columns[df.isnull().any()]\nmsno.bar(df[null_columns])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6dee2764ed54a6db3311b0a1de324c7e8ff7e1b7"},"cell_type":"markdown","source":"The only features that has missing values is the **avg_sales_lag3**, **avg_sales_lag6**, **avg_sales_lag12** and  **category_2**."},{"metadata":{"_uuid":"aa5cb1520b60ecd13c3b005f70187b117f54d58a"},"cell_type":"markdown","source":"Float columns i will put a average value"},{"metadata":{"trusted":true,"_uuid":"3dfc7f4266201b5ce97a173c04dff18c7e41104b"},"cell_type":"code","source":"for c in ['avg_sales_lag3', 'avg_sales_lag6', 'avg_sales_lag12']:\n    df[c] = df[c].fillna(df[c].mean())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0277031f231d2950c4d8087e676da2eb1658d5a8"},"cell_type":"markdown","source":"and the category column, i will put another category"},{"metadata":{"trusted":true,"_uuid":"d0acf723d642d4b67c088e39a989779794da11ff"},"cell_type":"code","source":"# add other category \ndf['category_2'] = df.category_2.fillna(df.category_2.max()+1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3646e32232125340850040b28978cdc8c85d360c"},"cell_type":"code","source":"# replace inf to zero\ndf = df.replace([np.inf, -np.inf], np.nan).fillna(0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c6a15e17812f43446c74b172b30edd74f6ced6e9"},"cell_type":"markdown","source":"#### one hot encoder for categorical columns"},{"metadata":{"trusted":true,"_uuid":"9b49a80ecfb0d7ebd4d0cfd169d23777b1c7415f"},"cell_type":"code","source":"#merchant_group_id\ncategorical_columns = ['merchant_category_id','subsector_id',\n                       'category_1', 'most_recent_sales_range', 'most_recent_purchases_range',\n                       'category_4', 'city_id', 'state_id', 'category_2']\n\ndf_enc = pd.get_dummies(df, columns=categorical_columns)\nprint(df_enc.shape)\ndf_enc.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a0bd7dc0d7fd804d728a6a15a85ba22e219d115"},"cell_type":"markdown","source":"#### Normalize MinMax"},{"metadata":{"trusted":true,"_uuid":"a7a7d7b9019f0f7ca2c430ed3719b938bab9739c"},"cell_type":"code","source":"from sklearn.preprocessing import MinMaxScaler\n\nscaler    = MinMaxScaler()\ndf_values = df_enc.drop('merchant_id', axis=1)\ndf_norm   = scaler.fit_transform(df_values)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"01e0a46081857332a650ef2d623c6a380ab82a13"},"cell_type":"markdown","source":"## Training Variational autoencoder (VAE)\n"},{"metadata":{"trusted":true,"_uuid":"30980f33d2580a79b883dad07a5581dad748b18f"},"cell_type":"code","source":"from keras.layers import Lambda, Input, Dense\nfrom keras.models import Model\nfrom keras.datasets import mnist\nfrom keras.losses import mse, binary_crossentropy\nfrom keras.utils import plot_model\nfrom keras import backend as K\nfrom keras.callbacks import ModelCheckpoint\nfrom keras.layers import Input, Dense, Lambda, Layer, Add, Multiply\nfrom keras.models import Model, Sequential\n\nimport argparse\nimport os","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"664b91c5a4fd3bdbf2fb3e1a273d25d506bbf40f"},"cell_type":"code","source":"# network parameters\noriginal_dim= df_enc.shape[1]-1\ninput_shape = (original_dim, )\nintermediate_dim = int(original_dim/2)\nbatch_size = 128\nlatent_dim = 64\nepochs     = 80\nepsilon_std = 1.0","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b9d58ae1975739516b61861b33561dbd9ff1acf1"},"cell_type":"markdown","source":"#### Build Model\n\nhttps://tiao.io/post/tutorial-on-variational-autoencoders-with-a-concise-keras-implementation/\n\nKeras is awesome. It is a very well-designed library that clearly abides by its guiding principles of modularity and extensibility, enabling us to easily assemble powerful, complex models from primitive building blocks. This has been demonstrated in numerous blog posts and tutorials, in particular, the excellent tutorial on Building Autoencoders in Keras. As the name suggests, that tutorial provides examples of how to implement various kinds of autoencoders in Keras, including the variational autoencoder (VAE)1."},{"metadata":{"trusted":true,"_uuid":"126c49635d33d00f6d1074157fb047fbabaa5247"},"cell_type":"code","source":"class KLDivergenceLayer(Layer):\n\n    \"\"\" Identity transform layer that adds KL divergence\n    to the final model loss.\n    \"\"\"\n\n    def __init__(self, *args, **kwargs):\n        self.is_placeholder = True\n        super(KLDivergenceLayer, self).__init__(*args, **kwargs)\n\n    def call(self, inputs):\n\n        mu, log_var = inputs\n\n        kl_batch = - .5 * K.sum(1 + log_var -\n                                K.square(mu) -\n                                K.exp(log_var), axis=-1)\n\n        self.add_loss(K.mean(kl_batch), inputs=inputs)\n\n        return inputs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b455245aa1d5005b841889aec2016af4633a7a33"},"cell_type":"code","source":"# VAE Architecture\n# * original_dim - Original Input Dimension\n# * intermediate_dim - Hidden Layer Dimension\n# * latent_dim - Latent/Embedding Dimension\ndef vae_arc(original_dim, intermediate_dim, latent_dim):\n    # Decode\n    decoder = Sequential([\n        Dense(intermediate_dim, input_dim=latent_dim, activation='relu'),\n        Dense(original_dim, activation='sigmoid')\n    ])\n\n    # Encode\n    x = Input(shape=(original_dim,))\n    h = Dense(intermediate_dim, activation='relu')(x)\n\n    z_mu = Dense(latent_dim)(h)\n    z_log_var = Dense(latent_dim)(h)\n\n    z_mu, z_log_var = KLDivergenceLayer()([z_mu, z_log_var])\n    z_sigma = Lambda(lambda t: K.exp(.5*t))(z_log_var)\n\n    eps = Input(tensor=K.random_normal(stddev=epsilon_std,\n                                       shape=(K.shape(x)[0], latent_dim)))\n    z_eps = Multiply()([z_sigma, eps])\n    z = Add()([z_mu, z_eps])\n\n    x_pred = decoder(z)\n    \n    return x, eps, z_mu, x_pred","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a21abb027790c58184d446c61cca653d7a5f584"},"cell_type":"markdown","source":"Note this is a valid definition of a Keras loss, which is required to compile and optimize a model. It is a symbolic function that returns a scalar for each data-point in y_true and y_pred. In our example, y_pred will be the output of our decoder network, which are the predicted probabilities, and y_true will be the true probabilities."},{"metadata":{"trusted":true,"_uuid":"4c8625a23a8d51420ad54724a2f41ed0d5f0f06f"},"cell_type":"code","source":"def nll(y_true, y_pred):\n    \"\"\" Negative log likelihood (Bernoulli). \"\"\"\n\n    # keras.losses.binary_crossentropy gives the mean\n    # over the last axis. we require the sum\n    return K.sum(K.binary_crossentropy(y_true, y_pred), axis=-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3304e75a33fc51dc17b95f1e521b0fac6836a8ca"},"cell_type":"code","source":"x, eps, z_mu, x_pred = vae_arc(original_dim, intermediate_dim, latent_dim)\nvae            = Model(inputs=[x, eps], outputs=x_pred)\nvae.compile(optimizer='adam', loss=nll)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bea113ce58a392e385202dcbe9964153d873e534"},"cell_type":"code","source":"vae.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e21818f8561b51cf9abe4ba7508c52413bbb8633"},"cell_type":"markdown","source":"![](https://tiao.io/post/tutorial-on-variational-autoencoders-with-a-concise-keras-implementation/vae_full.svg)"},{"metadata":{"_uuid":"0dfbadb401f0c5056f2923e570fb4fc77d7538dd"},"cell_type":"markdown","source":"#### Training VAE"},{"metadata":{"_uuid":"f3c812faf9883f19e250b17a907c77ef5352fa98"},"cell_type":"markdown","source":"Split dataset and train/test"},{"metadata":{"trusted":true,"_uuid":"5b31f3228db81c863893d6d572ffe3566664e503"},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\n# \nX_train, X_test, y_train, y_test = train_test_split(df_norm, df_norm, \n                                                    test_size=0.33, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"221bdb4ebe4b55b4671fd434135e3f3c588ab38d"},"cell_type":"code","source":"filepath   =\"weights.hdf5\"\ncheckpoint = ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=True, mode='min')\ncallbacks_list = [checkpoint]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8ef831536f09a40fa9923565f40ea0304dd95073"},"cell_type":"code","source":"# train\nhist = vae.fit(X_train, X_train,\n        epochs=epochs,\n        batch_size=batch_size,\n        callbacks=callbacks_list,\n        validation_data=(X_test, X_test))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"901f0e411d302195ee5daf7a47dda8a31213c618"},"cell_type":"code","source":"def plt_hist(hist):\n    # summarize history for loss\n    plt.plot(hist.history['loss'])\n    plt.plot(hist.history['val_loss'])\n    plt.title('model loss')\n    plt.ylabel('loss')\n    plt.xlabel('epoch')\n    plt.legend(['train', 'test'], loc='upper left')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8e73cf84f07f0cea5db6205bb4122ca8f03f2dcb"},"cell_type":"code","source":"plt_hist(hist)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b65947df4cbf1b41680f80b302ed92c799b6d67f"},"cell_type":"markdown","source":"## Visualization of latent space\n\nSince our latent space is not two-dimensional, we will use PCA to reduce dimensionality, so we can use some interesting visualizations that can be made at this point. One is to look at the neighborhoods of different classes in the latent 2D plane:"},{"metadata":{"trusted":true,"_uuid":"acc18c1ddff1f687a8dd5771d67486fea828ee80"},"cell_type":"code","source":"from sklearn.manifold import TSNE\nfrom sklearn.decomposition import PCA\n\ndef plt_reduce(x, color='merchant_category_id'):\n    '''\n    Plot Scatter with color\n    '''\n    plt.figure(figsize=(6, 6))\n    plt.scatter(x[:, 0], x[:, 1], c=df[color],\n            alpha=.4, s=3**2, cmap='viridis')\n    #plt.colorbar()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3908b33023a829897368a38cfd81336d34036ec0"},"cell_type":"code","source":"# Predict Embedding values\nencoder = Model(x, z_mu)\nz_df    = encoder.predict(df_norm, batch_size=batch_size)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5a6d02b9fbf75395551dec721d2df19ca6ddfe2e"},"cell_type":"markdown","source":"#### PCA - Principal Component Analysis"},{"metadata":{"trusted":true,"_uuid":"c768671526d3b426a4be256b81e1d880c19b7bcc"},"cell_type":"code","source":"# Reduce dimmension\npca      = PCA(n_components=2)\nx_reduce = pca.fit_transform(z_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"e23d4d910a3c3f9747360a56da071f5649110ece"},"cell_type":"code","source":"# Plot with merchant_category_id color\nplt_reduce(x_reduce, 'merchant_category_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8528107c3bc1bf5ce8e34563d1ecbec3ee98ddb9"},"cell_type":"code","source":"# Plot with subsector_id color\nplt_reduce(x_reduce, 'subsector_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7cbce08ff963ea318d1d299c0ba733f9af0575ab"},"cell_type":"code","source":"# Plot with city_id color\nplt_reduce(x_reduce, 'city_id')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c990ca440cbc96f7fa39b9a4c548eea805774d69"},"cell_type":"markdown","source":"### Save Embedding\n\nJoin embedding with merchant_id and save pandas"},{"metadata":{"trusted":true,"_uuid":"660a5549c7b4e48bcbb54e633abe21950f3723b2"},"cell_type":"code","source":"df_embedding = pd.DataFrame(z_df)\ndf_embedding['merchant_id'] = df.merchant_id\ndf_embedding.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6dfd31587a551571b5d5ababb0409e18caa7e184"},"cell_type":"code","source":"df_embedding.to_csv('merchant_id_embedding.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"12da8d58bbd36c88d680170d4fd2b464ccd43369"},"cell_type":"markdown","source":"##### continue...."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}