{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Tensorflow Extended (TFX)\n- In this kernel we try to use Tensorflow data validation library to easily visualize the statistics and given schema,the other main purpose of this tool is to compare train and test statistics (Train -Test Skew)\n- This tensorflow data validation packages uses Facets internally to render interactive visualization\n- This packages was used in production system to validate the data for anomalies before passing into \n    our model (ML/DL model -since model expects data in particular schema)\n- For further information kindly go through the **TFX documentation** \n\n[TFX](https://www.tensorflow.org/tfx)\n\n[Tensorflow Data validation](https://www.tensorflow.org/tfx/data_validation/get_started)\n\n\n*\nDisclaimer: we can analyse the Descriptive nature of the data by rich set of tools already available in the python \necosystem, this kernel purpose is to introduce the new tool (from Tensorflow ecosytem) which can be useful in analysing descriptive nature and production system also .*"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Installing Tensorflow data validation\n- Tfdv uses Apache Beam to full pass the dataset (Big datasets) and for aggregations \n- Apache Beam has some nice characteristics such as \n    - **unfied programing model **for Batch and stream processing\n    - **portable** Execute pipelines on multiple execution environments.\n    - Apache beam supports different language SDKs\n    - Apache beam runs with different backend runners such as \n        - Apache Flink\n        - Apache Spark\n        - Apache Gearpump\n        - Apache Nemo\n        - Apache Samza\n        - Google cloud dataflow\n        - Direct runners (without parallelism in local machines)\n        \n*Note this kernel uses Direct runners this may be slow while getting statistics from the dataset.***"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"!pip install --user tensorflow_data_validation","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow_data_validation as tfdv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generating Statistics\nSatistics is nothing but tfdv full passes the dataset and collects the characteristics (aggregation metrics) of \nthe dataset and use this statistics (Train statistics ) to validate the statistics of the test data in prediction or serving time.\n- Currently, We can prepare Satistics of the dataset in three ways :\n    - Directly from the CSVs\n    - From pandas Dataframe\n    - From Tensorflow specific- TFRecord"},{"metadata":{"trusted":true},"cell_type":"code","source":"dipole_moments=pd.read_csv(\"../input/dipole_moments.csv\")\nmagnetic_shielding_tensors=pd.read_csv(\"../input/magnetic_shielding_tensors.csv\")\nmulliken_charges=pd.read_csv(\"../input/mulliken_charges.csv\")\npotential_energy=pd.read_csv(\"../input/potential_energy.csv\")\nstructures=pd.read_csv(\"../input/structures.csv\")\nscalar_coupling_contributions=pd.read_csv(\"../input/scalar_coupling_contributions.csv\")\n#dipole_moments=pd.read_csv(\"../input/dipole_moments.csv\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#dipole_moments=tfdv.generate_statistics_from_csv('../input/dipole_moments.csv')\n#magnetic_shielding_tensors=tfdv.generate_statistics_from_csv('../input/magnetic_shielding_tensors.csv')\n#mulliken_charges=tfdv.generate_statistics_from_csv('../input/mulliken_charges.csv')\n#potential_energy=tfdv.generate_statistics_from_csv('../input/potential_energy.csv')\n#scalar_coupling_contributions=tfdv.generate_statistics_from_csv('../input/scalar_coupling_contributions.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run any of the above cells to generate statistics, i observed generating statistics from the pandas dataframe was \nbit faster than generating from the CSVs."},{"metadata":{},"cell_type":"markdown","source":"# Visualizing the Generated Statistics in Facets \n- This is as simple as calling **tfdv.visualize_statistics** api"},{"metadata":{"trusted":true},"cell_type":"code","source":"dipole_moments_stats=tfdv.generate_statistics_from_dataframe(dipole_moments)\ntfdv.visualize_statistics(dipole_moments_stats)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"magnetic_shielding_tensors_stats=tfdv.generate_statistics_from_dataframe(magnetic_shielding_tensors)\ntfdv.visualize_statistics(magnetic_shielding_tensors_stats)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mulliken_charges_stats=tfdv.generate_statistics_from_dataframe(mulliken_charges)\ntfdv.visualize_statistics(mulliken_charges_stats)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"potential_energy_stats=tfdv.generate_statistics_from_dataframe(potential_energy)\ntfdv.visualize_statistics(potential_energy_stats)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"structures_stats=tfdv.generate_statistics_from_dataframe(structures)\ntfdv.visualize_statistics(structures_stats)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scalar_coupling_contributions_stats=tfdv.generate_statistics_from_dataframe(scalar_coupling_contributions)\ntfdv.visualize_statistics(scalar_coupling_contributions_stats)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Saving schema for training statistics\n\n- Schema is nothing but the expected features /schema of the given dataset\n    - Schema can be prepared in two ways:\n        - Manually write the schema\n        - USe the generated Satistics to infer schema.\n- You can save/serialize the schema of your training dataset in to the disk and later you load and validate \n    the statistis of the test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"dipole_moments_schema=tfdv.infer_schema(dipole_moments_stats)\ntfdv.write_schema_text(dipole_moments_schema,\"dipole_moments_schema\")\n\nmagnetic_shielding_tensors_schema=tfdv.infer_schema(magnetic_shielding_tensors_stats)\ntfdv.write_schema_text(magnetic_shielding_tensors_schema,\"magnetic_shielding_tensors_schema\")\n\nmulliken_charges_schema=tfdv.infer_schema(mulliken_charges_stats)\ntfdv.write_schema_text(mulliken_charges_schema,\"mulliken_charges_schema\")\n\npotential_energy_schema=tfdv.infer_schema(potential_energy_stats)\ntfdv.write_schema_text(potential_energy_schema,\"potential_energy_schema\")\n\nstructures_schema=tfdv.infer_schema(structures_stats)\ntfdv.write_schema_text(structures_schema,\"structures_schema\")\n\nscalar_coupling_contributions_schema=tfdv.infer_schema(scalar_coupling_contributions_stats)\ntfdv.write_schema_text(scalar_coupling_contributions_schema,\"scalar_coupling_contributions_schema\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Schemas are stored in Output DIR = '.'"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(os.listdir(\".\")),tfdv.load_schema_text('magnetic_shielding_tensors_schema')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Validate Generated Schema\n- Look at the generated schema, and validate the data type of the each feature, if everything is allright you can\nsave the schema and use this schema to validate the test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"tfdv.display_schema(dipole_moments_schema)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tfdv.display_schema(magnetic_shielding_tensors_schema)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tfdv.display_schema(mulliken_charges_schema)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tfdv.display_schema(potential_energy_schema)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tfdv.display_schema(scalar_coupling_contributions_schema)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tfdv.display_schema(structures_schema)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train=pd.read_csv('../input/train.csv')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test=pd.read_csv('../input/test.csv')\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def map_atom_info(df, atom_idx):\n    df = pd.merge(df, structures, how = 'left',\n                  left_on  = ['molecule_name', f'atom_index_{atom_idx}'],\n                  right_on = ['molecule_name',  'atom_index'])\n    \n    df = df.drop('atom_index', axis=1)\n    df = df.rename(columns={'atom': f'atom_{atom_idx}',\n                            'x': f'x_{atom_idx}',\n                            'y': f'y_{atom_idx}',\n                            'z': f'z_{atom_idx}'})\n    return df\n\ntrain = map_atom_info(train, 0)\ntrain = map_atom_info(train, 1)\n\ntest = map_atom_info(test, 0)\ntest = map_atom_info(test, 1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape,test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Lets Generate Statistics for Train and Test datasets \n- This time use all the train features in single dataframe and generate statistics \n- Use the above generated Train statistics to the test datasets to identify anomalies or data deviations"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_stats=tfdv.generate_statistics_from_dataframe(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_stats=tfdv.generate_statistics_from_dataframe(test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_schema=tfdv.infer_schema(train_stats)\ntfdv.write_schema_text(train_schema,'train_schema')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_schema=tfdv.infer_schema(test_stats)\ntfdv.write_schema_text(test_schema,'test_schema')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Validation"},{"metadata":{"trusted":true},"cell_type":"code","source":"anomalies=tfdv.validate_statistics(test_stats,train_schema)\ntfdv.display_anomalies(anomalies)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tfdv.visualize_statistics(lhs_statistics=test_stats, rhs_statistics=train_stats,\n                          lhs_name='TEST_DATASET', rhs_name='TRAIN_DATASET')\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}