{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# An exploration of the Meta Kaggle data to see how the tables join together\nThe [Meta Kaggle](https://www.kaggle.com/datasets/kaggle/meta-kaggle) dataset is suggested as part of the data to use for [2023 Kaggle AI Report](https://www.kaggle.com/competitions/2023-kaggle-ai-report) competition.\nWritten as I started to explore Meta Kaggle for that competition; and, I thought it would be helpful to publish it here.","metadata":{"papermill":{"duration":0.01783,"end_time":"2023-05-20T14:59:36.933243","exception":false,"start_time":"2023-05-20T14:59:36.915413","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## How do the Meta Kaggle CSV file-based tables join together? \n I first noticed that there seems to be very little information about how these tables relate to each other and join up, such as a schema or ERD. So, I had a go at working that out - a jigsaw puzzle to solve. Then I will reuse this data model when I explore and filter the data down for the AI report analysis.\n \n ## Get all the table names and the columns in each table.  \n Load just the first row of all the CSV tables for get the data fields for all the tables.  ","metadata":{"papermill":{"duration":0.017863,"end_time":"2023-05-20T14:59:36.969315","exception":false,"start_time":"2023-05-20T14:59:36.951452","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# imports\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport os\nimport seaborn as sns\nimport networkx as nx\nimport plotly.graph_objs as go\nfrom glob import glob \n\n# change this to a local path to the data\ndata_dir_root = '/kaggle/input/'","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":1.569304,"end_time":"2023-05-20T14:59:36.896809","exception":false,"start_time":"2023-05-20T14:59:35.327505","status":"completed"},"tags":[],"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-05-23T20:28:28.704338Z","iopub.execute_input":"2023-05-23T20:28:28.705051Z","iopub.status.idle":"2023-05-23T20:28:29.319139Z","shell.execute_reply.started":"2023-05-23T20:28:28.705012Z","shell.execute_reply":"2023-05-23T20:28:29.318310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# load all the files to read the columns availabe in each dataset\nfields_df = pd.DataFrame()    \n# get all columns\nall_cols = []\nfor csv in glob(f'{data_dir_root}meta-kaggle/*.csv'):\n    file = csv.split('/')[-1].replace('.csv', '')\n    df = pd.read_csv(csv, nrows=1).fillna(123456)\n    df['dataset'] = file\n    fields_df = pd.concat([fields_df, df], axis=0)\n# redo the index\nfields_df.reset_index(drop=True, inplace=True)\nprint(\"fields_df shape: \" , fields_df.shape)\n# check the result\nfields_df.head() ","metadata":{"papermill":{"duration":0.959724,"end_time":"2023-05-20T14:59:37.948092","exception":false,"start_time":"2023-05-20T14:59:36.988368","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-23T20:28:30.446447Z","iopub.execute_input":"2023-05-23T20:28:30.446870Z","iopub.status.idle":"2023-05-23T20:28:30.926102Z","shell.execute_reply.started":"2023-05-23T20:28:30.446833Z","shell.execute_reply":"2023-05-23T20:28:30.924996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are many {label}Id columns that might be the primary or foreign key to join the tables. Next, get a list of all the {label}Id columns in the tables.","metadata":{"papermill":{"duration":0.018323,"end_time":"2023-05-20T14:59:37.985049","exception":false,"start_time":"2023-05-20T14:59:37.966726","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Find Potential Keys","metadata":{}},{"cell_type":"code","source":"potential_keys = [col for col in fields_df.columns if'Id' in col]\npotential_keys","metadata":{"papermill":{"duration":0.031344,"end_time":"2023-05-20T14:59:38.035063","exception":false,"start_time":"2023-05-20T14:59:38.003719","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-23T20:28:33.658791Z","iopub.execute_input":"2023-05-23T20:28:33.659168Z","iopub.status.idle":"2023-05-23T20:28:33.666782Z","shell.execute_reply.started":"2023-05-23T20:28:33.659139Z","shell.execute_reply":"2023-05-23T20:28:33.665769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"That is a long list! I will create a dataframe with just the keys and the dataset names. Then see if there are multiple datasets with the same keys.","metadata":{"papermill":{"duration":0.01878,"end_time":"2023-05-20T14:59:38.073063","exception":false,"start_time":"2023-05-20T14:59:38.054283","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Create a keys only datafram","metadata":{}},{"cell_type":"code","source":"# get the subset of keys\nkeys_df = fields_df[potential_keys].copy()\nkeys_df.head()","metadata":{"papermill":{"duration":0.058288,"end_time":"2023-05-20T14:59:38.150450","exception":false,"start_time":"2023-05-20T14:59:38.092162","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-23T20:28:35.131674Z","iopub.execute_input":"2023-05-23T20:28:35.132064Z","iopub.status.idle":"2023-05-23T20:28:35.159766Z","shell.execute_reply.started":"2023-05-23T20:28:35.132032Z","shell.execute_reply":"2023-05-23T20:28:35.159062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"How many datasets have the same keys?","metadata":{"papermill":{"duration":0.01954,"end_time":"2023-05-20T14:59:38.189417","exception":false,"start_time":"2023-05-20T14:59:38.169877","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Count up the datasets with the same key fields","metadata":{}},{"cell_type":"code","source":"keys_df.info(show_counts=True)","metadata":{"papermill":{"duration":0.051817,"end_time":"2023-05-20T14:59:38.260791","exception":false,"start_time":"2023-05-20T14:59:38.208974","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-23T20:28:36.364638Z","iopub.execute_input":"2023-05-23T20:28:36.365693Z","iopub.status.idle":"2023-05-23T20:28:36.380224Z","shell.execute_reply.started":"2023-05-23T20:28:36.365650Z","shell.execute_reply":"2023-05-23T20:28:36.379228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So the Id column is in every table and is probably a key and a row index; there are many {label}Id columns with only one occurrence, so they might be for other tables that do not exist in the extracted data, or they join to a key in another table that has a different name, e.g. ParentTagId might join with the other TagId's. UserId connects the highest number of tables. Which datasets are linked by the {label}Id columns that occur in multiple datasets? It will be easier with a visual map to navigate the data.","metadata":{"papermill":{"duration":0.021468,"end_time":"2023-05-20T14:59:38.302294","exception":false,"start_time":"2023-05-20T14:59:38.280826","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Create a heatmap to visualise the dataset joins","metadata":{}},{"cell_type":"code","source":"# get the dataset names\nkeys_df.index = fields_df.dataset\n# make everything easier to read\nkeys_df.sort_index(inplace=True)\nkeys_df.sort_index(inplace=True, axis=1)\n# drop the Id col\nkeys_df.drop('Id', axis=1, inplace=True)\n# replace the data with 1s and 0s\nkeys_df[keys_df > 0] = 1\nkeys_df.fillna(0, inplace=True)\n# check the result\nkeys_df.head()","metadata":{"papermill":{"duration":0.06893,"end_time":"2023-05-20T14:59:38.392143","exception":false,"start_time":"2023-05-20T14:59:38.323213","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-23T20:28:37.667192Z","iopub.execute_input":"2023-05-23T20:28:37.667548Z","iopub.status.idle":"2023-05-23T20:28:37.700960Z","shell.execute_reply.started":"2023-05-23T20:28:37.667519Z","shell.execute_reply":"2023-05-23T20:28:37.699977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# sum up the Id's to show which are used in multiple datasets\nfor col in keys_df.columns:\n    keys_df[col] = np.where(keys_df[col] > 0, keys_df[col].sum(), keys_df[col])","metadata":{"papermill":{"duration":0.053486,"end_time":"2023-05-20T14:59:38.465909","exception":false,"start_time":"2023-05-20T14:59:38.412423","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-23T20:28:38.085304Z","iopub.execute_input":"2023-05-23T20:28:38.085696Z","iopub.status.idle":"2023-05-23T20:28:38.115109Z","shell.execute_reply.started":"2023-05-23T20:28:38.085660Z","shell.execute_reply":"2023-05-23T20:28:38.114374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot the heatmap\nsns.set(rc={'figure.figsize':(14,10)})\nhm = sns.heatmap(keys_df, cmap=\"Blues\", cbar=False,linewidths=5)\nhm.set_xticklabels(hm.get_xticklabels(), rotation=90, horizontalalignment='center')\nplt.title('Which datasets are linked by which Id columns')\nplt.show()","metadata":{"papermill":{"duration":1.380858,"end_time":"2023-05-20T14:59:39.866818","exception":false,"start_time":"2023-05-20T14:59:38.485960","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-05-23T20:28:38.525547Z","iopub.execute_input":"2023-05-23T20:28:38.525971Z","iopub.status.idle":"2023-05-23T20:28:39.665485Z","shell.execute_reply.started":"2023-05-23T20:28:38.525935Z","shell.execute_reply":"2023-05-23T20:28:39.664450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The heatmap begins to show the relationships; however, Users, Organizations, KernelLanguages do not appear to have any specific {label}Id's to join on. Therefore the basic unlabelled Id column will probably be used to join with some of the {label}Id columns. I note that the {label}Id often contains the dataset name, i.e. Users -> UserId, Tags -> TagId, Datasets -> DatasetId etc. So let's look at the Users and see if it is the case for the Users.Id is it the matching key to UserId in the other tables?","metadata":{"papermill":{"duration":0.022657,"end_time":"2023-05-20T14:59:39.912842","exception":false,"start_time":"2023-05-20T14:59:39.890185","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### Does the Dataset name combine with Id to make a key?","metadata":{}},{"cell_type":"code","source":"# load the users data and User Organisations data\nusers_df = pd.read_csv(f'{data_dir_root}meta-kaggle/Users.csv')\nuser_orgs_df = pd.read_csv(f'{data_dir_root}meta-kaggle/UserOrganizations.csv')\nuser_followers_df = pd.read_csv(f'{data_dir_root}meta-kaggle/UserFollowers.csv')\nusers_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-23T20:28:39.817397Z","iopub.execute_input":"2023-05-23T20:28:39.817790Z","iopub.status.idle":"2023-05-23T20:29:03.603764Z","shell.execute_reply.started":"2023-05-23T20:28:39.817755Z","shell.execute_reply":"2023-05-23T20:29:03.602679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"user_orgs_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-23T20:29:03.605419Z","iopub.execute_input":"2023-05-23T20:29:03.606116Z","iopub.status.idle":"2023-05-23T20:29:03.616295Z","shell.execute_reply.started":"2023-05-23T20:29:03.606084Z","shell.execute_reply":"2023-05-23T20:29:03.615210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get my user id\nusers_df[users_df['UserName'] == 'anthony35813']","metadata":{"execution":{"iopub.status.busy":"2023-05-23T20:29:03.617863Z","iopub.execute_input":"2023-05-23T20:29:03.618773Z","iopub.status.idle":"2023-05-23T20:29:04.510030Z","shell.execute_reply.started":"2023-05-23T20:29:03.618742Z","shell.execute_reply":"2023-05-23T20:29:04.508957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# see what I get out of the organisation data using Id mapped to UserId\nuser_orgs_df[user_orgs_df['UserId'] == 2438504]","metadata":{"execution":{"iopub.status.busy":"2023-05-23T20:29:04.512218Z","iopub.execute_input":"2023-05-23T20:29:04.512609Z","iopub.status.idle":"2023-05-23T20:29:04.522258Z","shell.execute_reply.started":"2023-05-23T20:29:04.512580Z","shell.execute_reply":"2023-05-23T20:29:04.521154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# what do I get out of the followers data using Id mapped to UserId\nlen(user_followers_df[user_followers_df['UserId'] == 2438504])","metadata":{"execution":{"iopub.status.busy":"2023-05-23T20:29:04.523680Z","iopub.execute_input":"2023-05-23T20:29:04.524001Z","iopub.status.idle":"2023-05-23T20:29:04.541836Z","shell.execute_reply.started":"2023-05-23T20:29:04.523976Z","shell.execute_reply":"2023-05-23T20:29:04.540528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using myself as an example, the UserId matches up to Users.Id in the Users table. It didn't return an organisation, and that is correct as I have yet to enter one, and I am following 12 people. So, I think the Id column in the datasets can be renamed with the pattern {Dataset(-s)}Id, then redraw the mapping to see if a better picture of the relationships emerges.","metadata":{}},{"cell_type":"markdown","source":"### Redraw the datasets and keys relationships heatmap  ","metadata":{}},{"cell_type":"code","source":"fields_df = pd.DataFrame()\n# get all columns\nall_cols = []\nfor csv in glob(f'{data_dir_root}meta-kaggle/*.csv'):\n    file = csv.split('/')[-1].replace('.csv', '')\n    df = pd.read_csv(csv, nrows=1).fillna(123456)\n    df['dataset'] = file\n    # rename the Id column with the dataset name\n    df.rename(columns={'Id' : f'{file[:-1]}Id'}, inplace=True)\n    fields_df = pd.concat([fields_df, df], axis=0)\n# redo the index\nfields_df.reset_index(drop=True, inplace=True)\npotential_keys = [col for col in fields_df.columns if'Id' in col]\n# get the subset of keys\nkeys_df = fields_df[potential_keys].copy()\n# get the dataset names\nkeys_df.index = fields_df.dataset\n# make everything easier to read\nkeys_df.sort_index(inplace=True)\nkeys_df.sort_index(inplace=True, axis=1)\n# replace the data with 1s and 0s\nkeys_df[keys_df > 0] = 1\nkeys_df.fillna(0, inplace=True)\n# sum up the Id's to show which are used in multiple datasets\nfor col in keys_df.columns:\n    keys_df[col] = np.where(keys_df[col] > 0, keys_df[col].sum(), keys_df[col])","metadata":{"execution":{"iopub.status.busy":"2023-05-23T20:29:04.543869Z","iopub.execute_input":"2023-05-23T20:29:04.544281Z","iopub.status.idle":"2023-05-23T20:29:05.058115Z","shell.execute_reply.started":"2023-05-23T20:29:04.544242Z","shell.execute_reply":"2023-05-23T20:29:05.057094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot the heatmap\nsns.set(rc={'figure.figsize':(18,10)})\nhm = sns.heatmap(keys_df, cmap=\"Blues\", cbar=False,linewidths=5)\nhm.set_xticklabels(hm.get_xticklabels(), rotation=90, horizontalalignment='center')\nplt.title('Which datasets are linked by which Id columns.\\nVersion 2 Id renamed to dataset name')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-23T20:29:05.059544Z","iopub.execute_input":"2023-05-23T20:29:05.059955Z","iopub.status.idle":"2023-05-23T20:29:06.470044Z","shell.execute_reply.started":"2023-05-23T20:29:05.059918Z","shell.execute_reply":"2023-05-23T20:29:06.468928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thats better. Every dataset has at least one connection to another now to work with. UserId, KernelVersionId, ForumId and CompetitionId are the most connected, which makes sense.","metadata":{}},{"cell_type":"markdown","source":"### Output the map to CSV for reuse  ","metadata":{}},{"cell_type":"code","source":"## Output the keys to a csv file\nkeys_df.to_csv('keys.csv', index=True)","metadata":{"execution":{"iopub.status.busy":"2023-05-23T20:29:06.471174Z","iopub.execute_input":"2023-05-23T20:29:06.471481Z","iopub.status.idle":"2023-05-23T20:29:06.479906Z","shell.execute_reply.started":"2023-05-23T20:29:06.471455Z","shell.execute_reply":"2023-05-23T20:29:06.478971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## What are the available fields of data?  \nI'm more interested in the data that can be joined and accessed rather than the keys themselves. So what are the data fields available in all of the tables?","metadata":{}},{"cell_type":"code","source":"# get the fields not in the potential keys\nnon_keys = [col for col in fields_df.columns if col not in potential_keys]\ndata_df = fields_df[non_keys].copy()\ndata_df.info(verbose=True, show_counts=True)","metadata":{"execution":{"iopub.status.busy":"2023-05-23T20:29:06.481320Z","iopub.execute_input":"2023-05-23T20:29:06.481898Z","iopub.status.idle":"2023-05-23T20:29:06.508057Z","shell.execute_reply.started":"2023-05-23T20:29:06.481861Z","shell.execute_reply":"2023-05-23T20:29:06.506860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One hundred and twenty columns of data in the Meta Kaggle dataset (I created the dataset field at the top, which is not in the original). Most will be irrelevant to the focus of the AI report, but there are some exciting fields in there too!  ","metadata":{}},{"cell_type":"markdown","source":"## Using a graph network to visualise the entity relationships between the tables  \nThe relationships heatmap was a good start, but I want to show the data connections in a visualisation that is easier to explore and build on as more connections are discovered. For example, it could be an entity relationship diagram. However, a  graph network may provide more functionality later to see how the data link together and to search for other connections. Therefore I will use NetworkX to draw a graph of the relationships between the data. Then Plotly to display the network graph; Plotly helps enable interaction with the graph, i.e. zooming in and out as it becomes more complex.","metadata":{}},{"cell_type":"markdown","source":"### Build the graph","metadata":{}},{"cell_type":"code","source":"# create a graph\nMeta_G = nx.Graph()\n# add the nodes and edges from the keys_df\nfor col in keys_df.columns:\n    Meta_G.add_node(col)\n    for row in keys_df.index:\n        if keys_df.loc[row, col] > 0:\n            Meta_G.add_node(row)\n            Meta_G.add_edge(col, row)\n","metadata":{"execution":{"iopub.status.busy":"2023-05-23T20:29:06.510539Z","iopub.execute_input":"2023-05-23T20:29:06.510871Z","iopub.status.idle":"2023-05-23T20:29:06.544972Z","shell.execute_reply.started":"2023-05-23T20:29:06.510838Z","shell.execute_reply":"2023-05-23T20:29:06.543655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# add the other nodes and edges from the data_df for each dataset id\n# change the row index to the dataset name\ndata_df.index = fields_df.dataset\n# drop the dataset column\ndata_df.drop('dataset', axis=1, inplace=True)\n# fill all the NaN's with a string to avoid float errors\ndata_df.fillna('Null!', inplace=True)\n# add the nodes and edges from the data_df connected to the dataset name\nfor col in data_df.columns:\n    Meta_G.add_node(col)\n    for row in data_df.index:\n        if data_df.loc[row, col] != 'Null!':\n            Meta_G.add_node(row)\n            Meta_G.add_edge(col, row)\n    ","metadata":{"execution":{"iopub.status.busy":"2023-05-23T20:29:06.546368Z","iopub.execute_input":"2023-05-23T20:29:06.547013Z","iopub.status.idle":"2023-05-23T20:29:06.617726Z","shell.execute_reply.started":"2023-05-23T20:29:06.546971Z","shell.execute_reply":"2023-05-23T20:29:06.616853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Draw the Graph  ","metadata":{}},{"cell_type":"code","source":"# plot the graph\ndef plot_graph(Graph):\n    \"\"\"plot the graph with plotly\"\"\"\n    pos = nx.kamada_kawai_layout(Graph)\n\n    # create the edge and node traces\n    edge_x = []\n    edge_y = []\n\n    for edge in Meta_G.edges():\n        x0, y0 = pos[edge[0]]\n        x1, y1 = pos[edge[1]]\n        edge_x.extend((x0, x1, None))\n        edge_y.extend((y0, y1, None))\n\n    edge_trace = go.Scatter(\n        x=edge_x, y=edge_y,\n        line=dict(width=0.5, color='#888'),\n        hoverinfo='none',\n        mode='lines')\n\n    node_x = []\n    node_y = []\n    for node in Meta_G.nodes():\n        x, y = pos[node]\n        node_x.append(x)\n        node_y.append(y)\n\n    node_trace = go.Scatter(\n        x=node_x, y=node_y,\n        mode='markers',\n        hoverinfo='text',\n        marker=dict(\n            showscale=True,\n            colorscale='Blues_r',\n            reversescale=True,\n            color=[],\n            size=20,\n            colorbar=dict(\n                thickness=15,\n                title='Node Connections',\n                xanchor='left',\n                titleside='right'\n            ),\n            line_width=2))\n\n    # label the nodes with there names\n    node_adjacencies = []\n    node_text = []\n    for adjacencies in Meta_G.adjacency():\n        node_adjacencies.append(len(adjacencies[1]))\n        node_text.append(adjacencies[0])\n\n    node_trace.marker.color = node_adjacencies\n    node_trace.text = node_text\n\n    # create the figure\n    fig = go.Figure(data=[edge_trace, node_trace],\n                    layout=go.Layout(\n                        title='Network graph of the connections between the datasets - so far!',\n                        titlefont_size=16,\n                        showlegend=False,\n                        hovermode='closest',\n                        margin=dict(b=20,l=5,r=5,t=40),\n                        width=1200,\n                        height=1000,\n                        annotations=[ dict(\n                            text=\"\",\n                            showarrow=False,\n                            xref=\"paper\", yref=\"paper\",\n                            x=0.005, y=-0.002 ) ],\n                        xaxis=dict(showgrid=False, \n                                   zeroline=False, showticklabels=False),\n                        yaxis=dict(showgrid=False, \n                                   zeroline=False, showticklabels=False))\n                    )\n\n    fig.show()\n\nplot_graph(Meta_G)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-05-23T20:29:06.618967Z","iopub.execute_input":"2023-05-23T20:29:06.619264Z","iopub.status.idle":"2023-05-23T20:29:07.634893Z","shell.execute_reply.started":"2023-05-23T20:29:06.619238Z","shell.execute_reply":"2023-05-23T20:29:07.633875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Output the graph network for reuse","metadata":{}},{"cell_type":"code","source":"# output the graph to a gexf file\nnx.write_gexf(Meta_G, 'meta_graph.gexf', prettyprint=True)","metadata":{"execution":{"iopub.status.busy":"2023-05-23T20:30:03.743871Z","iopub.execute_input":"2023-05-23T20:30:03.744250Z","iopub.status.idle":"2023-05-23T20:30:03.756200Z","shell.execute_reply.started":"2023-05-23T20:30:03.744221Z","shell.execute_reply":"2023-05-23T20:30:03.755028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusion  ","metadata":{}},{"cell_type":"markdown","source":"That's better now the Meta Kaggle dataset joins are mapped out. It is becoming more apparent how the data model works. There might be some more connections to make, e.g., KernelTagId, DatasetId and DataSourceId are not very connected, and that doesn't seem right, but they may just be row indexes. Hopefully, this is a good start and will be helpful to other Kagglers!","metadata":{}}]}