{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":53666,"databundleVersionId":6589269,"sourceType":"competition"},{"sourceId":7876,"sourceType":"datasetVersion","datasetId":5227}],"dockerImageVersionId":30635,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Table of Contents\n1. [Introduction](#Introduction)\n2. [Static vs Interactive Visualizations](#Static-vs-Interactive-Visualizations)\n3. [Plotly](#Plotly)\n    1. [Plotly Traces](#Plotly-Traces)\n    2. [Buttons in Plotly](#Buttons-in-Plotly)\n4. [Datashader and Holoviews](#Datashader-and-Holoviews)\n5. [Dash](#Dash)\n    1. [Quick introduction to HTML](#Quick-introduction-to-HTML)\n    2. [Building a Dashboard with Dash](#Building-a-Dashboard-with-Dash)\n6. [Conclusion](#Conclusion)","metadata":{}},{"cell_type":"markdown","source":"# Introduction\n\nWhen learning about data visualization, most of us began by learning static visualization packages like matplotlib and seaborn (though matplotlib is also able to generate interactive plots, at the cost of more lines of code).\n\nStatic visualization is easy to use and in most cases it provides sufficient information to analyze our data. However, interactive plots have the advantadge of being more appealing due to the interactivity itself, and also provides us the opportunity to further explore our data in a simple and direct way.\n\nOn this notebook, I'll show you [plotly](https://plotly.com/) and [datashader](https://datashader.org/), the first a visualization library that will let you build interactive plots with ease, while the second provides extra functionality to improve visualizations on big datasets.\n\n# Static vs Interactive Visualizations\n\nFirst, let me show you the difference between static and interactive visualizations. For static visualizations, I'll use [seaborn](https://seaborn.pydata.org/), and I'll use the [California Housing Prices](https://www.kaggle.com/datasets/camnugent/california-housing-prices) and the [Child Mind Institute - Detect Sleep States](https://www.kaggle.com/competitions/child-mind-institute-detect-sleep-states) datasets to ilustrate how to make visualizations for small and big datasets, respectively.\n\nTo work with the sleep states dataset, I'll use dask, since this dataset is big enough to not fit in memory, then, I'll convert the dask dataframe into a pandas dataframe once I finish performing some memory-saving operations.","metadata":{}},{"cell_type":"code","source":"pip install fastparquet dash dash_bootstrap_components pyngrok","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:33:29.285795Z","iopub.execute_input":"2024-01-22T19:33:29.286358Z","iopub.status.idle":"2024-01-22T19:33:39.80457Z","shell.execute_reply.started":"2024-01-22T19:33:29.286326Z","shell.execute_reply":"2024-01-22T19:33:39.80212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport json\nimport dask.dataframe as dd\nimport seaborn as sns\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff\nimport holoviews as hv\nimport holoviews.operation.datashader as hd\nimport datashader as ds\nimport datashader.transfer_functions as tf\nimport colorcet\n\n# Dashboarding\nfrom dash import (\n    Dash, html, dash_table, dcc, callback, Output, Input, State\n)\nfrom dash_bootstrap_components.themes import SANDSTONE\n# Expose virtual machine local host\nfrom pyngrok import ngrok","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-01-22T19:33:39.811669Z","iopub.execute_input":"2024-01-22T19:33:39.81212Z","iopub.status.idle":"2024-01-22T19:33:39.820967Z","shell.execute_reply.started":"2024-01-22T19:33:39.812086Z","shell.execute_reply":"2024-01-22T19:33:39.819505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"housing_dtypes = {\n    'longitude': 'float32', 'latitude': 'float32',\n    'housing_median_age': 'uint8', 'total_rooms': 'uint16',\n    'total_bedrooms': 'float32', 'population': 'uint16',\n    'households': 'uint16', 'median_income': 'float32',\n    'median_house_value': 'float32', 'ocean_proximity': 'category'\n}","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:33:39.824033Z","iopub.execute_input":"2024-01-22T19:33:39.824294Z","iopub.status.idle":"2024-01-22T19:33:39.839297Z","shell.execute_reply.started":"2024-01-22T19:33:39.824271Z","shell.execute_reply":"2024-01-22T19:33:39.838667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"housing_path = \"/kaggle/input/california-housing-prices/housing.csv\"\nsleep_path = \"/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet\"\nhousing_data = pd.read_csv(housing_path,dtype=housing_dtypes)\nsleep_data = dd.read_parquet(sleep_path)\nsleep_data.timestamp = dd.to_datetime(\n    sleep_data.timestamp, format='ISO8601', utc=True\n)\nsleep_data.series_id = sleep_data.series_id.astype('category')\nsleep_data = sleep_data.compute()\nsleep_data = sleep_data.reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:33:39.840937Z","iopub.execute_input":"2024-01-22T19:33:39.841574Z","iopub.status.idle":"2024-01-22T19:44:04.642072Z","shell.execute_reply.started":"2024-01-22T19:33:39.841545Z","shell.execute_reply":"2024-01-22T19:44:04.63832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"housing_data.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:04.646031Z","iopub.execute_input":"2024-01-22T19:44:04.646786Z","iopub.status.idle":"2024-01-22T19:44:04.684811Z","shell.execute_reply.started":"2024-01-22T19:44:04.646713Z","shell.execute_reply":"2024-01-22T19:44:04.681623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sleep_data.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:04.689887Z","iopub.execute_input":"2024-01-22T19:44:04.690602Z","iopub.status.idle":"2024-01-22T19:44:04.718053Z","shell.execute_reply.started":"2024-01-22T19:44:04.690529Z","shell.execute_reply":"2024-01-22T19:44:04.716057Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"housing_data.info()","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:04.720615Z","iopub.execute_input":"2024-01-22T19:44:04.72114Z","iopub.status.idle":"2024-01-22T19:44:04.755227Z","shell.execute_reply.started":"2024-01-22T19:44:04.721091Z","shell.execute_reply":"2024-01-22T19:44:04.752788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sleep_data.info()","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:04.757794Z","iopub.execute_input":"2024-01-22T19:44:04.758585Z","iopub.status.idle":"2024-01-22T19:44:04.783749Z","shell.execute_reply.started":"2024-01-22T19:44:04.758522Z","shell.execute_reply":"2024-01-22T19:44:04.781073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Start datetime: {sleep_data.timestamp.min()}')\nprint(f'End datetime: {sleep_data.timestamp.max()}')","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:04.787708Z","iopub.execute_input":"2024-01-22T19:44:04.78908Z","iopub.status.idle":"2024-01-22T19:44:05.604666Z","shell.execute_reply.started":"2024-01-22T19:44:04.788996Z","shell.execute_reply":"2024-01-22T19:44:05.601779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, consider the following plot.","metadata":{}},{"cell_type":"code","source":"sns.scatterplot(housing_data, x='total_rooms', y='population');","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:05.614978Z","iopub.execute_input":"2024-01-22T19:44:05.615843Z","iopub.status.idle":"2024-01-22T19:44:07.731082Z","shell.execute_reply.started":"2024-01-22T19:44:05.615746Z","shell.execute_reply":"2024-01-22T19:44:07.725892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If you hover the mouse over this figure... Nothing happens. You can't either zoom it or click on a given point to know more about it. In this example, we clearly see some outliers, but since the plot is static, to know more about it, we have to make another plot, or perform further analysis, something like the following.","metadata":{}},{"cell_type":"code","source":"zoomed_data = housing_data.loc[(housing_data.population > 20000) | (housing_data.total_rooms > 35000)]","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:07.739663Z","iopub.execute_input":"2024-01-22T19:44:07.743266Z","iopub.status.idle":"2024-01-22T19:44:07.779738Z","shell.execute_reply.started":"2024-01-22T19:44:07.743198Z","shell.execute_reply":"2024-01-22T19:44:07.773623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.scatterplot(zoomed_data, x='total_rooms', y='population');","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:07.794479Z","iopub.execute_input":"2024-01-22T19:44:07.801416Z","iopub.status.idle":"2024-01-22T19:44:10.19397Z","shell.execute_reply.started":"2024-01-22T19:44:07.801322Z","shell.execute_reply":"2024-01-22T19:44:10.187532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Clearly, this approach may take time, and it may end distorting the figure dimensions. This approach is suboptimal.\n\nLet's see how it looks when using an interactive plot.","metadata":{}},{"cell_type":"code","source":"px.scatter(\n    housing_data, x='total_rooms', y='population',\n    hover_data='total_bedrooms'\n).show()","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:10.201376Z","iopub.execute_input":"2024-01-22T19:44:10.202297Z","iopub.status.idle":"2024-01-22T19:44:10.23059Z","shell.execute_reply.started":"2024-01-22T19:44:10.20222Z","shell.execute_reply":"2024-01-22T19:44:10.216291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, if we hover the mouse over the figure, we get the values corresponding to each point we hover the mouse on. Moreover, we can make zoom over the figure by selecting the zone we want to focus on, and come back to normal by double-clicking on the figure.\n\nWe don't need to write extra lines of code, and we can focus on any given zone we want on the same plot. Furthermore, we can enrich our plot by encoding information on each point, so it's available when we hover on a point. Did you notice that the 'total_bedrooms' feature shows when we hover over a point? That's the magic of plotly and its interactive plots.\n\nInteractive plots seem pretty cool, but this advantages comes with a cost: interactive plots have a more complex structure, which makes them harder to visualize for most web browsers, and make them less compatible with some devices. \n\nIn fact, the interactive plot took just 0.091 seconds to be generated, but the browser took about 2-3 seconds to show it to me, while the static plot was shown almost instantly. Furthermore, if you have several interactive plots, you may overcharge your web browser with them, making it to crash. This issue may be partially solved by using tools designed for big datasets, but I'll talk about it later.\n\nI hope that the examples above made clear the difference between static and interactive plots.\n\n# Plotly\n\nAs you may have noticed, to produce the interactive plot I used a package called plotly, which is like the seaborn of interactive plots. There are other packages that work in a similar way, like [Bokeh](https://bokeh.org/) or [Vega-Altair](https://altair-viz.github.io/), but plotly has the advantage to be the entry point to learn [Dash](https://dash.plotly.com/), which is a package designed to build interactive dashboards.\n\nNow, if you search on the internet, you will find three main plotly module:\n- [plotly.express](https://plotly.com/python/plotly-express/): A module suited for simple, high-performing interactive plots.\n- [plotly.figure_factory](https://plotly.com/python/figure-factories/): A module containing tools to build plots from specific domains, like quiver plots for flow analysis or dist plots for probability distribution's visualization.\n- [plotly.graph_objects](https://plotly.com/python/graph-objects/): A module to build general-purpose interactive visualizations, which can be embeded with additional interactivity features.\n\nThe main difference between the three packages is that:\n- Figures from the plotly.graph_objects can be updated dinamically by adding something called 'trace' (I'll talk more about it later). In particular, supports the addition of buttons and other web site-like elements.\n- Figures from plotly.express are easier and faster to generate, but can't be updated in the same way as figures from plotly.graph_objects can (at least not directly).\n- The plotly.figure_factory includes plots that aren't available in neither the plotly.express module nor the plotly.graph_objects module.\n\nAs stated above, on plotly.graph_objects figures you can add elements like buttons. This buttons can be programmed, between other things, to change some aspects from your figure, like color pallets; your data, like the feature you're plotting; or both.\n\nThis way, you can have a single figure, plotting, for example, several time series, but one at a time, so you don't need to saturate your notebook/web browser with plots. However, if you want to change two or more properties of your figure that depends on your data, it becomes very challenging, and is easier to implement with Dash instead. In most cases, one button that changes data will be enough.\n\nIn any case, to start playing with buttons, is necessary to understand what is a trace in plotly.\n\n## Plotly Traces\n\nIn plotly, you can think of traces as visualization layers, like onions. Each one of these layers can hide or coexist with other layers, like the elements of a power point. And the layers appear or disappear depending on how the user interacts with them.\n\nTake, for example, the following figure.","metadata":{}},{"cell_type":"code","source":"px.scatter(\n    housing_data, x='total_rooms', y='population',\n    color='ocean_proximity', hover_data='total_bedrooms'\n).show()","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:10.238571Z","iopub.execute_input":"2024-01-22T19:44:10.239359Z","iopub.status.idle":"2024-01-22T19:44:10.26342Z","shell.execute_reply.started":"2024-01-22T19:44:10.239297Z","shell.execute_reply":"2024-01-22T19:44:10.256356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is almost the same as the interactive plot I made before, except that the data is now grouped by its proximity to the ocean. Try clicking on a category on the legend... You can see that the points from the clicked category are gone now. Click the same category on the legend, and the points will come back.\n\nAll the plotly modules implement this kind of interactivity to some extent. You can think about it as if plotly puts each category into a different visualization layer. Then, plotly shows all the layers at the same time, but the user can hide each layer at will.\n\nLet's try building the same figure, but with the plotly.graph_objects module. Maybe this way the concept of 'trace' comes more evident.","metadata":{}},{"cell_type":"code","source":"fig = go.Figure()\nfor cat in housing_data.ocean_proximity.cat.categories:\n    plot_data = housing_data.loc[housing_data.ocean_proximity == cat]\n    fig.add_trace(\n        go.Scatter(\n            x=plot_data.total_rooms, y=plot_data.population,\n            mode='markers', name=cat\n        )\n    )\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:10.278574Z","iopub.execute_input":"2024-01-22T19:44:10.281314Z","iopub.status.idle":"2024-01-22T19:44:10.305088Z","shell.execute_reply.started":"2024-01-22T19:44:10.281199Z","shell.execute_reply":"2024-01-22T19:44:10.297055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see in more detail what I did here:\n- First, I instantiated an empty figure using go.Figure()\n- Next, I iterated over all the posible ocean proximities into a for loop.\n- I filtered the samples with the desired ocean proximity and saved the results into plot_data.\n- I added a trace (that is, a visualization layer) to the figure, where each trace is a scatter plot of the desired data. I said to plotly.graph_objects that I want the data to be represented as points (markers) using the 'mode' parameter, and I gave the grouped scatter a name (the same as the plotted category) using the 'name' parameter, which shows in the figure's legend.\n- Finally, I showed the figure on the output cell using the 'show' method.\n\nAs you can see, the syntax of the plotly.graph_objects module is different from the one of plotly.express, as plotly.express works in a very similar way to seaborn, while plotly.graph_objects only receives array-like objects as data.\n\nIn any case, we got essentially the same result as with plotly.express, but the traces became explicit now. However, figures from plotly.graph_objects are significantly heavier than the ones of plotly.express, so use this module wisely.\n\nNow, why to use plotly.graph_objects instead of plotly.express if the former is heavier and requires more lines of code than the latter?. The answer to that is: buttons.\n\n## Buttons in Plotly\n\nAs I said before, the only way to implement buttons on a poltly figure is through the plotly.graph_objects, and they are very useful if we want to plot a lot of data once at a time, or to give the user some control of the graphic preperties of the figure.\n\nLet's try plotting the same figure, but now implementing a button to choose which category to show.","metadata":{}},{"cell_type":"code","source":"cats = housing_data.ocean_proximity.cat.categories\nbuttons = []  # contains button data\noption = dict(\n    label='All',\n    method='update',\n    args=[\n        {'visible': [True] * len(cats)},\n        {'title': 'All'}\n    ]\n)\nbuttons.append(option)\nfig = go.Figure()\nfor i, cat in enumerate(cats):\n    plot_data = housing_data.loc[housing_data.ocean_proximity == cat]\n    fig.add_trace(\n        go.Scatter(\n            x=plot_data.total_rooms, y=plot_data.population,\n            mode='markers', name=cat\n        )\n    )\n    \n    # Dictionary with button (option) data\n    option = dict(\n        label=cat,\n        method='update',\n        args=[\n            {'visible': np.isin(cats, cat)},\n            {'title': cat}\n        ]\n    )\n    buttons.append(option)\n\nfig.update_layout(\n    updatemenus=[\n        dict(\n            type='dropdown',\n            direction='down',\n            buttons=buttons\n        )\n    ]\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:10.31427Z","iopub.execute_input":"2024-01-22T19:44:10.317001Z","iopub.status.idle":"2024-01-22T19:44:10.362678Z","shell.execute_reply.started":"2024-01-22T19:44:10.316948Z","shell.execute_reply":"2024-01-22T19:44:10.350178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note that button on the top left? Now you can choose which category to focus on by just selecting it on the dropdown button. This can be pretty convenient if you want to visualize several categories or features into a single figure.\n\nNow, let's talk a bit further about traces. When we create a figure, plotly keeps its traces in a kind of list, and each time we add a trace, it is appended at the end of it. So, we got something like [trace0, trace1, ...].\n\nNow that I made it clear, let's talk about the structure of a button.\n- In plotly, a button is a list with several options, which can be buttons themselves.\n- Each button's option is dictionary containing the option's attribute in the form of attribute: value. This attributes define the label of the option, which properties of the figure it controls, etc.\n- Any button should have at least 3 main attributes setted:\n    1. Label, which controls the option's name in the button.\n    2. Method, which controls how the data is updated.\n    3. Args, which controls which properties of the figure will be updated.\n\nWell, this is the basic structure of a button, so let's talk further about the method and args attributes.\n\nThere are 4 possible values for the method attribute:\n- 'restyle', that modifies data or data attributes of the figure, that is, the data shown, marker size and other data attributes.\n- 'relayout', that modifies layout attributes, that is, aesthetic properties like title, figure size and so on.\n- 'update', that merges the behaviour of 'restyle' and 'relayout' and updates all the figure attributes.\n- 'animate', in case you're making an animated plot, pauses or starts an animation.\n\nSo, if you're unsure about which method to use, go with 'update', since it modifies everything.\n\nNow, about the 'args' attribute, it is a list of dictionaries where each dictionary defines a figure's property that will be modified. In the plot above, the button modifies two attributes: which trace will be visible through the 'visible' attribute, and the figure's title through the 'title' attribute. And since the buttons modifies both a data attribute (which traces are visible) and an aesthetic attribute (the title), I use the update method to modify the data.\n\nRemember how I talk about plotly keeps a figure's traces in a list? Well, the 'visible' attribute is a list with a length equal to the number of traces with boolean elements, which defines if the trace in the position i will be visible or not. Hence, this attribute is in charge of hiding a trace or making it visible.\n\nFinally, to add the buttons to the figure, we use the 'update_layout' method, which can be used to modify the figure's appearance. Through the 'updatemenus' parameter, we can update the figure's menus (i.e, the buttons) by passing to it a list of dictionaries containing an attribute to update and its new value. Here, I defined the button type (can be button to set each option in different buttons or dropdown to put all the options in a single button), the direction the options will be dropped, and the buttons themselves I want to add. Note that you can add several buttons by passing a list of dictionaries, each dictionary containing the attributes of a button.\n\nFinally, as a side note, if you set 'type' as 'button' (that is, a normal button) you will get a button that can have two status: active (if click it once) and inactive (when you click it again). In this case, the 'args' attribute defines the figure's attributes to modify when the button is **active**, and you will have an attribute called 'args2', that sets the figure's attributes when the button is **inactive**.\n\n# Datashader and Holoviews\n\nAll the above sounds good for small datasets (the california housing dataset has about 20k samples), but what if we have, say, 1M samples? or 10M?. On this case, we have (at least) two issues:\n- Scatter plots can't be shown clearly due to the massive amount of points.\n- The huge amount of data can make your navigator to crash, possibly loosing all your progess so far.\n\nHow we can solve this problem? There are several libraries designed to work with big datasets, but in this case I'll use [datashader](https://datashader.org/).\n\nWhat datashader does is to compute the point density on each pixel on the screen, and plot a sort of density heatmap. This way it can clearly show how many data points are on a given region with high precision, without saturating the web browser with unnecessary information.\n\nLet's check how to use it. By default, datashader can be used itself as a visualization package. If we try to plot data from the sleep states dataset (which has about 2M samples), we got something like this:","metadata":{}},{"cell_type":"code","source":"cvs = ds.Canvas(plot_width=850, plot_height=500)\nagg = cvs.points(sleep_data, 'anglez', 'enmo')\nimg = ds.tf.shade(agg, cmap=colorcet.fire)\nimg","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:10.372476Z","iopub.execute_input":"2024-01-22T19:44:10.373547Z","iopub.status.idle":"2024-01-22T19:44:14.015257Z","shell.execute_reply.started":"2024-01-22T19:44:10.373454Z","shell.execute_reply":"2024-01-22T19:44:14.00953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The result is a heatmap-like plot that shows the data with a fire color scale. With that, we not only know the position of a point, but also the point density on a given region as small as a single pixel. And very fast, even with that amount of data.\n\nWhat I've done above is to set the canvas size using the ds.Canvas class, aggregate the data points using the points method on the canvas object, and generate to actual figure using the tf.shade function by passing it an aggregated canvas and a color scale (which I provided using the colorcet package).\n\nNote that I used the 'points' method on the canvas to indicate datashader that I wanted it to aggregate the data samples as points, but as you may expect, that's not the only method to aggregate data to the canvas. For example, if you use the 'line' method instead, datashader will represent the data as connected lines. On the [datashader user guide](https://datashader.org/user_guide/) you can see more details about it. Also, you can learn more about how datashader works with data on its [documentation](https://datashader.org/getting_started/Pipeline.html#transformation).\n\nIn any case, note that this plot lacks a lot of features: it doesn't have axes, only a plain image. We can solve it by using a backend to show the figure with all these elements. And you may have guessed that the backend I'll use is plotly. I'll do this through the [Holoviews](https://holoviews.org/) package, that can take a datashader object (between others) and plot it using the desired backend.","metadata":{}},{"cell_type":"code","source":"hv.extension('plotly')","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:14.023013Z","iopub.execute_input":"2024-01-22T19:44:14.024146Z","iopub.status.idle":"2024-01-22T19:44:14.242085Z","shell.execute_reply.started":"2024-01-22T19:44:14.024058Z","shell.execute_reply":"2024-01-22T19:44:14.238323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"points = hv.Points(sleep_data, ['anglez', 'enmo'])\nhd.datashade(points, width=900, height=500, cmap=colorcet.fire)","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:14.251218Z","iopub.execute_input":"2024-01-22T19:44:14.252017Z","iopub.status.idle":"2024-01-22T19:44:14.272003Z","shell.execute_reply.started":"2024-01-22T19:44:14.251967Z","shell.execute_reply":"2024-01-22T19:44:14.266389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we get a figure with axes!. Note that the syntax with holoviews is very similar to the one of datashader, with the exception that we skip the first step of creating a canvas. Also, it supports other backends, like bokeh or matplotlib: just chanhe 'plotly' with the name of the backend you want to use.\n\nThat covers the basics of plotting big datasets, so let's talk about dashboards.\n\n# Dash\n\nAs I said before, [Dash](https://dash.plotly.com/) is a Python module for the creation of dashboards. What is a dashboard? You can think of it as a gallery of visualizations, though in most cases the visualizations can interact between them, and change dinamycally depending on some input.\n\nAs most dashboards are uploaded to the internet, you need at least some basic knowledge about web design, in particular about [HTML](https://www.w3schools.com/html/), so I'll quickly introduce it to you before continuing.\n\n## Quick introduction to HTML\n\nHTML is a language used to create the structure of a website. It uses 'tags' to define the mentioned structure, and each tag has a well-defined behaviour. Some of the most common tags are:\n- html: Defines the start and end of a HTML document.\n- head: Used to define the website's metadata and other properties not shown on the website, but recognized by the web browser.\n- body: Defines the start and end of the content shown to the user.\n- div: Used to create a box to contain some content on the website.\n- h1-h6: Used to define titles. the number represents the size of the title, where h1 is the largest size and h6 the smallest.\n- p: Used to define a paragraph.\n- span: Used to give extra styling properties to text inline.\n- table: Defines a table.\n- hr: Creates an horizontal rule.\n\nThere are much more tags with different behaviours, but they are the most used.\n\n## Building a Dashboard with Dash\n\nAs dash is designed mainly to build web apps, it has a module called [html](https://dash.plotly.com/dash-html-components) that contains several classes to create HTML tags. For example, html.Div is used to create a div tag with content in it, that can also be or contain other HTML tags.\n\nThere's also a module, though not contained in Dash, called [dash_bootstrap_components](https://dash-bootstrap-components.opensource.faculty.ai/). If you know a bit about web development you may have heared about [Bootstrap](https://getbootstrap.com/), but in case you didn't know, it is a framework designed to facilitate web styling. In particular, dash_bootstrap_components provides extra components for dash that like with bootstrap. In particular, you can import themes to give your web app a given style by default.\n\nAll these elements combined together make dash a powerful but simple-to-use framework for dashboard design on Python. Now, let's put it into practice and build a dashboard ourselves.\n\nI'll plot some figures and put them together on a basic dashboard using dash.","metadata":{}},{"cell_type":"code","source":"# Create a figure that will be fixed in our dashboard\nfig1 = go.Figure(\n    go.Scatter(\n        x=housing_data.total_rooms,\n        y=housing_data.population,\n        mode='markers'\n    )\n)","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:14.280536Z","iopub.execute_input":"2024-01-22T19:44:14.282385Z","iopub.status.idle":"2024-01-22T19:44:14.314963Z","shell.execute_reply.started":"2024-01-22T19:44:14.282313Z","shell.execute_reply":"2024-01-22T19:44:14.306977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initialize app with Sandstone style theme\napp = Dash(\n    __name__, external_stylesheets=[SANDSTONE]\n)","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:14.326437Z","iopub.execute_input":"2024-01-22T19:44:14.326918Z","iopub.status.idle":"2024-01-22T19:44:14.374859Z","shell.execute_reply.started":"2024-01-22T19:44:14.326875Z","shell.execute_reply":"2024-01-22T19:44:14.367431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define a function to build the app layout\ndef function_layout():\n    # Put all the app's content in a div tag\n    app_layout = html.Div([\n        html.H1('This is a title'),\n        html.H2('And this is a subtitle'),\n        html.Hr(),  # Horizontal rule to separate title from figures\n        \n        # Create a container to put fig1 on (children refers\n        # to the content of this tag)\n        html.Div(children=[\n            dcc.Graph(figure=fig1)\n        ]),\n        \n        # Create a container that will contain a dropdown element\n        # This button will make the dashboard to show different\n        # plots dependign on the active option\n        html.Div(children=[\n            dcc.Dropdown(\n                options=[\n                    dict(label='sleep', value='sleep'),\n                    dict(label='housing', value='housing')\n                ],\n                value='housing',\n                clearable=False,\n                searchable=False,\n                id='dataset_type'\n            )\n        ]),\n        \n        # This plot will change depending on the previous dropdown\n        html.Div(children=[\n            dcc.Graph(id='plot')\n        ]),\n        \n        # Store the data needed to generate the plot from the\n        # previous Div\n        dcc.Store(\n            id='housing_data',\n            data=housing_data[['total_bedrooms', 'population']].to_json()\n        ),\n        dcc.Store(\n            id='sleep_data',\n            data=sleep_data.loc[sleep_data.step < 20, ['anglez', 'enmo']].to_json()\n        )\n    ])\n\n    return app_layout\n\napp.layout = function_layout\n\n# Make Dash to change plots depending on the active value on dropdown\n@app.callback(\n    # This controls the desired output (can be more than one)\n    # The output corresponds to the property 'figure' on the component\n    # with id 'plot', which corresponds to the Graph element\n    Output(component_id='plot', component_property='figure'),\n\n    # This takes as input the value on the Dropdown element\n    Input(component_id='dataset_type', component_property='value'),\n\n    # This gets the data stored on the Store elements\n    State(component_id='housing_data', component_property='data'),\n    State(component_id='sleep_data', component_property='data')\n)\ndef update_plot(dataset_type, housing_data, sleep_data):\n    housing_data = pd.DataFrame.from_dict(json.loads(housing_data))\n    if dataset_type == 'housing':\n        fig = go.Figure(\n            go.Scatter(\n                x=housing_data.total_bedrooms,\n                y=housing_data.population,\n                mode='markers'\n            )\n        )\n    else:\n        sleep_data = pd.DataFrame.from_dict(json.loads(sleep_data))\n        fig = go.Figure(\n            go.Scatter(\n                x=sleep_data.anglez,\n                y=sleep_data.enmo,\n                mode='markers'\n            )\n        )\n    return fig","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:14.390431Z","iopub.execute_input":"2024-01-22T19:44:14.392363Z","iopub.status.idle":"2024-01-22T19:44:14.972066Z","shell.execute_reply.started":"2024-01-22T19:44:14.39225Z","shell.execute_reply":"2024-01-22T19:44:14.967546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Expose Kaggle's virtual machine local host to show the dashboards\n# This step is not required in a local machine\n\n# Authenticate on ngrok\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nngrok_token = user_secrets.get_secret(\"ngrok_token\")\nngrok.set_auth_token(ngrok_token)\ntunnel = ngrok.connect(8050)","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:14.98305Z","iopub.execute_input":"2024-01-22T19:44:14.984185Z","iopub.status.idle":"2024-01-22T19:44:15.347Z","shell.execute_reply.started":"2024-01-22T19:44:14.984095Z","shell.execute_reply":"2024-01-22T19:44:15.343807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Run app (this will open the app on a new tab)\nif __name__ == '__main__':\n    app.run(\n        debug=True,\n        jupyter_mode='external',\n        jupyter_server_url=tunnel.public_url\n    )","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:15.352592Z","iopub.execute_input":"2024-01-22T19:44:15.355252Z","iopub.status.idle":"2024-01-22T19:44:16.082401Z","shell.execute_reply.started":"2024-01-22T19:44:15.355134Z","shell.execute_reply":"2024-01-22T19:44:16.079952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And we got an app running, though working on a virtual machine can make things a bit more challenging.\n\nLet me explain some parts of the code:\n- I used the html.Div element to create a HTML container for several elements of the dashboard, like text or plots.\n- The dcc.Graph element (where dcc stands for [Dash Core Components](https://dash.plotly.com/dash-core-components)) is used to add a plotly figure as a HTML element. It only accepts plotly.go.Figure objects, so in this particular case I had to enclose the go.Scatter into a go.Figure to make it compatible with Dash.\n- The dcc.Dropdown element is used to make a dropdown with several options, and a default value stated by the 'value' property. On this dashboard, it is used to switch between datasets for the second figure in the dashboard (the one that comes just after the button). Because it make dynamic changes on the website, I assign it an id, in this case 'dataset_type'.\n- The dcc.Graph element that follows is left empty and only an id is defined. That is because the figure shown wil change depending on the value of the dropdown.\n- On the dcc.Store elements I save the data I'll use for the dynamic plot. Dash only accepts data on json format, so I use the .to_json method to make the conversion. Also, Dash stores the data on the browser so it doesn't accept big datasets, so I have shring the sleep_data dataset.\n- After defining the layout function, I assign it to the 'layout' attribute of the app. This will make it possible for the app to call the layout function each time the site is updated, so it can change the layout dynamically. This is called 'live update'.\n- After that, I define a function to update the second plot based on the dropdown's value. The function receives as input the current value of the dropdown and outputs the figure to be shown. It also contains to 'states'. This states call the respective data saved on the browser. We include this information about the function's inputs and outputs on the 'app.callback' decorator.\n- Finally, I run the app on the given url. Since I'm doing all this on Kaggle, I use ngrok to access the local host of this virtual machine and create a url to connect to it. If you want to know more about ngrok, I'll leave its website [here](https://ngrok.com/).\n\nSome notes about dashboards in dash:\n- Since it keeps the data on the dcc.Store components in the browser, the memory usage is typically limited to 2 to 5 MB. Hence, if you want to use a big dataset on your dash app, you will need to make the app to update the data it keep in the browser in some way. For example, you can give the user a button to visualize only a piece of data at a time. Take into consideration, however, that switching between data is slow, so include big data on your dashboard only when necessary.\n- Each time a user opens your web app, the browser will load any data you have stores in the dcc.Store components. This will slow down your app when it is starting, so avoid storing data as much as possible.\n- A dash callback can have several inputs, outputs and/or states. Therefore, when you want to update several elements of your app that share the same elements, you can do it in a single function call. This is a good practice and should be prefered above updating one element at a time, since the latter is slower and prone to errors.\n\nAs a final tip, what if we want to include a figure from another plotly module on our dashboard?. The truth is that nothing stops us to do so, even if the figures are generated from different modules. That is because all the figures created on plotly inherit from the graph_objects module. In particular, they're all of type Figure which is compatible with Dash.\n\nTo ilustrate that, let's make one plot with each module and see which class they come from.","metadata":{}},{"cell_type":"code","source":"fig_px = px.scatter(housing_data, x='total_rooms', y='population')\nfig_go = go.Figure(\n    go.Scatter(x=housing_data.total_rooms, y=housing_data.population)\n)\nfig_ff = ff.create_distplot([housing_data.population], ['population'])","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:16.084808Z","iopub.execute_input":"2024-01-22T19:44:16.085434Z","iopub.status.idle":"2024-01-22T19:44:17.072469Z","shell.execute_reply.started":"2024-01-22T19:44:16.085407Z","shell.execute_reply":"2024-01-22T19:44:17.061938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Plotly express plot type: {type(fig_px)}')\nprint(f'Plotly graphic objects plot type: {type(fig1)}')\nprint(f'Plotly figure factory plot type: {type(fig_ff)}')","metadata":{"execution":{"iopub.status.busy":"2024-01-22T19:44:17.078556Z","iopub.execute_input":"2024-01-22T19:44:17.084096Z","iopub.status.idle":"2024-01-22T19:44:17.108189Z","shell.execute_reply.started":"2024-01-22T19:44:17.083999Z","shell.execute_reply":"2024-01-22T19:44:17.102989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, all the three objects are of the same type.\n\n# Conclusion\n\nInteractive plots are better than static ones when you want to explore your data or want some grade of interaction with the viewer. However, they can make the web browser slower, or even make it crash when interactive plots are used in excess, specially when plotting big datasets. Using specialized visualization tools like datashader can help to mitigate this issue.\n\nIn the particular case of plotly, it can be integrated natively with dash, which make it suited not only for experimentation, but also to publish your discoveries. Hence, you can transit smoothly from experimentation to publishing faster by just saving your most valuable plots and integrating them to a dash web app.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}