Learn to Create a Heat Map in Python with Data from NCHS (2018)

Learn to Create a Heat Map in Python With Data From NCHS (2018)

© 2021 SAGE Publications, Ltd. All Rights Reserved. This PDF has been generated from SAGE Research Methods Datasets. SAGE SAGE Research Methods: Data 2021 SAGE Publications, Ltd. All Rights Reserved. Visualization Learn to Create a Heat Map in Python With Data From NCHS (2018)

How-to Guide for Python

Introduction In this guide, you will learn how to create a heat map using the Python programming language. Readers are provided links to the example dataset and encouraged to replicate this example. An additional practice example is suggested at the end of this guide. The example assumes you have downloaded the relevant data files to a folder on your computer and that you are using the JupyterLab environment. The relevant code should, however, work in other environments too.

Contents 1. Heat Map

2. An Example in Python: Monthly Deaths by Influenza and Pneumonia in California, 2014–2018

2.1. The Python Procedure 2.1.1. JupyterLab Notebooks 2.1.2. Testing Out the Programming Environment 2.1.3. Creating Our Notebook, Importing Necessary Modules 2.1.4. Reading In and Formatting Our Data 2.1.5. Plotting the Chart 2.1.6. Saving the Plot 2.1.7. Alternative Version With Classified Color Scale

3. Your Turn

1. Heat Map Heat maps are visualizations that use a two-dimensional arrangement of colored rectangles, where the row corresponds to one data dimension and the column to another. Each rectangular block represents an intersection between the two plotted series and is colored by the values for observations in that particular intersection. Heat maps use position and color hue to encode values.

One or both axes of the heat map can be quantitative, in which case the blocks represent values falling into different bins or value ranges. The axes can also be qualitative, thus representing different categories. Heat maps are also often used to show time series. A similar tabular visualization that shows intersections of data series with qualitative values (e.g., true/false; low, medium, high) is usually called a matrix.

2. An Example in Python: Monthly Deaths by Influenza and Pneumonia in California, 2014–2018 Figure 1 shows a heat map of monthly influenza and pneumonia deaths in California. The heat map gives a good impression of the seasonal variation in deaths attributed to these causes and highlights months where particularly many lives were lost to these causes. A continuous linear yellow–orange–red color scheme was used with darker hues corresponding to larger death counts. Data values are displayed for each individual cell.

The horizontal axis is labeled month and lists months from January to December. The vertical axis ranges from 2014 to 2018, in increments of 1. The data are tabulated below.

Year January February March April May June July August September October November December

2014 986 693 521 475 446 382 392 364 361 373 447 503

2015 918 730 577 521 452 420 379 358 385 417 455 573

2016 679 680 749 521 420 360 421 330 376 403 417 635

2017 985 722 617 511 466 403 360 372 368 399 396 762

2018 1,674 817 771 554 427 389 359 324 319 358 441 485

As the count increases, the intensity of the color shade increases. Text under the map reads, “Source: NCHS, 2018.”

Figure 1. Heat Map of Monthly Influenza and Pneumonia Deaths in California 2014–2018

2.1 The Python Procedure Python is a general-purpose programming language that supports several

Page 4 of 33 Learn to Create a Heat Map in Python With Data From NCHS (2018) SAGE SAGE Research Methods: Data 2021 SAGE Publications, Ltd. All Rights Reserved. Visualization programming paradigms and has a very clear syntax. It is a versatile tool, particularly for data manipulation and visualization. As Python was originally created as a learning tool, it is also reasonably easy to read for beginners. For more information, visit https://www.python.org/.

You can write Python code with any plain text editor, such as Sublime Text or Visual Studio Code. For the purposes of this tutorial, you do not need to install anything additional, as we will be using a web-based programming environment.

Note: This tutorial uses Python 3. Many online articles about Python programming and other sources discuss Python 2, which differs slightly, but in important ways from Python 3. Although code written in Python 2 often works in Python 3 and vice versa, this not always the case, and mixing the two Python versions can lead to errors or unexpected results.

2.1.1 JupyterLab Notebooks The traditional way of programming would be to write some code in a text file, then building and running it to generate an output. In a notebook, on the other hand, the code is broken down into cells, which can be run one at a time, displaying results right in the editor. This makes working with code and experimenting with changing parameters much more flexible and is particularly suitable for interactive data exploration, where the Python programming language shines. Sharing small code projects (such as visualizations!) generally becomes much simpler with the notebook approach, since you can save the entire notebook and send it to others.

We will be using JupyterLab, a modern web-based notebook interface for Python, that requires no installation on the user’s part. To try a notebook online, just open https://jupyter.org/try and click Try JupyterLab. A cloud-hosted ready-to-use online JupyterLab environment will be activated after a short wait. Try refreshing the window if loading stalls.

Page 5 of 33 Learn to Create a Heat Map in Python With Data From NCHS (2018) SAGE SAGE Research Methods: Data 2021 SAGE Publications, Ltd. All Rights Reserved. Visualization Take note that this JupyterLab session is hosted on https://mybinder.org/, and it will timeout after ~15 minutes if inactive. Make sure to download and save your notebooks locally before leaving the computer. If your session has expired, start a new one from https://jupyter.org/try and use the interface to upload your saved notebook to continue where you left off.

Note: The online trial of JupyterLab is a good place to start if you want to experiment with programming in Python, but for continued use in the future, it is recommended to install the Conda package and environment management system and JupyterLab locally on your system.

To obtain Conda, it is easiest to install one of two distributions: Anaconda, a powerful Python and R distribution that includes over 250 packages for various uses, or Miniconda, a minimal version Anaconda that includes only conda, Python, package dependencies, and a few other useful packages (JupyterLab not included). For more information on obtaining Conda, you can visit https://docs.conda.io/projects/conda/en/latest/user-guide/install/. If you choose the Miniconda distribution, you will need to install JupyterLab locally from your terminal (Mac) or Command Prompt/PowerShell (Win) with conda install -c conda- forge jupyterlab. For more information on installing JupyterLab, visit https://jupyterlab.readthedocs.io/en/stable/getting_started/installation.html.

If you already have an installation of Anaconda or similar and a preferred plain text editor at your disposal, the relevant code covered in this tutorial should work in other environments as well.

2.1.2 Testing Out the Programming Environment If you open https://jupyter.org/try and click Try JupyterLab, you will be welcomed by a rather complex demo example. We will ignore this for now and create our own new notebook instead. Under File choose New > New Notebook. In the Select Kernel prompt, choose Python 3.

Page 6 of 33 Learn to Create a Heat Map in Python With Data From NCHS (2018) SAGE SAGE Research Methods: Data 2021 SAGE Publications, Ltd. All Rights Reserved. Visualization You now see an empty notebook called Untitled.ipynb with one empty cell: a text box where you type code for execution. The cells of a notebook are convenient for structuring code in small chunks that can be run one at a time—as opposed to the more common way of building and running a whole script at once. A single cell can contain as many or few lines of code as you want. You could also change the cell to hold markdown-formatted text instead of code to write longer comments or add illustrations.

You can test what JupyterLab does by writing some code in the empty cell. Click inside the cell and type in the following:

print("Hello, world!")

Hit shift + enter or press the small play arrow ▸ above in the toolbar to run the cell.

2.1.3 Creating Our Notebook, Importing Necessary Modules Create a new notebook and save it with a name, for example, polarchart.ipynb. You will refer back to this should your JupyterLab session time out.

If we were running this project locally, we would first need to install all the modules necessary for generating the visualization. However, the trial environment launched from https://jupyter.org/try conveniently comes equipped with everything for our purposes.

• Matplotlib (MPL for short) is one of the most popular visualization libraries for Python. It has a huge feature set, and there are dozens of often very different—and sometimes odd—ways of achieving the same thing. Do not be alarmed if you search the internet for how to do something in Matplotlib and cannot comprehend some particular instructions. It is probably the case that the article you have found is based on a different approach to MPL than

Page 7 of 33 Learn to Create a Heat Map in Python With Data From NCHS (2018) SAGE SAGE Research Methods: Data 2021 SAGE Publications, Ltd. All Rights Reserved. Visualization what we are working with in this tutorial. Pyplot (PLT) is a Matplotlib module which provides features similar to MATLAB, https://matplotlib.org/ • Pandas (PD) is a powerful data analysis and manipulation toolset, https://pandas.pydata.org/ • Seaborn (SNS) brings added functionality to Matplotlib, for example, new chart types. All of MPL’s features are still accessible, but Seaborn offers an easier to use interface for some of them. This guide is written based on Seaborn 0.10.1—other versions can differ somewhat.

You can run the following code in the first cell of your notebook to import the necessary items:

import matplotlib.pyplot as plt

import pandas as pd

import seaborn as sns

One initially confusing thing about working with Matplotlib is the proliferation of different submodules you need to invoke at different times when writing your code. At times we are giving commands to Matplotlib, other times to Pyplot, but also conceivably to Seaborn, and so on. You will start getting used to it after a while, but it is sure to cause some confusion more than once.

It is a common practice to use the shortcut plt for Pyplot and pd for Pandas. There are other such established shortcuts in the world of Python, as well, for example, sns for Seaborn and np for Numpy. There is technically no need to use such shortcuts, but as most online articles will follow this convention, we will as well.

2.1.4 Reading In and Formatting Our Data Save the tutorial csv data file Weekly_Counts_of_Deaths_by_State_and _Select_Causes__2014-2018.csv to a folder on your computer. The example

Page 8 of 33 Learn to Create a Heat Map in Python With Data From NCHS (2018) SAGE SAGE Research Methods: Data 2021 SAGE Publications, Ltd. All Rights Reserved. Visualization uses the same folder where the JupyterLab notebook is saved, so no other path than the file name is necessary to refer to the csv files. If you choose to save your files elsewhere, just update the path accordingly.

We will begin by importing our data. Enter this in the next cell:

data = pd.read_csv('Weekly_Counts_of_Deaths_by_State_and_Select_Causes__2014-2018.csv', sep=';', thousands=',')

data

Instead of using Python’s own csv module, we will use the read_csv() function from the Pandas library. This creates a “pandas dataframe” of our csv, essentially a spreadsheet table with rows and columns, which lets the user specify decimal separators and other important parameters during import and is a required data format for some plotting modules. This file also uses thousands separators. The argument thousands=',' allows the function to read the numerical data correctly.

Should you have a different decimal separator or delimiter in your dataset, you could specify it during import like this importedfile = read_csv("file.csv", delimiter = "\t", decimal = ",").

Depending on your datasets in the future, you may also need to experiment with different encoding types—if wrong characters appear in names, you might need to change the encoding. Here it is by default encoding = "utf-8". See the full list: https://docs.python.org/3/library/codecs.html#standard-encodings.

You can check and see that the import worked by adding data to the end of the cell (Figure 2). The traditional long-form version of this command is print(data), but one of the convenient things about working with a notebook is that you also can see what a variable contains simply by typing the name and running the

Page 9 of 33 Learn to Create a Heat Map in Python With Data From NCHS (2018) SAGE SAGE Research Methods: Data 2021 SAGE Publications, Ltd. All Rights Reserved. Visualization cell—this will show the table and with nicer formatting. Note that if you place two variable names in the same cell like this, only the last one will be shown in the preview. To inspect many different variables in the same cell or inspect any variables in environments other than JupyterLab, you will still need to use the print() command.

The code line is as follows:

[2]: data = pd.read_csv(‘Weekly_Counts_of_Deaths_by_State_and_Select_Causes__2014-2018.csv’, sep=‘;’, thousands=‘,’) data

The output table is as follows.

[2]:

Jurisdiction Week Malignant Diabetes Alzheimer MMWR MMWR All Natural Septicemia of Ending neoplasm mellitus disease Flag_neopl Flag_diab Flag_alz Year Week Cause Cause (A40-A41) Occurrence Date (C00-C97) (E10-E14) (G30)

01/04/ Alabama 2014 1 355 327 NaN 60.0 NaN 10.0 NaN NaN NaN 2014

01/11/ Alabama 2014 2 872 792 23.0 163.0 23.0 35.0 NaN NaN NaN 2014

01/18/ Alabama 2014 3 1044 971 21.0 209.0 34.0 31.0 NaN NaN NaN 2014

01/25/ Alabama 2014 4 1022 967 25.0 205.0 23.0 25.0 NaN NaN NaN 2014

02/01/ Alabama 2014 5 1040 953 18.0 200.0 26.0 38.0 NaN NaN NaN 2014

12/01/ Suppressed Suppressed Wyoming 2018 48 106 94 NaN 22.0 NaN 10.0 NaN 2018 (counts 1-9) (counts 1-9)

12/08/ Suppressed Suppressed Wyoming 2018 49 92 80 0.0 22.0 NaN NaN NaN 2018 (counts 1-9) (counts 1-9)

12/15/ Suppressed Suppressed Wyoming 2018 50 101 92 0.0 19.0 NaN NaN NaN 2018 (counts 1-9) (counts 1-9)

12/22/ Suppressed Suppressed Wyoming 2018 51 105 95 NaN 24.0 NaN NaN NaN 2018 (counts 1-9) (counts 1-9)

12/29/ Suppressed Suppressed Wyoming 2018 52 92 86 0.0 16.0 NaN NaN NaN 2018 (counts 1-9) (counts 1-9)

Text under the table reads, “13833 rows times 30 columns.”

Figure 2. The Imported csv

This is a rather large table with 30 columns. The preview does not show them all;

Page 11 of 33 Learn to Create a Heat Map in Python With Data From NCHS (2018) SAGE SAGE Research Methods: Data 2021 SAGE Publications, Ltd. All Rights Reserved. Visualization hidden columns are indicated with the ellipsis symbol (…).> If you wish to change the number of columns/rows displayed, type pd.options.display.max_columns = 30/ pd.options.display.max_rows = 100.

Next, we want to select the data for plotting. You can enter all the following code in one cell if you like, but we will break it up for ease of explaining the various subparts.

A simple way of selecting a subset is by passing the loc method a set of conditions so to speak. The conditional expression is actually a pandas Series of boolean (True or False) values for each row in the original DataFrame. In this case, it returns True only for rows where the field TIME is 2019. Passing this Series selects the rows with True values.

Using boolean operators like & for “and” or | for “or” one can add multiple conditions for selection. For more details, see the Pandas introduction to subsetting.

To select just the data for California, input the following.

data_select = data.loc[data['Jurisdiction of Occurrence']=='California']

data_select

Then, as we are not going to use the flag columns, we should filter them out. The 17th column, index 16, is the last data column:

data_select = data_select.loc[:,data.columns[0:16]]

data_select

If we have many conditions, selecting a subset of the data can become a bit difficult to read as the name of the original dataframe is repeated for each

Page 12 of 33 Learn to Create a Heat Map in Python With Data From NCHS (2018) SAGE SAGE Research Methods: Data 2021 SAGE Publications, Ltd. All Rights Reserved. Visualization condition. An alternative way of selecting by conditions is using the query() method of the dataframe. See the User Guide on indexing for further details on selection and indexing methods: https://pandas.pydata.org/docs/user_guide/ indexing.html

To plot a heat map, we need a simple table with rows and columns, where every cell contains a numerical value—in this case, death counts.

Now, to have a look at influenza and pneumonia deaths by week, we can use Pandas’ pivot functionality:

data_pivot = data_select.pivot(index='MMWR Year', columns= 'MMWR Week', values='Influenza and pneumonia (J10-J18)')

data_pivot

The function pivot reshapes a DataFrame based on column values and returns a new DataFrame. The column passed to index becomes the index, the column or columns passed to columns becomes new columns, and values take one or several data values. Since we assign it values='Influenza and pneumonia (J10-J18)' all other causes of deaths are omitted.

2.1.5 Plotting the Chart We can now plot this table of weekly deaths as a heat map with the simple command ax = sns.heatmap(data_select_pivot) (Figure 3).

The code line is as follows:

[2]: data = pd.read_csv(‘Weekly_Counts_of_Deaths_by_State_and_Select_Causes__2014-2018.csv’, sep=‘;’, thousands=‘,’) data

Page 13 of 33 Learn to Create a Heat Map in Python With Data From NCHS (2018) SAGE SAGE Research Methods: Data 2021 SAGE Publications, Ltd. All Rights Reserved. Visualization The output table is as follows.