{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "http://pandas.pydata.org/pandas-docs/stable/10min.html" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# **10 Minutes to pandas plus Geopandas Example**\n", "\n", "This is a short introduction to pandas, geared mainly for new users. You can see more complex recipes in the [Cookbook](http://pandas.pydata.org/pandas-docs/stable/cookbook.html#cookbook). It is from official pandas documentation.\n", "\n", "I add a geopandas example at the end to show what Python can do quite easily." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# The Main Tutorial" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Customarily, we import as follows:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# tools for data analysis\n", "import pandas as pd\n", "import numpy as np\n", "\n", "# general python tools\n", "import requests, zipfile, io, os\n", "\n", "#plotting packages\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "\n", "# working with geodata\n", "from shapely.geometry import Point\n", "import geopandas as gpd # geopandas!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Object Creation\n", "\n", "See the [Data Structure Intro section](http://pandas.pydata.org/pandas-docs/stable/dsintro.html#dsintro) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Creating a Series by passing a list of values, letting pandas create a default integer index:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "s = pd.Series([1,3,5,np.nan,6,8])" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0 3.0\n", "1 9.0\n", "2 15.0\n", "3 NaN\n", "4 18.0\n", "5 24.0\n", "dtype: float64" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "s*3" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "s+2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Creating a DataFrame by passing a numpy array, with a datetime index and labeled columns:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "dates = pd.date_range('20130101', periods=6)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "dates" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df = pd.DataFrame(np.random.randn(6,4), index=dates, columns=list('ABCD'))" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ABCD
2013-01-01-0.351095-1.4120390.801410-0.493340
2013-01-02-1.5946460.5736160.650687-0.337713
2013-01-030.5821100.7770100.2534690.077428
2013-01-04-0.9238680.960250-2.0357410.061624
2013-01-05-0.205958-0.079455-1.0517521.288812
2013-01-061.950005-0.7603630.975510-0.984661
\n", "
" ], "text/plain": [ " A B C D\n", "2013-01-01 -0.351095 -1.412039 0.801410 -0.493340\n", "2013-01-02 -1.594646 0.573616 0.650687 -0.337713\n", "2013-01-03 0.582110 0.777010 0.253469 0.077428\n", "2013-01-04 -0.923868 0.960250 -2.035741 0.061624\n", "2013-01-05 -0.205958 -0.079455 -1.051752 1.288812\n", "2013-01-06 1.950005 -0.760363 0.975510 -0.984661" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df[\"A\"] # '' \"\" can be used interchangeably" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.A" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Creating a DataFrame by passing a dict of objects that can be converted to series-like." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df2 = pd.DataFrame({'A':1.,\n", " 'B':pd.Timestamp('20130102'),\n", " 'C':pd.Series(1,index=list(range(4)),dtype='float32'),\n", " 'D':np.array([3]*4,dtype='int32'),\n", " 'E':pd.Categorical([\"test\",\"train\",\"test\",\"train\"]),\n", " 'F':'foo'})" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Having specific [dtypes](http://pandas.pydata.org/pandas-docs/stable/basics.html#basics-dtypes)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df2.dtypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you’re using IPython, tab completion for column names (as well as public attributes) is automatically enabled. Here’s a subset of the attributes that will be completed:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "# df2." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the columns A, B, C, and D are automatically tab completed. E is there as well; the rest of the attributes have been truncated for brevity." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Viewing Data\n", "\n", "See the [Basics section](http://pandas.pydata.org/pandas-docs/stable/basics.html#basics) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "See the top & bottom rows of the frame" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.tail(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Display the index, columns, and the underlying numpy data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.index" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.columns" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "scrolled": true }, "outputs": [], "source": [ "df.values" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Describe shows a quick statistic summary of your data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Transposing your data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.T" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sorting by an axis" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.sort_index(axis=1, ascending=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sorting by value" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.sort_values(by='B')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Selection" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note:** While standard Python / Numpy expressions for selecting and setting are intuitive and come in handy for interactive work, for production code, we recommend the optimized pandas data access methods, .at, .iat, .loc, .iloc and .ix." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "See the indexing documentation [Indexing and Selecting Data](http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing) and [MultiIndex / Advanced Indexing](http://pandas.pydata.org/pandas-docs/stable/advanced.html#advanced)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Getting" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Selecting a single column, which yields a Series, equivalent to df.A" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df['A']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Selecting via [], which slices the rows." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df[0:3]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df['20130102':'20130104']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Selection by Label\n", "\n", "See more in [Selection by Label](Selection by Label)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dates[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For getting a cross section using a label" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.loc[dates[0]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Selection by Label" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.loc[:,['A','B']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Showing label slicing, both endpoints are included" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.loc['20130102':'20130104',['A','B']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Reduction in the dimensions of the returned object" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.loc['20130102',['A','B']]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For getting a scalar value" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.loc[dates[0],'A']" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Selection by Position\n", "\n", "See more in [Selection by Position](http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-integer)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Select via the position of the passed integers" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.iloc[3]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By integer slices, acting similar to numpy/python" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.iloc[3:5,0:2]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By lists of integer position locations, similar to the numpy/python style" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.iloc[[1,2,4],[0,2]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For slicing rows explicitly" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.iloc[1:3,:]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For slicing columns explicitly" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.iloc[:,1:3]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For getting a value explicitly" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.iloc[1,1]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For getting fast access to a scalar (equiv to the prior method)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.iat[1,1]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Boolean Indexing" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using a single column’s values to select data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "scrolled": true }, "outputs": [], "source": [ "df[df.A > 0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A where operation for getting." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df[df > 0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Using the isin() method for filtering:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "scrolled": true }, "outputs": [], "source": [ "df2 = df.copy()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df2['E'] = ['one','one', 'two','three','four','three']" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df2" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df2[df2['E'].isin(['two','four'])]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setting" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Setting a new column automatically aligns the data by the indexes" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "s1 = pd.Series([1,2,3,4,5,6], index=pd.date_range('20130102',periods=6))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "s1" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df['F'] = s1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Setting values by label" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.at[dates[0],'A'] = 0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Settomg values by position" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.iat[0,1] = 0" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Setting by assigning with a numpy array" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.loc[:,'D'] = np.array([5] * len(df))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The result of the prior setting operations" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A where operation with setting." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df2 = df.copy()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df2[df2 > 0] = -df2" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Missing Data\n", "\n", "pandas primarily uses the value np.nan to represent missing data. It is by default not included in computations. See the Missing Data section" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Reindexing allows you to change/add/delete the index on a specified axis. This returns a copy of the data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df1 = df.reindex(index=dates[0:4], columns=list(df.columns) + ['E'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df1.loc[dates[0]:dates[1],'E'] = 1" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To drop any rows that have missing data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df1.dropna(how='any')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Filling missing data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df1.fillna(value=5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get the boolean mask where values are nan" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "pd.isnull(df1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Operations\n", "\n", "See the [Basic section on Binary Ops](http://pandas.pydata.org/pandas-docs/stable/basics.html#basics-binop)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Stats\n", "\n", "Operations in general exclude missing data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Performing a descriptive statistic" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Same operation on the other axis" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.mean(1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Operating with objects that have different dimensionality and need alignment. In addition, pandas automatically broadcasts along the specified dimension." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "s = pd.Series([1,3,5,np.nan,6,8], index=dates).shift(2)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "s" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.sub(s, axis='index')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Apply" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Applying functions to the data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.apply(np.cumsum)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "x = df.apply(lambda x: x.max() - x.min())" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "x" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Histogramming\n", "\n", "See more at [Histogramming and Discretization](http://pandas.pydata.org/pandas-docs/stable/basics.html#basics-discretization)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "s = pd.Series(np.random.randint(0, 7, size=10))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "s" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "s.value_counts()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### String Methods\n", "\n", "Series is equipped with a set of string processing methods in the str attribute that make it easy to operate on each element of the array, as in the code snippet below. Note that pattern-matching in str generally uses [regular expressions](https://docs.python.org/2/library/re.html) by default (and in some cases always uses them). See more at [Vectorized String Methods](http://pandas.pydata.org/pandas-docs/stable/text.html#text-string-methods)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "s = pd.Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "s.str.lower()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Merge" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Concat\n", "\n", "pandas provides various facilities for easily combining together Series, DataFrame, and Panel objects with various kinds of set logic for the indexes and relational algebra functionality in the case of join / merge-type operations.\n", "\n", "See the [Merging section](http://pandas.pydata.org/pandas-docs/stable/merging.html#merging)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Concatenating pandas objects together with concat():" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df = pd.DataFrame(np.random.randn(10, 4))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "# break it into pieces\n", "pieces = [df[:3], df[3:7], df[7:]]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pieces" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "pd.concat(pieces)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Join\n", "\n", "SQL style merges. See the [Database style joining](http://pandas.pydata.org/pandas-docs/stable/merging.html#merging-join)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "left = pd.DataFrame({'key': ['foo', 'foo'], 'lval': [1, 2]})" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "right = pd.DataFrame({'key': ['foo', 'foo'], 'rval': [4, 5]})" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "left" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "right" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "pd.merge(left, right, on='key')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Append\n", "\n", "Append rows to a dataframe. See the [Appending](http://pandas.pydata.org/pandas-docs/stable/merging.html#merging-concatenation)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df = pd.DataFrame(np.random.randn(8, 4), columns=['A','B','C','D'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "s = df.iloc[3]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.append(s, ignore_index=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Grouping\n", "\n", "By “group by” we are referring to a process involving one or more of the following steps\n", "\n", "* **Splitting** the data into groups based on some criteria\n", "* **Applying** a function to each group independently\n", "* **Combining** the results into a data structure\n", "\n", "See the [Grouping section](http://pandas.pydata.org/pandas-docs/stable/groupby.html#groupby)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar', 'foo', 'bar', 'foo', 'foo'],\n", " 'B' : ['one', 'one', 'two', 'three','two', 'two', 'one', 'three'],\n", " 'C' : np.random.randn(8),\n", " 'D' : np.random.randn(8)})" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Grouping and then applying a function sum to the resulting groups." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.groupby('A').sum()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.groupby(['A','B']).sum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reshaping\n", "\n", "See the sections on [Hierarchical Indexing](http://pandas.pydata.org/pandas-docs/stable/advanced.html#advanced-hierarchical) and [Reshaping](http://pandas.pydata.org/pandas-docs/stable/reshaping.html#reshaping-stacking)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Stack" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "tuples = list(zip(*[['bar', 'bar', 'baz', 'baz', 'foo', 'foo', 'qux', 'qux'],\n", " ['one', 'two', 'one', 'two', 'one', 'two', 'one', 'two']]))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "index = pd.MultiIndex.from_tuples(tuples, names=['first', 'second'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = pd.DataFrame(np.random.randn(8, 2), index=index, columns=['A', 'B'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df2 = df[:4]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The stack() method “compresses” a level in the DataFrame’s columns." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "stacked = df2.stack()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "stacked" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With a “stacked” DataFrame or Series (having a MultiIndex as the index), the inverse operation of stack() is unstack(), which by default unstacks the **last level**:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "stacked.unstack()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "stacked.unstack(1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "stacked.unstack(0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Pivot Tables\n", "\n", "See the section on [Pivot Tables](http://pandas.pydata.org/pandas-docs/stable/reshaping.html#reshaping-pivot)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = pd.DataFrame({'A' : ['one', 'one', 'two', 'three'] * 3,\n", " 'B' : ['A', 'B', 'C'] * 4,\n", " 'C' : ['foo', 'foo', 'foo', 'bar', 'bar', 'bar'] * 2,\n", " 'D' : np.random.randn(12),\n", " 'E' : np.random.randn(12)})" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can produce pivot tables from this data very easily:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "pd.pivot_table(df, values='D', index=['A', 'B'], columns=['C'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Time Series\n", "\n", "pandas has simple, powerful, and efficient functionality for performing resampling operations during frequency conversion (e.g., converting secondly data into 5-minutely data). This is extremely common in, but not limited to, financial applications. See the [Time Series section](http://pandas.pydata.org/pandas-docs/stable/timeseries.html#timeseries)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "rng = pd.date_range('1/1/2012', periods=100, freq='S')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ts = pd.Series(np.random.randint(0, 500, len(rng)), index=rng)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "ts.resample('5Min').sum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Time zone representation" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "rng = pd.date_range('3/6/2012 00:00', periods=5, freq='D')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ts = pd.Series(np.random.randn(len(rng)), rng)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "ts" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ts_utc = ts.tz_localize('UTC')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "ts_utc" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Convert to another time zone" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "ts_utc.tz_convert('US/Eastern')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Converting between time span representations" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "rng = pd.date_range('1/1/2012', periods=5, freq='M')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ts = pd.Series(np.random.randn(len(rng)), index=rng)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "ts" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ps = ts.to_period()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "ps" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "ps.to_timestamp()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Converting between period and timestamp enables some convenient arithmetic functions to be used. In the following example, we convert a quarterly frequency with year ending in November to 9am of the end of the month following the quarter end:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "prng = pd.period_range('1990Q1', '2000Q4', freq='Q-NOV')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ts = pd.Series(np.random.randn(len(prng)), prng)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ts.index = (prng.asfreq('M', 'e') + 1).asfreq('H', 's') + 9" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "ts.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Categoricals" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since version 0.15, pandas can include categorical data in a DataFrame. For full docs, see the [categorical introduction](http://pandas.pydata.org/pandas-docs/stable/categorical.html#categorical) and the [API documentation](http://pandas.pydata.org/pandas-docs/stable/api.html#api-categorical)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = pd.DataFrame({\"id\":[1,2,3,4,5,6], \"raw_grade\":['a', 'b', 'b', 'a', 'a', 'e']})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Convert the raw grades to a categorical data type." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df[\"grade\"] = df[\"raw_grade\"].astype(\"category\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df[\"grade\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Rename the categories to more meaningful names (assigning to Series.cat.categories is inplace!)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df[\"grade\"].cat.categories = [\"very good\", \"good\", \"very bad\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Reorder the categories and simultaneously add the missing categories (methods under Series .cat return a new Series per default)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df[\"grade\"] = df[\"grade\"].cat.set_categories([\"very bad\", \"bad\", \"medium\", \"good\", \"very good\"])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df[\"grade\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sorting is per order in the categories, not lexical order." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.sort_values(by=\"grade\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Grouping by a categorical column shows also empty categories." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "df.groupby(\"grade\").size()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plotting\n", "[Plotting](http://pandas.pydata.org/pandas-docs/stable/visualization.html#visualization) docs." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ts = ts.cumsum()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "ts.plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "On DataFrame, plot() is a convenience to plot all of the columns with labels:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = pd.DataFrame(np.random.randn(1000, 4), index=ts.index,\n", " columns=['A', 'B', 'C', 'D'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = df.cumsum()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "plt.figure(); df.plot(); plt.legend(loc='best')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Getting Data In/Out" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### CSV\n", "[Writing to a csv file](http://pandas.pydata.org/pandas-docs/stable/io.html#io-store-in-csv)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.to_csv('foo.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[Reading from a csv file](http://pandas.pydata.org/pandas-docs/stable/io.html#io-read-csv-table)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "pd.read_csv('foo.csv')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### HDF5\n", "Reading and writing to [HDFStores](http://pandas.pydata.org/pandas-docs/stable/io.html#io-hdf5)\n", "\n", "Writing to a HDF5 Store" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.to_hdf('foo.h5','df')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Reading from a HDF5 Store" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "pd.read_hdf('foo.h5','df')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Excel\n", "\n", "Reading and writing to [MS Excel](http://pandas.pydata.org/pandas-docs/stable/io.html#io-excel)\n", "\n", "Writing to an excel file" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.to_excel('foo.xlsx', sheet_name='Sheet1')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Reading from an excel file" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "pd.read_excel('foo.xlsx', 'Sheet1', index_col=None, na_values=['NA'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Gotchas\n", "If you are trying an operation and you see an exception like:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false } }, "outputs": [], "source": [ "if pd.Series([False, True, False]):\n", " print(\"I was true\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "See [Comparisons](http://pandas.pydata.org/pandas-docs/stable/basics.html#basics-compare) for an explanation and what to do.\n", "\n", "See [Gotchas](http://pandas.pydata.org/pandas-docs/stable/gotchas.html#gotchas) as well." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "# Geopandas\n", "\n", "Let's combine a bunch of that knowledge and gain some more!\n", "\n", "I am going an example that you can work through. We want to create a map of the continental US that has (by state) a heatmap showing how many starbucks are in that State." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# First we read in the data\n", "df = pd.read_csv(\"https://introcs.cs.princeton.edu/java/data/starbucks.csv\",names=['longitude','latitude','store_name','address'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# the data come from Princeton and may not be up to date.\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We need to tell python that we are dealing with a \"geodataframe\" so we make official geometries from the lat/longs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Convering a regular dataframe into a geodataframe\n", "\n", "**Important:** Not covered is CRS projections. If you want to compare distances, you will need to think carefully about the projection you are using and make sure that your layers use the same one." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = gpd.GeoDataFrame(df,geometry=[Point(xy) for xy in zip(df.longitude,df.latitude)])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looks good, but we can do better. This is just overlapping points. We want to overlay those onto states to create our heatmap. We will use publicly available data from the Census to get shapefiles for the US states. I use a few other tools (like requests) to do the pull. This will create a folder called states_shapefiles/ in your working directory that contains the necessary information" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# make a folder to store the US states shapefiles\n", "if 'states_shapefiles' not in os.listdir():\n", " os.mkdir('states_shapefiles')\n", "# and go in\n", "os.chdir('states_shapefiles/')\n", "\n", "# pull in the us state boundaries here\n", "r = requests.get(\"https://www2.census.gov/geo/tiger/GENZ2018/shp/cb_2018_us_state_5m.zip\", stream=True)\n", "z = zipfile.ZipFile(io.BytesIO(r.content))\n", "z.extractall()\n", "\n", "os.chdir('..')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states = gpd.read_file('states_shapefiles/')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Geodata is great. Graphics are automatic and here geopandas knows to plot geometries instead of points!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states.plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that this is super zoomed out. That's because we have non-continental US states in this map. Let's get rid of these. First let's convert the state FIP code to a number so we can compare it numerically (it's loaded in as a string)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states.STATEFP = pd.to_numeric(states.STATEFP,errors='coerce')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we get rid of irrelevant states" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states = states.loc[states.STATEFP.le(58) & states.NAME.isin(['Alaska','Hawaii']).eq(False),:]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states.plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Mapping the points into the states\n", "\n", "Great! Now we can use powerful tools under the hood of geopandas to place these points into our states. Let's make a function to help us. If the starbucks is located in a state we already deleted, then we will get a missing value. Otherwise it will return the state that starbucks is in" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def return_state(point,states=states):\n", " # find out where the starbucks is (if starbucks on state border could return multiple true/false)\n", " # or all false (none) if no matches\n", " in_truefalse = states.loc[states.geometry.contains(point),'NAME']\n", " # if this isn't empty then let's get the state\n", " if not in_truefalse.empty:\n", " return in_truefalse.values[0]\n", " return np.nan\n", " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's apply this function to each point in the dataframe of starbucks locations to get the state that each point is in." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.loc[:,'state'] = df.geometry.apply(return_state)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Of course all of the Hawaii ones didn't work. Let's look at the tail." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df.tail()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Great! Now let's groupby state and count the number of stores." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "numstores = df.groupby('state',as_index=False).address.count().rename({'address':'numstores'},axis=1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "numstores.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Merging Data Together and Bringing in Population" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# a simple merge of the states with this new count.\n", "states = states.merge(numstores,left_on='NAME',right_on='state',how='left')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# population from the US Census\n", "pops = pd.read_csv('https://www2.census.gov/programs-surveys/popest/datasets/2010/2010-eval-estimates/co-est2010-totals.csv',encoding='latin-1')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# we don't need all the counties, just the states (County code= 0)\n", "pops = pops.loc[pops.COUNTY.eq(0),['STNAME','CENSUS2000POP']]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pops.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states = states.merge(pops,left_on='NAME',right_on='STNAME',how='left')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# again the census data is annoyingly a string by default.\n", "states.CENSUS2000POP = pd.to_numeric(states.CENSUS2000POP,errors='coerce')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states.loc[:,'starbucks_per_million'] = 1e6*states.numstores/states.CENSUS2000POP" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states.starbucks_per_million.hist()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As a brief aside, we can make these look WAY nicer very easily with seaborn" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# seaborn is just a wrapper for matplotlib that makes it \"pretty\". This sets the style to be matplotlib settings\n", "sns.set(font='Palatino')\n", "sns.set_style('whitegrid',{'font':'Palatino','grid.linestyle': 'dotted'})" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states.starbucks_per_million.hist()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "I will disable seaborn for the geoplotting though since the wrapper makes grid lines bigger and changes colors in unexpected ways." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sns.reset_orig() # this goes back to the original matplotlib settings" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Making the chloropleth graph!\n", "\n", "Ok so now we have the stores per million with some interesting variation. So let's plot it on the state map. Sound complicated? Not really actually" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states.plot(column='starbucks_per_million',cmap='Blues')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Ok this is close, but we have some issues:\n", "1. First is that the scaling is skewed such that some states are really dark and others are essentially white\n", "2. We have axes that mean nothing\n", "3. We have no legend that tells us what anything means.\n", "4. We have no borders around states to identify them" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "2, and 4 are quite easy to fix: First we create a blank figure and fill it (so we can plot multiple things on the same axis)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots()\n", "states.boundary.plot(linewidth=.25,ax=ax,color='grey')\n", "states.plot(column='starbucks_per_million',cmap='Blues',ax=ax)\n", "ax.axis('off')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Adding a Legend\n", "This is looking great, but what do the colors mean? Let's start by taking a closer look at the data we are plotting." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "states.starbucks_per_million.describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see from this that the variation is from 0 to 100 but it is not distributed equally. Let's pull key quantiles from this distribution and make a legend that corresponds to those.\n", "\n", "First get the 20-100 percentile of the distribution that will define the right tails of the buckets:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bins = states.starbucks_per_million.describe(percentiles=[.2,.4,.6,.8,1])[['20%','40%','60%','80%','max']].values" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bins" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The Final Result" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots()\n", "states.boundary.plot(linewidth=.25,ax=ax,color='grey')\n", "states.plot(column='starbucks_per_million',cmap='Blues',ax=ax)\n", "ax.axis('off')\n", "states.plot(column='starbucks_per_million',\n", " cmap='Blues', scheme='User_Defined', \n", " classification_kwds=dict(bins=bins),ax=ax,legend=True,\n", " legend_kwds=dict(loc='upper right',\n", " bbox_to_anchor=(1.3, .8), # this places the legend (the units are inches)\n", " fontsize='small',\n", " title=\"Stores per 1 Million\",\n", " frameon=False))\n", "plt.title('Starbucks Stores Across the US By Population')" ] } ], "metadata": { "interpreter": { "hash": "326b8e1be21a4b3e9df534b73e5211e8d89f7021163bfabd07ca575e6b1022aa" }, "kernelspec": { "display_name": "Python 3.7.10 64-bit ('base': conda)", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.10" } }, "nbformat": 4, "nbformat_minor": 4 }