This stuy case is based on the research question whether there is relation between town temperature and health risks. We would like to compare towns that have the most highest temperature with other towns in the Unitied States. The hypothesis is that the health risk value is higher for towns with higher temperature compared to ordinary towns. In particular the risk related to 'Adult Binge Drinking'. For this we need two datasets. The dataset of hottest towns and the dataset of USHealth_data.csv. The end goal is to run statistical comparion test. Before we can do such we first need to clean, organize and merge the data. The column 'Value' for instance contains not all float type data.
from bokeh.io import output_notebook
output_notebook()
Loading BokehJS ...
Load first dataset: hotest towns
First we load the text file hottest_towns.txt. This data we put in a dataframe. If we look with an editor in the file we see that the first two lines are not of interest, so we skip these. Furthermore there are no column labels so we add these.
It seems that we need to do some cleaning as well in the place column. States are abbreviated. Let us split the column like we did in the hottest town data set and than change the state with the map function.
Indicator Category 0
Indicator 0
Year 0
Sex 0
Race/Ethnicity 0
Value 0
Place 0
BCHC Requested Methodology 0
Source 4133
Methods 24226
Notes 19850
90% Confidence Level - Low 28214
90% Confidence Level - High 28214
95% Confidence Level - Low 22304
95% Confidence Level - High 22263
City 0
State 2154
dtype: int64
Let us inspect the indicator values. We were interested in adult binge drinking
set(df['Indicator'])
{'AIDS Diagnoses Rate (Per 100,000 people)',
'All Types of Cancer Mortality Rate (Age-Adjusted; Per 100,000 people)',
'All-Cause Mortality Rate (Age-Adjusted; Per 100,000 people)',
'Asthma Emergency Department Visit Rate (Age-Adjusted; Per 10,000)',
'Bike Score',
'Chlamydia Rate (Per 100,000 People)',
'Congenital Syphilis Rate (Per 100,000 Live Births)',
'Diabetes Mortality Rate (Age-Adjusted; Per 100,000 people)',
'Female Breast Cancer Mortality Rate (Age-Adjusted; Per 100,000 people)',
'Firearm-Related Emergency Department Visit Rate (Age-Adjusted; Per 10,000 people)',
'Firearm-Related Mortality Rate (Age-Adjusted; Per 100,000 people)',
'Gonorrhea Rate (Per 100,000 People)',
'HIV Diagnoses Rate (Per 100,000 people)',
'HIV-Related Mortality Rate (Age-Adjusted; Per 100,000 people)',
'Heart Disease Mortality Rate (Age-Adjusted; Per 100,000 people)',
'Homicide Rate (Age-Adjusted; Per 100,000 people)',
'Infant Mortality Rate (Per 1,000 live births)',
'Life Expectancy at Birth (Years)',
'Lung Cancer Mortality Rate (Age-Adjusted; Per 100,000 people)',
'Median Household Income (Dollars)',
'Motor Vehicle Mortality Rate (Age-Adjusted; Per 100,000 people)',
'Opioid-Related Unintentional Drug Overdose Mortality Rate (Age-Adjusted; Per 100,000 people)',
'Percent Foreign Born',
'Percent Living Below 200% Poverty Level',
'Percent Unemployed',
'Percent Who Only Speak English at Home',
'Percent Who Speak Spanish at Home',
'Percent of 3 and 4 Year Olds Currently Enrolled in Preschool',
'Percent of Adults 65 and Over Who Received Pneumonia Vaccine',
'Percent of Adults Who Are Obese',
'Percent of Adults Who Binge Drank',
'Percent of Adults Who Currently Smoke',
'Percent of Adults Who Meet CDC-Recommended Physical Activity Levels',
'Percent of Adults Who Received Seasonal Flu Shot',
'Percent of Children (Tested) Under Age 6 with Elevated Blood Lead Levels',
'Percent of Children Living in Poverty',
'Percent of Children Who Received Seasonal Flu Shot',
'Percent of High School Graduates (Over Age 18)',
'Percent of High School Students Who Are Obese',
'Percent of High School Students Who Binge Drank',
'Percent of High School Students Who Currently Smoke',
'Percent of High School Students Who Meet CDC-Recommended Physical Activity Levels',
'Percent of Households Whose Housing Costs Exceed 35% of Income',
'Percent of Low Birth Weight Babies Born',
'Percent of Mothers Under Age 20',
'Percent of Population 65 and Over',
'Percent of Population Under 18',
'Percent of Population Uninsured',
'Percent of Population with a Disability',
'Persons Living with HIV/AIDS Rate (Per 100,000 people)',
'Pneumonia and Influenza Mortality Rate (Age-Adjusted; Per 100,000 people)',
'Primary and Secondary Syphilis Rate (Per 100,000 People)',
'Race/Ethnicity (Percent)',
'Rate of Laboratory Confirmed Infections Caused by Salmonella (Per 100,000 people)',
'Rate of Laboratory Confirmed Infections Caused by Shiga Toxin-Producing E-Coli (Per 100,000 people)',
'Sex (Percent)',
'Suicide Rate (Age-Adjusted; Per 100,000 people)',
'Total Population (People)',
'Transit Score',
'Tuberculosis Incidence Rate (Per 100,000 people)',
'Walkability'}
df = df[df['Indicator'] == 'Percent of Adults Who Binge Drank']
len(df)
535
The question is whether the names of the cities and the states are comparable
As we can see we have to get rid of the (additional text) at some names. We can do such using regular expressions. The regular expression \s\([a-zA-Z\s\-]+\) matches any name starting with an opening bracket (, followed by a name including spaces or '-' followed by a closing bracket )
Now we need to create the second dataset without the hottest towns. We can use a trick to do such. We can create a list of towns and select all that is not in that list.
x = ht['City'].tolist()
nh = df.where(~df['City'].isin(x))
nh.head(2)
count 392.000000
mean 19.822194
std 7.395218
min 1.700000
25% 14.600000
50% 18.700000
75% 24.000000
max 54.000000
Name: Value, dtype: float64
y_nhot.describe()
count 103.000000
mean 18.218447
std 11.350605
min 6.300000
25% 12.800000
50% 15.300000
75% 21.150000
max 87.500000
Name: Value, dtype: float64
Visual inspection
A histogram can give insight in the distribution and whether both categories are comparable. From the following plot we can see that both distributions are comparable, but hottest town is a bit to the right if you ignore the outlier of not hottest towns
import numpy as np
from bokeh.layouts import gridplot
from bokeh.plotting import figure, show
def make_plot(data, color, label):
measured = data
hist, edges = np.histogram(measured, density=True, bins=50)
p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:],
fill_color=color, line_color="white", alpha=0.5, legend_label=label)
p.y_range.start = 0
p.legend.location = "center_right"
p.legend.background_fill_color = "#fefefe"
p.xaxis.axis_label = 'Percent of Adults Who Binge Drank'
p.yaxis.axis_label = 'Probability'
p.grid.grid_line_color="white"
return p
# Normal Distribution
p = figure(title="Distribution", tools='', background_fill_color="#fafafa")
p = make_plot(y_hot, 'red', 'hottest towns')
p = make_plot(y_nhot, 'green', 'normal towns')
show(p)
from scipy.stats import ttest_ind
ttest_ind(np.array(y_hot), np.array(y_nhot), equal_var = False)
