Heart failure Case study
Preparing for machine learning
Data processing
In the why we love numpy case, we applied linear regression to a randomly generated dataset. We can apply the same principles to data coming from experiments or studies as long as they are nicely structured in a 2d array format. Unfortunately, data from real-life cases is often not nicely structured. We need to manipulate the unstructured and/or messy data into a structured or clean form. We need to drop rows and columns because they are not needed for the analysis or because we cannot use them in case of too many missing values. Maybe we need to relabel columns or reformat characters into numerical values. Maybe we need to combine data from several sources. Cleaning and manipulating data into a structured form is called data processing. Data processing starts with data in its raw form and converts it into a more readable format (tables, graphs etc.), giving it the form and context necessary to be interpreted by computers and utilized by users.
In previous courses, you learned about the basics in programming python and object oriented python. In this course, we use python and the libraries NumPy and Pandas. These libraries are high-performance libraries especially suitable for data manipulations and data computations
.
Data processing Example: Heart failure casus
Cardiovascular diseases kill approximately 17 million people globally every year, and they mainly exhibit as myocardial infarctions and heart failures. Heart failure occurs when the heart cannot pump enough blood to meet the needs of the body. Available electronic medical records of patients quantify symptoms, body features, and clinical laboratory test values, which can be used to perform biostatistics analysis aimed at highlighting patterns and correlations otherwise undetectable by medical doctors. Machine learning can predict patients’ survival from their data and can individuate the most important features among those included in their medical records[1]. As a data scientist, you are required to inspect if the data can be used for modelling and to select the most important features for predicting the patient's survival. Data for the analysis is available in heart_failure_clinical_records_dataset.csv. The data description is to be found in the table data_description.csv
[1] https://doi.org/10.1186/s12911-020-1023-5
import pandas as pd
import numpy as npStep 1: Inspect the data
The first step is inspecting the data and getting an idea about the meaning of the variables, format, and units.
# load and display the meta data, the data that describes the data
md = pd.read_csv('data/data_description.csv', sep=';')
md
Feature
Explanation
Measurement
0
Age
Age of patient
years
1
Anaemia
Decrease of red blood cells or hemoglobin
Boolean
2
High blood pressure
If a patient has hypertension
Boolean
3
Creatinine phosphokinase
Level of the CPK enzyme in the blood
mcg/L
4
Diabetes
If the patient has diabetes
Boolean
5
Ejection fraction
Percentage of blood leaving the heart at each ...
Percentage
6
Sex
Woman or Man
Binary
7
Platelets
Platelets in the blood
kiloplatelets/mL
8
Serum creatinine
Level of creatinine in the blood
mg/dL
9
Serum sodium
Level of sodium in the blood
mEq/L
10
Smoking
If the patient smokes
Boolean
11
Time
Follow-up period
Days
12
death event
If the patient died during the follow-up period
Boolean
The death event will be used to predict the survival rate and will be the class variable. The variable death event is a boolean. If the death event is 1 (True) then the patient died. If the death event = 0 (False) then the patient survived
# load and display data
df = pd.read_csv('data/heart_failure_clinical_records_dataset.csv')
print(f'this dataset contains {len(df)} rows')
df.head(5)this dataset contains 299 rowsage
anaemia
creatinine_phosphokinase
diabetes
ejection_fraction
high_blood_pressure
platelets
serum_creatinine
serum_sodium
sex
smoking
time
DEATH_EVENT
0
75.0
0
582
0
20
1
265000.00
1.9
130
1.0
0.0
4
1
1
55.0
0
7861
0
38
0
263358.03
1.1
136
1.0
NaN
6
1
2
65.0
0
146
0
20
0
162000.00
1.3
129
1.0
1.0
7
1
3
50.0
1
111
0
20
0
210000.00
1.9
137
1.0
0.0
7
1
4
65.0
1
160
1
20
0
327000.00
2.7
116
0.0
0.0
8
1
list(df.columns)['age',
'anaemia',
'creatinine_phosphokinase',
'diabetes',
'ejection_fraction',
'high_blood_pressure',
'platelets',
'serum_creatinine',
'serum_sodium',
'sex',
'smoking',
'time',
'DEATH_EVENT']Mind you, the column names of the metadata are slightly different than the one from the clinical records. Also, the order is different. We must take that into account if we want to make use of the metadata to select a subset of the clinical records.
Missing data
Looking at the dataframe values we also see NaN in the column smoking. This means that the data contains missing data. Let us inspect the missing data
# first inspect missing data
df.isnull().sum()age 0
anaemia 0
creatinine_phosphokinase 0
diabetes 0
ejection_fraction 0
high_blood_pressure 0
platelets 0
serum_creatinine 0
serum_sodium 0
sex 1
smoking 1
time 0
DEATH_EVENT 0
dtype: int64The columns sex and smoking do have missing values. When columns have a lot of missing data we can think of dropping the column from the dataframe. In this case, we can either fill the column with a guessed value or we can drop the row.
df = df.dropna(axis = 0) # drop NaN rows
print(f'this dataset contains {len(df)} rows')
df.isnull().sum()this dataset contains 297 rows
age 0
anaemia 0
creatinine_phosphokinase 0
diabetes 0
ejection_fraction 0
high_blood_pressure 0
platelets 0
serum_creatinine 0
serum_sodium 0
sex 0
smoking 0
time 0
DEATH_EVENT 0
dtype: int64df.head()age
anaemia
creatinine_phosphokinase
diabetes
ejection_fraction
high_blood_pressure
platelets
serum_creatinine
serum_sodium
sex
smoking
time
DEATH_EVENT
0
75.0
0
582
0
20
1
265000.0
1.9
130
1.0
0.0
4
1
2
65.0
0
146
0
20
0
162000.0
1.3
129
1.0
1.0
7
1
3
50.0
1
111
0
20
0
210000.0
1.9
137
1.0
0.0
7
1
4
65.0
1
160
1
20
0
327000.0
2.7
116
0.0
0.0
8
1
5
90.0
1
47
0
40
1
204000.0
2.1
132
1.0
1.0
8
1
Furthermore, we can see that all the binary data and boolean Yes/No data is displayed by either a zero or a one. It might be unclear what this means when plotting the data.
df['sex'].value_counts()1.0 193
0.0 104
Name: sex, dtype: int64In the metadata we see the description "Woman" or "man", so we might want to change that.
df = pd.read_csv('data/heart_failure_clinical_records_dataset.csv')
df['sex'] = df['sex'].astype('category') # make the format categorical
df['sex'] = df['sex'].map({0:"Woman", 1: "Man"}) # map the values to the category
df['sex'].value_counts()Man 194
Woman 104
Name: sex, dtype: int64Inspect the datatypes
We changed the sex column to category, but what datatypes are the other columns?
df.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 299 entries, 0 to 298
Data columns (total 13 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 age 299 non-null float64
1 anaemia 299 non-null int64
2 creatinine_phosphokinase 299 non-null int64
3 diabetes 299 non-null int64
4 ejection_fraction 299 non-null int64
5 high_blood_pressure 299 non-null int64
6 platelets 299 non-null float64
7 serum_creatinine 299 non-null float64
8 serum_sodium 299 non-null int64
9 sex 298 non-null category
10 smoking 298 non-null float64
11 time 299 non-null int64
12 DEATH_EVENT 299 non-null int64
dtypes: category(1), float64(4), int64(8)
memory usage: 28.5 KBWe know that some of the integers should be booleans (logical). Let's change that
df["anaemia"] = df["anaemia"].astype('bool')
df["high_blood_pressure"] = df["high_blood_pressure"].astype('bool')
df["diabetes"] = df["diabetes"].astype('bool')
df["smoking"] = df["smoking"].astype('bool')
df["DEATH_EVENT"] = df["DEATH_EVENT"].astype('bool')
df.head()
age
anaemia
creatinine_phosphokinase
diabetes
ejection_fraction
high_blood_pressure
platelets
serum_creatinine
serum_sodium
sex
smoking
time
DEATH_EVENT
0
75.0
False
582
False
20
True
265000.00
1.9
130
Man
False
4
True
1
55.0
False
7861
False
38
False
263358.03
1.1
136
Man
True
6
True
2
65.0
False
146
False
20
False
162000.00
1.3
129
Man
True
7
True
3
50.0
True
111
False
20
False
210000.00
1.9
137
Man
False
7
True
4
65.0
True
160
True
20
False
327000.00
2.7
116
Woman
False
8
True
df['anaemia'].value_counts()False 170
True 129
Name: anaemia, dtype: int64df.info()<class 'pandas.core.frame.DataFrame'>
RangeIndex: 299 entries, 0 to 298
Data columns (total 13 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 age 299 non-null float64
1 anaemia 299 non-null bool
2 creatinine_phosphokinase 299 non-null int64
3 diabetes 299 non-null bool
4 ejection_fraction 299 non-null int64
5 high_blood_pressure 299 non-null bool
6 platelets 299 non-null float64
7 serum_creatinine 299 non-null float64
8 serum_sodium 299 non-null int64
9 sex 298 non-null category
10 smoking 299 non-null bool
11 time 299 non-null int64
12 DEATH_EVENT 299 non-null bool
dtypes: bool(5), category(1), float64(3), int64(4)
memory usage: 18.3 KBStep 2: Explore data
It is useful to understand the range of the data. A function that displays the descriptives of the numerical data is describe
df.describe()age
creatinine_phosphokinase
ejection_fraction
platelets
serum_creatinine
serum_sodium
time
count
299.000000
299.000000
299.000000
299.000000
299.00000
299.000000
299.000000
mean
60.833893
581.839465
38.083612
263358.029264
1.39388
136.625418
130.260870
std
11.894809
970.287881
11.834841
97804.236869
1.03451
4.412477
77.614208
min
40.000000
23.000000
14.000000
25100.000000
0.50000
113.000000
4.000000
25%
51.000000
116.500000
30.000000
212500.000000
0.90000
134.000000
73.000000
50%
60.000000
250.000000
38.000000
262000.000000
1.10000
137.000000
115.000000
75%
70.000000
582.000000
45.000000
303500.000000
1.40000
140.000000
203.000000
max
95.000000
7861.000000
80.000000
850000.000000
9.40000
148.000000
285.000000
What we can see is that the data ranges differ per feature. If we want to use the data for prediction we need to normalize the data later on. We can do that with numpy. From the describe table we can also see that most of the data is not symmetric distributed. Let us inspect the distributions by plotting.
Plotting the data
Plot distributions of numeric values
#plot numeric values distributions
df_num = df.select_dtypes(include=['float64', 'int64'])from bokeh.layouts import gridplot
from bokeh.plotting import figure, output_file, show
def make_plot(title, hist, edges):
p = figure(title=title, tools='', background_fill_color="#fafafa")
p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:],
fill_color="navy", line_color="white", alpha=0.5)
p.y_range.start = 0
p.xaxis.axis_label = 'value'
p.yaxis.axis_label = 'count'
p.grid.grid_line_color="white"
return p
# Distribution
g = []
for i in range(len(df_num.columns)):
hist, edges = np.histogram(df_num[df_num.columns[i]], bins=40)
p = make_plot(f" {df_num.columns[i]}", hist, edges)
g.append(p)
output_file('histogram.html', title="distribution plots")
show(gridplot(g, ncols=4, plot_width=250, plot_height=250, toolbar_location=None))
grouped = pd.DataFrame(df.groupby('time')['DEATH_EVENT'].sum())
print(grouped.head(10)) DEATH_EVENT
time
4 1.0
6 1.0
7 2.0
8 2.0
10 6.0
11 2.0
12 0.0
13 1.0
14 2.0
15 2.0p = figure(title="death events in time")
p.vbar(x='time', top='DEATH_EVENT', width=0.9, source=grouped)
p.xaxis.axis_label = 'number of days'
p.yaxis.axis_label = 'number of deaths'
show(p)
output_file('bar.html')
Step 3: Clean data
Based on the inspecting and exploration of the data it is decided to drop the column time. The feature time will not be used for prediction. All the other variables will be used for further analysis. For computation convenience, the int64 data is used instead of booleans and categories. Furthermore, the data needs to be transformed and normalized.
import numpy as np
df = pd.read_csv('data/heart_failure_clinical_records_dataset.csv')
df = df.dropna(axis = 0) # drop NaN rows
df = df.drop(['time'],axis = 1) # drop time column
df = df[df.apply(lambda x: np.abs(x - x.mean()) / x.std() < 3).all(axis=1)] # remove outliers
print(f'this dataset contains {len(df)} rows and {len(df.columns)} columns')
df.head()this dataset contains 280 rows and 12 columnsage
anaemia
creatinine_phosphokinase
diabetes
ejection_fraction
high_blood_pressure
platelets
serum_creatinine
serum_sodium
sex
smoking
DEATH_EVENT
0
75.0
0
582
0
20
1
265000.0
1.9
130
1.0
0.0
1
2
65.0
0
146
0
20
0
162000.0
1.3
129
1.0
1.0
1
3
50.0
1
111
0
20
0
210000.0
1.9
137
1.0
0.0
1
5
90.0
1
47
0
40
1
204000.0
2.1
132
1.0
1.0
1
6
75.0
1
246
0
15
0
127000.0
1.2
137
1.0
0.0
1
Step 4: Split into features matrix and class vector. Normalize features
y = np.array(df['DEATH_EVENT'])
X = np.array(df.iloc[:,0:11])
print(y.shape)
print(X.shape)
y = y.reshape(-1, 1)
print(y.shape)(280,)
(280, 11)
(280, 1)# normaliseer data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler = scaler.fit(X)
X = scaler.transform(X)We now have a cleaned normalized feature matrix and a class variable vector. We successfully prepared the dataset for machine learning algorithms in order to predict the heart failure death event.
Last updated