# Tidy data

### Structured and unstructured data

Data comes from many sources: sensor measurements, laboratory equipment measurements, dataloggers in factories, computer administration systems, cameras, apps, databases, websites, documents, spreadsheets, or even flat files. Most of this data is unstructured, like text, which is a sequence of words often not structured. Images and audio records or videos are examples of unstructured data. More structured data is data in tabular form with rows and columns, for instance, tables in spreadsheets or queries from a database.

A challenge in data science is to structure raw unstructured data into structured data since most of the data science tools and algorithms need a basic structure of rectangular data called a Data frame. In a Data frame, the rows contain the observations and the columns contain the features.

### Key terms for structured data

_**data frame**_ a two-dimensional matrix, rectangular data (like a spreadsheet), with rows containing records and columns containing features and labels (optional)

_**record**_ a row within the data frame containing the observation synonyms: sample, event, instance, example, case

_**feature**_ a column within the data frame containing the feature information synonyms: attribute, input, variable, x, independent variables

_**label**_ Sometimes the data frame contains columns with outcomes, to be modelled by a prediction model. synonyms: dependent variable, response, target, outcome, y

```
##   Sepal.Length Sepal.Width Petal.Length Petal.Width Species
## 1          5.1         3.5          1.4         0.2  setosa
## 2          4.9         3.0          1.4         0.2  setosa
## 3          4.7         3.2          1.3         0.2  setosa
## 4          4.6         3.1          1.5         0.2  setosa
## 5          5.0         3.6          1.4         0.2  setosa
## 6          5.4         3.9          1.7         0.4  setosa
```

a typical data frame. The principles are as follows:

* Each variable must have its own column
* Each observation must have its own row
* Each value must have its own cell

\[1] Source: R for Data Science, Hadley Wickham & Garrett Grolemund, CC-BY-NC-ND 3.0 US, [https://r4ds.had.co.nz/tidy-data.html](https://r4ds.had.co.nz/tidy-data.html)

### Data types

There are two basic types of structured data: numerical and categorical. Numerical data can be continuous and discrete. Categorical is data that has categories (like names of universities: RUG, Hanze, Stenden). But can be binary (yes/no) or in a certain order (less, sufficient, too much). The data type is important to determine which type of statistical model or machine learning model or visual display should be used in our data analysis. This is the reason why we classify data in software by type.

### Key terms for data types

_**numeric**_

data that can be measured on a numeric scale - _continuous_ data can be any value on an interval (interval, float, numeric) - _discrete_ data can only take whole numbers (integer, counts)

_**categorical**_

data that contains a specific set of categories (factors, nominals) - _logical_ categorical data with only two values (binary, boolean) - _ordinal_ categorical data that has an ordening

#### Inspect for data types

One of the important tasks in data science is checking for data types. Sometimes we need to typecast (change it to another data type) to be able to analyze our data with the specific statistical or graphical display tool we would like to use. This can be due to a mistake - the data was stored in the wrong data type format, for instance, numbers were stored as text.&#x20;

```r
#convert column "a" of a DataFrame to numeric
df["a"] = pd.to_numeric(df["a"])
```

This can also be because we want to use a specific tool that only can take numerical data. We can for instance then change the (yes/no) to (0/1). This is called `encoding`.

### Non-two-dimensional data structures

Nowadays graphs (networks) are also used to represent data structures. This lab will focus on two-dimensional data structures. The graph structures are out of scope.
