```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, tidy.opts=list(width.cutoff=60),tidy=TRUE, out.height = ".7\\textheight",fig.align='center')
library(tidyverse)
library(data.table)
library(lubridate)
```

---

# Where to get R?

1. Download [R](https://www.r-project.org/).

2. Download [RStudio](https://www.rstudio.com/products/rstudio/download/preview/).

3. Download Anaconda and create a Jupyter notebook that runs R. 

4. RGUI (default with R)

5. Any other ``IDE'' like Visual Studio Code

6. Installation specific extras:
  - **Windows:** Install [Rtools](https://cran.r-project.org/bin/windows/Rtools/). 
  - **Mac:** Install [Homebrew](https://brew.sh/). I also recommend that you configure/open your C++ toolchain (see [here](https://github.com/rmacoslib/r-macos-rtools#installer-package-for-macos-r-toolchain-).)
  - **Linux:** None (you should be good to go).

---

# Why R and RStudio?
```{R, indeeddotcom, echo=F}
# The popularity data
pop_df =
  data.frame(
  lang = c("SQL", "Python", "R", "SAS", "Matlab", "SPSS", "Stata"),
  n_jobs = c(107130, 66976, 48772, 25644, 11464, 3717, 1624),
  free = c(T, T, T, F, F, F, F)
  )
## Plot it
pop_df %>%
  mutate(lang = lang %>% factor(ordered = T)) %>%
  ggplot(aes(x = lang, y = n_jobs, fill = free)) +
  geom_col() +
  geom_hline(yintercept = 0) +
  aes(x = reorder(lang, -n_jobs), fill = reorder(free, -free)) +
  xlab("Statistical language") +
  scale_y_continuous(label = scales::comma) +
  ylab("Number of jobs") +
  labs(
    title = "Comparing statistical languages",
    subtitle = "Number of job postings on Indeed.com, 2019/01/06"
    ) +
  scale_fill_manual(
    "Free?",
    labels = c("True", "False"),
    values = c("#f92672", "darkslategray")
    ) +
  ggthemes::theme_pander(base_size = 17) +
  # theme_ipsum() +
  theme(legend.position = "bottom")
```

---

# Why R and RStudio? (cont.)

### Data science positivism

- Alongside Python, R has become the *de facto* language for data science.
  - See: [*The Impressive Growth of R*](https://stackoverflow.blog/2017/10/10/impressive-growth-r/), [*The Popularity of Data Science Software*](http://r4stats.com/articles/popularity/)
- Open-source (free!) with a global user-base spanning academia and industry.
  - "Do you want to be a profit source or a cost center?"

### Bridge to applied economics and other tools

- Already has all of the statistics and econometrics support, and is amazingly adaptable as a “glue” language to other
programming languages and APIs.
- The RStudio IDE and ecosystem allow for further, seamless integration.

---

# R References

- Pretty much every major package in R has a cheatsheet: https://www.rstudio.com/resources/cheatsheets/ with information

- Grant McDermott's lecture series is very useful: https://github.com/uo-ec607/lectures

- Len Goff (my grand-TA) put together two useful walkthroughs (that I borrowed from for this). 
  1. Slides: http://www.columbia.edu/~ltg2111/reveal/introtoR.html#/
  2. Jupyter Notebook: http://www.columbia.edu/~ltg2111/teaching/Intro%20to%20R.html

---

# Some R basics

1. Everything is an object.

2. Everything has a name.

3. You do things using functions.

4. Functions come pre-written in packages (i.e. "libraries"), although you can — and should — write your own functions too.

</br>

Points 1. and 2. can be summarised as an [object-orientated programming](https://en.wikipedia.org/wiki/Object-oriented_programming) (OOP) approach.

---

# R vs Stata (borrowed from Grant McDermott)

If you're coming from Stata, some additional things worth emphasizing:

- Multiple objects (e.g. data frames) can exist happily in the same workspace.
   - No more `keep`, `preserve`, `restore` hackery. (Though, props to [Stata 16](https://www.stata.com/new-in-stata/multiple-datasets-in-memory/).)
   - This is a direct consequence of the OOP approach.

- Load packages at the start of every new R session. Make peace with this.
  - "Base" R comes with tons of useful in-built functions and the tools to write your own functions.
  - However, many of R's best data science functions and tools come from external packages written by other users.

- R easily and infinitely parallelizes. For free.
  - Compare the cost of a [Stata/MP](https://www.stata.com/statamp/) license, nevermind the fact that you effectively pay per core...

- You don't need to `tset or xtset` your data. (Although you can too.)

---

# Variables, if statements, functions, printing output 

- Very important : `<-` == `=`, while `==` is for logical evaluation

```{r variable}
#Define a variable
x<-3
y=3
print(x, y)

#If/else statements
if(x<5){
    print("The value of x is less than 5")
} else{
    print("The value of x is greater than or equal to 5")
}

```

---

# Variables, if statements, functions, printing output (cont.)

```{r add5}
add5<-function(x){
    if(is.numeric(x)){
        theanswer<-x+5
        return(theanswer)
    }
    else{
        print("Argument x is not a number")
        return(0)
    }
}
print(paste0("42 plus five is equal to: ", add5(42)))

print(add5("sdafgdsfg"))
```
---

# Data organization

- Base R has a variety of data type objects called vectors, lists, matrices, arrays, factors, and dataframes.  

- Today I am focused on dataframes (and data tables) because they will be your workhorse data type.

```{r dataframes}
  df<-data.frame(variable1 = 1:10, variable2=c(2,4,6,8,10,12,14,16,18,20))
  print(head(df))

```

---

# Creating new variables
- Straight-forward to create new variables 

```{r dataframes2}
  df$variable2_squared <- df$variable2^2
  df['variable2_square_v2'] <- df$variable2^2
  print(head(df))
  df$variable2_squared<-NULL
  df['variable2_square_v2'] <- NULL
```

---

# Do I have define every function every time? 

- What if you want to create a function to sum two variables?

```{r untidy}

add_variables <- function(df,variable1='variable1',
                          variable2='variable2',new_var='new_var') {
  df[new_var] <- df[variable1]+df[variable2]
  return(df)
}

print(add_variables(df))

```

- This is a silly example, but most of coding can be simplified to a set of simple examples cobbled together into complex examples

---

# Reading in CSVs and .dta files

Much like Stata can read in csvs and whatnot, R can read in lots of file types

```{r readin}

projectpath<-"C:/Users/kgcsp/OneDrive/Documents/Education/DataTA/"
qcew_name <-paste0(projectpath,"raw data/qcew_wages_industrybyyear_naics.csv")
qcew <- read.csv(qcew_name)

```

It can also read stata files directly with the `foreign` package

```{r readstata, results='show'}
#install.packages(('readstata13'))
library(readstata13)

qcew <- read.dta13(paste0(projectpath,"raw data/qcew_wages_industrybyyear_naics.dta"))
print(head(qcew))

```

---

# Summarize the data a bit:

```{r meanmax}

print(paste0("Mean industry wage: ", mean(qcew$wage)))
print(paste0("Max industry wage: ", max(qcew$wage)))

```

Ick, so many decimals, let's cut those! 

```{r meanmaxpretty}

print(paste0("Mean industry wage: $", round(mean(qcew$wage),2)))
print(paste0("Mean industry wage: $", round(mean(qcew$wage),-3)/1000, "k"))

```

---

# Subset the data

- Let's subset the data now

```{r subset data}
print(qcew[qcew$naics==2121 & qcew$year==1990,])
```

- Huh, we just had to type `qcew` multiple times to subset the dataframe with Base R

- That is a little tedious and could make for hard to edit code in a larger file

---

# R code example (linear regression)

```{r fit}
fit = lm(mpg ~ wt, data = mtcars)
summary(fit)
```

---

# Base R plot

```{r mtcars_baseplot}
par(mar = c(4, 4, 1, .1)) ## Just for nice plot margins on this slide deck
plot(mtcars$wt, mtcars$mpg)
abline(fit, col = "red")
```

---

# ggplot2

```{r mtcars_ggplot,  message=FALSE}
library(ggplot2)
ggplot(data = mtcars, aes(x = wt, y = mpg)) + 
  geom_smooth(method = "lm", col = "red") + 
  geom_point() 
```

- Check out this tutorial: https://ggplot2.tidyverse.org/

---

# Keeping organized

- Tons of user-defined objects of dataframes can start to pile up in RStudio

- And functions can be tedious to write and troubleshoot

- R libraries like the `tidyverse` and `data.table` can save you tons of time

---

# Enter the tidyverse

- Hadley Wickham came up with the tidyverse to keep dataframe management simple

- What is ``tidy'' data? 

1. Each variable forms a column.
2. Each observation forms a row.
3. Each type of observational unit forms a table.

Answer: basically a Stata .dta file and is more likely to be in long format than wide.

```{r installtidy}

install.packages('tidyverse')

tidyverse_packages()

```

Lots of packages, but the most-used are

1. [**dplyr**](https://dplyr.tidyverse.org/)
2. [**tidyr**](https://tidyr.tidyverse.org/)

---

# Read in CSVs with dplyr

```{r readcsvdplyr, echo=T}
#Shows data types when read-in, unles you set `show_col_types` to false or col_tyes=`col()`
qcew <- read_csv(qcew_name)
print(qcew)
```

# # What did dplyr's read_csv do?

- The data frame looks funny -- it lists the type under the column

- This is a `tibble`, a special form of dataframe that is specific to `dplyr` with a few advantages for subsetting and viewing data

- You can convert back with `as.data.frame()` if needed for legacy code

---

# Data cleaning as ``grammar'': Key dplyr verbs

There are five key dplyr verbs that you need to learn.

1. `filter`: Filter (i.e. subset) rows based on their values.

2. `arrange`: Arrange (i.e. reorder) rows based on their values.

3. `select`: Select (i.e. subset) columns by their names: 

4. `mutate`: Create new columns.

5. `summarise`: Collapse multiple rows into a single summary value.<sup>1</sup>

6. pipes, `%>%`: Pass function output forward so you read your code left-to-right

</br>

Check out the tidyverse(https://dplyr.tidyverse.org/) website or Grant McDermott's slides(https://github.com/uo-ec607/lectures/blob/master/05-tidyverse/05-tidyverse.pdf) for explicit tutorials

---

# Lubridate to convert string dates to real dates

- Base R's default for dates is POSIXct, but it requires you specify the format for the date

- Within the tidyverse is `lubridate`, which guesses dates
  
```{r lubridate}
ymd(20201215)
```
```{r}
mdy("4/1/17")
```
```{r}
ymd_hms("2010-12-13 15:30:30")
```
```{r}
year(mdy("4/1/17"))
```

- The package `anytime` will guess the date in POSIXct, but that means adding one more dependency

---

# Pipes are great!

- Did you notice how I introduced a `%>%` above? That is called a pipe a name from coding in a UNIX system.

- The first argument of the next function is the object just before in the pipe.

- Pipes are like commas between phrases which help you read left to right.

```{r, eval = F}
## These next two blocks of code do exactly the same thing.
mpg %>% filter(manufacturer=="audi") %>% 
  group_by(model) %>% summarise(hwy_mean = mean(hwy))

summarise(group_by(filter(mpg, manufacturer=="audi"), 
  model), hwy_mean = mean(hwy))
```

The first line works through objects left to right as you think about them. 

The second line inverts the order

---

# Data.table

- While the tidyverse tends to break up operations step-by-step, data.table aims to do everything in one concise expression.

- We can execute complex data wrangling commands as a single, fluid thought.

---

# Read data with data.table

```{r readwithdatatable, echo=T}
#With data.table
qcew <- fread(qcew_name)
print(qcew)
```

---

# What did fread do?

- `fread` is way faster and creates a `data.table`, which is distinct from a data.frame

- Prints head and tail by default

---

# Data.table syntax

- data.table is a powerful data wrangling package that combines concise syntax with incredible performance. It is also very lightweight, despite being feature rich.

- The basic syntax is `DT[i, j, by]`
  - `i` On which rows?
  - `j` What to do?
  - `by` Grouped by what?

data.table introduces some new ideas like modify by reference (e.g. `:=`), as well as syntax (e.g. `.()`, `.SD` `.SDcols`, etc.).

- All of these ideas support data.table's core goals: Maximise performance and flexibility, whilst maintaining a concise and consistent syntax. They are worth learning.

Pro tip: Use keys to order your data and yield dramatic speed-ups.

The tidyverse and data.table are often viewed as substitutes, but you can profitably combine both into your workflow.

---

# Why use data.table?

- Another powerful tool to rival `dplyr` is `data.table`

- There are many battles about these two tools

- Data.table exceeds in a few areas:

1. Concise
2. Insanely fast
3. Memory efficient
4. Feature rich (and stable)
5. Dependency free

- Check out the CRAN document (https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html) 
- Also, Grant McDermott's slides on data.table (https://github.com/uo-ec607/lectures/blob/master/05-datatable/05-datatable.pdf) for tutorials and treatment of the `dplyr` vs. `data.table` debate

---

# Why not both?

Do you love dplyr's syntax, but want data.table's performance? 
--
Well, you're in luck!

--

Hadley Wickham's **dtplyr** package provides a data.table "back-end" for dplyr.
- Basically, write your code as if you were using dplyr and then it gets automatically translated to (and evaluated as) data.table code.

--

If this sounds appealing to you (and it should) I strongly encourage you to check out the [package website](https://dtplyr.tidyverse.org/) for details. But here's quick example, using our benchmark from earlier.

---

# How do I get help?

- If you start grabbing random packages off the internet, you're bound to have issues

- Luckily the syntax `?[command]` is a great way to get help

```{r help,eval=FALSE}
?read_csv
```
- If this conflicts with what it says online, then you may have an old package

- Also, R `vignette`'s give short examples of how to use packages: 
  -`vignette('dplyr')`
  -`vignette('ggplot2-in-packages')`

---

# Reshaping in R!

- I totally skipped reshaping today due to time constraints, but R has lots of tools for reshaping data
  - `dplyr` uses `pivot_wider` vs. `pivot_longer` reshape the `tibble`s
  - `data.table` uses `melt` for wide-to-long and `dcast`for long-to-wide (roughly)

---

# Conclusion

1. R and RStudio are powerful, free tools that are on the upswing in data science and economics applications

2. Base R has powerful tools to do everything you need

3. `dplyr` introduces a `grammar` of statistical programming, which makes for verbose, but clear coding

4. `data.table` trades concision for incredibly efficient and fast code

5. Plethora of other commands I skipped today that you will want to check out:
  
  - `data.table`'s  reshape data.table's
  