---
title: "Diamonds quality"
output: html_document
editor_options: 
  chunk_output_type: inline
---

```{r setup, include=FALSE}
# cache results
knitr::opts_chunk$set(cache=TRUE, echo = TRUE, eval = TRUE, message=FALSE, warning = FALSE, fig.align='center')
```
We explore the qualities of diamonds and their association with price. Why are low quality diamonds (with respect to cut, color and clarity) more expensive? 

```{r library}
## Analysis
library(dplyr)
library(ggplot2)
library(modelr)
```




Here is the dataset:

```{r import}
select(diamonds, price, carat:clarity) 
```

We observe that, apparently, low quality diamonds are more expensive:


```{r}
# best cut is Ideal, worst is Foor
group_by(diamonds, cut) %>% 
  summarise(median_price = median(price))

ggplot(diamonds, aes(x = cut, y = price)) + 
  geom_boxplot()

# best color is D, worst is J
group_by(diamonds, color) %>% 
  summarise(median_price = median(price))

ggplot(diamonds, aes(x = color, y = price)) + 
  geom_boxplot()

# best clarity is IF, worst is I1
group_by(diamonds, clarity) %>% 
  summarise(median_price = median(price))

ggplot(diamonds, aes(x = clarity, y = price)) + 
  geom_boxplot()



```

We first observe that best quality diamonds are smaller:

```{r}
# best cut is Ideal, worst is Foor
ggplot(diamonds, aes(x = cut, y = carat)) + 
  geom_boxplot()

# best color is D, worst is J
ggplot(diamonds, aes(x = color, y = carat)) + 
  geom_boxplot()

# best clarity is IF, worst is I1
ggplot(diamonds, aes(x = clarity, y = carat)) + 
  geom_boxplot()



```



We next remove the effect of carat on price:


```{r quality}
# Log-transform price and carat. The log-transformation is particularly useful here because it makes the pattern linear, and linear patterns are the easiest to work with.
diamonds = diamonds %>% 
  filter(carat <= 2.5) %>% 
  mutate(lprice = log2(price), lcarat = log2(carat))

# model prince in terms of carat
mod = lm(lprice ~ lcarat, data = diamonds)
cor(diamonds$lprice, diamonds$lcarat)
ggplot(diamonds, aes(x = lcarat, y = lprice)) +
  geom_hex()

# add residuals
diamonds = diamonds %>% add_residuals(mod)
cor(diamonds$resid, diamonds$lcarat)
ggplot(diamonds, aes(x = lcarat, y = resid)) +
  geom_hex()


```


We check again the correlation of quality on the residulas. Now low quality diamonds are less expensive.

```{r}

# best cut is Ideal, worst is Foor
ggplot(diamonds, aes(x = cut, y = price)) + 
  geom_boxplot()

ggplot(diamonds, aes(x = cut, y = resid)) + 
  geom_boxplot()


# best color is D, worst is J
ggplot(diamonds, aes(x = color, y = price)) + 
  geom_boxplot()

ggplot(diamonds, aes(x = color, y = resid)) + 
  geom_boxplot()


# best clarity is IF, worst is I1
ggplot(diamonds, aes(x = clarity, y = price)) + 
  geom_boxplot()

ggplot(diamonds, aes(x = clarity, y = resid)) + 
  geom_boxplot()




```


