[![](https://img.shields.io/badge/download%20html-blue)](../rendered-html/data-manipulation.html)
[![](https://img.shields.io/badge/download%20Rmd-blue)](data-manipulation.Rmd)

```{r setup, include=FALSE}
# Set global options for all code chunks in this document
knitr::opts_chunk$set(echo = TRUE)  # Display code alongside its output
```

## Introduction to the Tidyverse

The tidyverse is a collection of R packages designed for data science that share a common philosophy and design. These packages are essential for modern data analysis in R, especially for bioinformatics and genomics research.

Key packages we'll focus on:
- **dplyr**: For efficient data manipulation and transformation
- **tidyr**: For cleaning and reshaping messy data into tidy format

Let's start by loading the required packages:

```{r load-packages}
# First, check if tidyverse is installed and install if needed
# This is good practice for reproducible code
if (!require("tidyverse")) install.packages("tidyverse")

# Load the tidyverse meta-package
# This includes: dplyr, tidyr, ggplot2, and other essential packages
library(tidyverse)
```

## Helpful Resources

Before we begin, here are some valuable resources to keep handy:

1. **RStudio Cheat Sheets**: 
   - [Data Transformation with dplyr](https://github.com/rstudio/cheatsheets/blob/main/data-transformation.pdf)
   - [Data Tidying with tidyr](https://github.com/rstudio/cheatsheets/blob/main/tidyr.pdf)
   These cheat sheets are invaluable quick references for data manipulation functions.

2. **Keyboard Shortcuts**:
   - Pipe operator (`%>%`): Ctrl/Cmd + Shift + M
   - Assignment operator (`<-`): Alt + - (Windows/Linux) or Option + - (Mac)
   These shortcuts will speed up your coding significantly!

## Sample Dataset

We'll use a gene expression dataset that mimics real RNA-seq data:

```{r create-data}
# Create a sample gene expression dataset with:
# - Gene identifiers
# - Expression values for control and treated samples (with replicates)
# - Genomic information (chromosome location)
# - Biological pathway information
gene_data <- tibble(
  gene_id = c("BRCA1", "TP53", "EGFR", "KRAS", "HER2"),     # Important cancer genes
  control_1 = c(100, 150, 80, 200, 120),                     # Control replicate 1
  control_2 = c(110, 140, 85, 190, 125),                     # Control replicate 2
  treated_1 = c(200, 300, 90, 180, 240),                     # Treatment replicate 1
  treated_2 = c(190, 280, 95, 185, 230),                     # Treatment replicate 2
  chromosome = c("17", "17", "7", "12", "17"),               # Chromosome locations
  pathway = c("DNA repair", "Cell cycle", "Growth", "Signaling", "Growth")  # Biological pathways
)

# Display the dataset
print(gene_data)
```

## Understanding the Pipe Operator (`%>%`)

The pipe operator is a fundamental concept in modern R programming, especially for data analysis workflows.

### What is Piping?

The pipe operator (`%>%`) takes the output from one function and passes it as the first argument to the next function. Think of it like a pipeline where data flows through a series of operations.

Benefits:
1. Makes code more readable (left-to-right flow)
2. Reduces need for intermediate variables
3. Makes complex operations more manageable

### Traditional vs. Piped Syntax

```{r piping-example}
# Traditional nested syntax (hard to read)
mean(sqrt(abs(log(c(1:10)))))

# Same operation with pipes (easier to follow)
# Each step is clearly visible
c(1:10) %>%
  log() %>%      # First take the log
  abs() %>%      # Then the absolute value
  sqrt() %>%     # Then the square root
  mean()         # Finally calculate the mean

# Example with our gene data
# Traditional syntax (nested, harder to understand)
head(arrange(filter(gene_data, chromosome == "17"), desc(control_1)))

# Same operation with pipes (clear sequence of operations)
gene_data %>%
  filter(chromosome == "17") %>%         # First, keep only chromosome 17 genes
  arrange(desc(control_1)) %>%           # Then sort by control_1 values
  head()                                 # Finally, show first few rows
```

### Why Use Pipes?

1. **Readability**: Code flows naturally from left to right
2. **Maintainability**: Easy to add/remove/modify steps
3. **Reduced Nesting**: No more deeply nested function calls
4. **Code Organization**: Clear data transformation workflow

### Pro Tips for Using Pipes

1. Start with the data object
2. Add one operation per line
3. Indent lines after the first pipe
4. Use pipes for 2 or more operations
5. Use keyboard shortcut: Ctrl/Cmd + Shift + M

```{r pipe-tips}
# Example of a well-formatted analysis pipeline
result <- gene_data %>%
  filter(chromosome == "17") %>%                    # Filter for chromosome 17
  select(gene_id, control_1, treated_1) %>%        # Keep only needed columns
  mutate(fold_change = treated_1 / control_1) %>%  # Calculate fold change
  arrange(desc(fold_change))                       # Sort by fold change

print("Well-formatted pipe example result:")
print(result)
```

## Data Manipulation with dplyr

### 1. Selecting Columns (select)

The `select()` function helps you choose which columns to keep or remove. This is essential for focusing on relevant data:

```{r selecting}
# Select specific columns of interest
gene_data %>%
  select(gene_id, control_1, treated_1)  # Keep only gene IDs and one replicate each

# Select columns by pattern matching
# Useful when you have many similarly named columns
gene_data %>%
  select(starts_with("control"))  # Get all control samples

# Remove unwanted columns
# Useful for excluding replicate 2 data
gene_data %>%
  select(-ends_with("2"))  # Remove all replicate 2 columns
```

### 2. Filtering Rows (filter)

Use `filter()` to subset rows based on conditions. Essential for focusing on genes of interest:

```{r filtering}
# Filter genes on chromosome 17
# Common in cancer studies (many cancer genes on chr17)
gene_data %>%
  filter(chromosome == "17")

# Multiple conditions
# Find highly expressed genes on chr17
gene_data %>%
  filter(chromosome == "17",    # Must be on chromosome 17
         control_1 > 100)       # Must have high expression

# Complex conditions using %in%
# Find genes in specific pathways
gene_data %>%
  filter(pathway %in% c("Growth", "Signaling"))  # Growth or signaling genes
```

### 3. Creating New Columns (mutate)

`mutate()` adds new columns based on calculations. Essential for data analysis:

```{r mutating}
# Calculate summary statistics
gene_data %>%
  mutate(
    mean_control = (control_1 + control_2) / 2,     # Average control expression
    mean_treated = (treated_1 + treated_2) / 2,     # Average treated expression
    fold_change = mean_treated / mean_control       # Fold change calculation
  )

# Log transform expression values
# Common in RNA-seq analysis
gene_data %>%
  mutate(across(
    .cols = c(control_1, control_2, treated_1, treated_2),  # All expression columns
    .fns = log2,                                           # Apply log2 transform
    .names = "log2_{.col}"                                 # Name new columns
  ))
```

### 4. Summarizing Data (summarize/summarise)

`summarize()` creates summary statistics. Perfect for getting overview of your data:

```{r summarizing}
# Calculate mean expression per condition
gene_data %>%
  summarise(
    mean_control_1 = mean(control_1),     # Average control expression
    mean_treated_1 = mean(treated_1),     # Average treated expression
    n_genes = n()                         # Number of genes analyzed
  )

# Group by pathway and summarize
# Useful for pathway-level analysis
gene_data %>%
  group_by(pathway) %>%                   # Group genes by pathway
  summarise(
    n_genes = n(),                        # Genes per pathway
    mean_control = mean(control_1),       # Average control expression
    mean_treated = mean(treated_1)        # Average treated expression
  )
```

## Data Tidying with tidyr

### 1. Reshaping Data (pivot_longer and pivot_wider)

Data reshaping is crucial in bioinformatics, especially when preparing data for different analyses:

**Wide format**: Each sample in a separate column
- Good for: Spreadsheet viewing, manual data entry
- Example: RNA-seq count matrix

**Long format**: Each observation in a separate row
- Good for: Statistical tests, plotting
- Example: DESeq2 results

```{r reshaping}
# Convert from wide to long format
# This is often needed for visualization and statistical testing
gene_data_long <- gene_data %>%
  pivot_longer(
    cols = c(control_1, control_2, treated_1, treated_2),  # Expression columns
    names_to = "sample",                                   # Column for sample names
    values_to = "expression"                               # Column for expression values
  )

print("Long format (ideal for plotting and statistics):")
print(gene_data_long)

# Now ready for visualization with ggplot2
library(ggplot2)  # For plotting
```

## Advanced dplyr Operations

### Joining Data Frames

Often we need to combine information from multiple data frames. Let's create a second dataset with additional gene information:

```{r joining}
# Create a second dataset with gene annotations
gene_annotations <- tibble(
  gene_id = c("BRCA1", "TP53", "EGFR", "KRAS", "HER2", "PTEN"),
  full_name = c("Breast Cancer 1", "Tumor Protein 53", "Epidermal Growth Factor Receptor",
                "KRAS Proto-Oncogene", "Human Epidermal Growth Factor Receptor 2", 
                "Phosphatase and Tensin Homolog"),
  is_oncogene = c(FALSE, FALSE, TRUE, TRUE, TRUE, FALSE)
)

# Inner join - only keep genes present in both datasets
gene_data %>%
  inner_join(gene_annotations, by = "gene_id")

# Left join - keep all genes from gene_data
gene_data %>%
  left_join(gene_annotations, by = "gene_id")

# Full join - keep all genes from both datasets
gene_data %>%
  full_join(gene_annotations, by = "gene_id")
```

### Set Operations

dplyr provides functions for set operations between data frames:

```{r set-operations}
# Create two datasets with some overlapping genes
set1 <- gene_data %>% filter(chromosome == "17")
set2 <- gene_data %>% filter(pathway == "Growth")

# Union - combine unique rows
bind_rows(set1, set2) %>% distinct()

# Intersect - find common rows
inner_join(set1, set2, by = names(set1))

# Setdiff - find rows in set1 not in set2
anti_join(set1, set2, by = names(set1))
```

### Advanced Grouping Operations

```{r advanced-grouping}
# Group by multiple columns
gene_data %>%
  group_by(chromosome, pathway) %>%
  summarise(
    n_genes = n(),
    mean_expression = mean(control_1),
    .groups = "drop"
  )

# Grouped mutations
gene_data %>%
  group_by(chromosome) %>%
  mutate(
    rel_expression = control_1 / mean(control_1),
    rank = min_rank(desc(control_1))
  ) %>%
  ungroup()
```

## Creating Custom Functions

### Basic Function Creation

```{r custom-functions}
# Create a function to calculate fold change
calculate_fold_change <- function(treated, control) {
  if (any(control <= 0)) {
    warning("Control values should be positive")
    return(NA)
  }
  return(treated / control)
}

# Use the function with mutate
gene_data %>%
  mutate(
    fc_1 = calculate_fold_change(treated_1, control_1),
    fc_2 = calculate_fold_change(treated_2, control_2)
  )
```

### Functions with Multiple Arguments

```{r complex-functions}
# Function to filter and summarize expression data
analyze_expression <- function(data, chr = NULL, min_expression = 0) {
  # Start with the data
  result <- data
  
  # Filter by chromosome if specified
  if (!is.null(chr)) {
    result <- result %>% filter(chromosome == chr)
  }
  
  # Apply expression threshold
  result <- result %>%
    filter(control_1 > min_expression) %>%
    mutate(
      mean_control = (control_1 + control_2) / 2,
      mean_treated = (treated_1 + treated_2) / 2,
      fold_change = mean_treated / mean_control
    ) %>%
    select(gene_id, chromosome, pathway, mean_control, mean_treated, fold_change)
  
  return(result)
}

# Use the custom function
gene_data %>%
  analyze_expression(chr = "17", min_expression = 100)
```

### Creating Pipeline Functions

```{r pipeline-functions}
# Function to perform standard analysis pipeline
standard_analysis <- function(data, group_col) {
  group_col <- enquo(group_col)  # Quote the grouping column
  
  data %>%
    group_by(!!group_col) %>%
    summarise(
      n_genes = n(),
      mean_expression = mean(control_1),
      up_regulated = sum(treated_1 > control_1),
      down_regulated = sum(treated_1 < control_1),
      .groups = "drop"
    ) %>%
    mutate(
      pct_up = up_regulated / n_genes * 100,
      pct_down = down_regulated / n_genes * 100
    )
}

# Use the pipeline function
gene_data %>%
  standard_analysis(pathway)
```

## Practice Exercises

### Exercise 1: Basic Data Manipulation
Using the gene_data dataset:
1. Filter for genes on chromosome 17
2. Calculate the fold change between treated and control conditions
3. Select only gene_id and fold change columns

```{r exercise1_data_manipulation}
gene_data %>%
  filter(chromosome == "17") %>%
  mutate(
    mean_control = (control_1 + control_2) / 2,
    mean_treated = (treated_1 + treated_2) / 2,
    fold_change = mean_treated / mean_control
  ) %>%
  select(gene_id, fold_change)
```

### Exercise 2: Data Reshaping
1. Convert the expression data to long format
2. Calculate mean expression per condition
3. Create a summary of fold changes per pathway

```{r exercise2_data_reshaping}
# Convert to long format and summarize
gene_data %>%
  pivot_longer(
    cols = c(control_1, control_2, treated_1, treated_2),
    names_to = "sample",
    values_to = "expression"
  ) %>%
  group_by(gene_id, pathway) %>%
  summarise(
    mean_expression = mean(expression),
    .groups = "drop"
  )
```

### Exercise 3: Custom Functions
1. Create a function to normalize expression values
2. Apply the function to both control and treated samples
3. Calculate statistics on the normalized values

```{r exercise3_custom_functions}
# Create normalization function
normalize_expression <- function(x) {
  (x - mean(x)) / sd(x)
}

# Apply to data
gene_data %>%
  mutate(across(
    .cols = c(control_1, control_2, treated_1, treated_2),
    .fns = normalize_expression,
    .names = "norm_{.col}"
  )) %>%
  select(gene_id, starts_with("norm_"))
```

## Common Mistakes and Tips

1. **Chain Order Matters**
```{r chain-order}
# This works (filter then summarize)
gene_data %>%
  filter(chromosome == "17") %>%
  summarise(mean_exp = mean(control_1))

# This fails (summarize then filter)
# gene_data %>%
#   summarise(mean_exp = mean(control_1)) %>%
#   filter(chromosome == "17")  # Error: object 'chromosome' not found
```

2. **Grouping Persistence**
```{r grouping}
# Grouping remains until explicitly removed
grouped_data <- gene_data %>%
  group_by(pathway) %>%
  summarise(
    mean_control = mean(control_1),
    .groups = "drop"  # Explicitly remove grouping
  )
```

3. **Missing Values**
```{r missing-values}
# Create data with NA
gene_data_na <- gene_data %>%
  mutate(control_1 = ifelse(gene_id == "BRCA1", NA, control_1))

# Handle NA values
gene_data_na %>%
  summarise(
    mean_with_na = mean(control_1),
    mean_without_na = mean(control_1, na.rm = TRUE)
  )
```

## Next Steps

After mastering these advanced concepts, you can explore:
- Data visualization with ggplot2

# Create example visualization
ggplot(gene_data_long, 
       aes(x = gene_id, y = expression, fill = sample)) +  # Map variables to plot
  geom_bar(stat = "identity", position = "dodge") +        # Create grouped bars
  theme_minimal() +                                        # Use clean theme
  labs(title = "Gene Expression by Sample",                # Add informative labels
       x = "Gene",
       y = "Expression Level") +
  theme(axis.text.x = element_text(angle = 45, hjust = 1))  # Rotate gene names

### 2. Separating and Uniting Columns (tidyr)

Sometimes we need to work with complex identifiers or combine/split information:

```{r separate-unite}
# Create complex identifiers (common in bioinformatics)
gene_data_info <- gene_data %>%
  mutate(gene_info = paste(gene_id, chromosome, sep = "_chr"))  # Create combined ID

# Separate the gene_info column into components
# Useful when working with complex identifiers
gene_data_split <- gene_data_info %>%
  separate(gene_info,                    # Column to split
          into = c("gene", "chr_num"),   # New column names
          sep = "_chr")                  # Split at "_chr"

print("Data with separated columns:")
print(gene_data_split)

# Unite columns back together
# Useful for creating standardized identifiers
gene_data_united <- gene_data_split %>%
  unite("gene_chr",                      # Name of new column
        gene, chr_num,                   # Columns to combine
        sep = "_chr")                    # Separator to use

print("\nData with united columns:")
print(gene_data_united)
```

## Practice Exercises

### Exercise 1: Basic Data Manipulation

Try these operations with the gene expression data:

```{r exercise1, eval=FALSE}
# 1. Calculate log2 fold change for each gene
gene_data %>%
  mutate(
    mean_control = (control_1 + control_2) / 2,     # Average control
    mean_treated = (treated_1 + treated_2) / 2,     # Average treated
    log2_fold_change = log2(mean_treated / mean_control)  # Log2 fold change
  )

# 2. Find genes with absolute log2 fold change > 1
# This indicates a 2-fold or greater change
gene_data %>%
  mutate(
    mean_control = (control_1 + control_2) / 2,
    mean_treated = (treated_1 + treated_2) / 2,
    log2_fold_change = log2(mean_treated / mean_control)
  ) %>%
  filter(abs(log2_fold_change) > 1)  # |log2 FC| > 1 means >2-fold change
```

### Exercise 2: Grouped Analysis

Analyze expression patterns by pathway:

```{r exercise2, eval=FALSE}
# 1. Calculate average expression and standard deviation by pathway
gene_data %>%
  group_by(pathway) %>%
  summarise(
    mean_control = mean(control_1),           # Average control expression
    sd_control = sd(control_1),               # Variability in control
    mean_treated = mean(treated_1),           # Average treated expression
    sd_treated = sd(treated_1),               # Variability in treated
    n_genes = n()                             # Number of genes in pathway
  )

# 2. Find pathways with the highest average fold change
gene_data %>%
  group_by(pathway) %>%
  summarise(
    mean_fc = mean(treated_1 / control_1),    # Average fold change
    max_fc = max(treated_1 / control_1)       # Maximum fold change
  ) %>%
  arrange(desc(mean_fc))                      # Sort by average fold change
```

### Exercise 3: Data Reshaping

Practice reshaping data for different analyses:

```{r exercise3, eval=FALSE}
# 1. Convert to long format and calculate statistics
gene_data %>%
  pivot_longer(
    cols = c(control_1, control_2, treated_1, treated_2),
    names_to = c("condition", "replicate"),
    names_pattern = "(.+)_(.+)",              # Split names into condition and replicate
    values_to = "expression"
  ) %>%
  group_by(condition) %>%
  summarise(
    mean_expr = mean(expression),             # Average by condition
    sd_expr = sd(expression)                  # Variability by condition
  )

# 2. Create a correlation matrix between samples
gene_data %>%
  select(control_1, control_2, treated_1, treated_2) %>%  # Select expression columns
  cor()                                                   # Calculate correlations
```

## Tips for Effective Data Manipulation

1. **Start with Raw Data**
   - Keep original data unchanged
   - Document all transformations
   - Make analysis reproducible

2. **Use Appropriate Data Structures**
   - Wide format for viewing/manual entry
   - Long format for analysis/plotting
   - Consider memory usage for large datasets

3. **Check Your Work**
   - Verify transformations with small examples
   - Use summary statistics to catch errors
   - Plot data to visualize changes

4. **Optimize Performance**
   - Use appropriate data types
   - Minimize redundant calculations
   - Consider using data.table for large datasets

## Next Steps

After mastering these basics, explore:

1. **Advanced dplyr Functions**
   - `across()` for applying functions to multiple columns
   - `rowwise()` for row-wise operations
   - Window functions for ranking and comparisons

2. **Complex Data Transformations**
   - Nested data frames
   - List columns
   - Custom functions for repeated operations

3. **Integration with Other Tools**
   - Bioconductor packages
   - Statistical analysis packages
   - Advanced visualization tools

Remember: The key to effective data manipulation is choosing the right tools for your specific needs and ensuring your code is clear and reproducible.