If you find any typos, errors, or places where the text may be improved, please let us know by providing feedback either in the feedback survey (given during class) or by using GitHub.
On GitHub open an issue or submit a pull request by clicking the " Edit this page" link at the side of this page.
Many of you probably work largely and most consistently on your own, but as your move through your career (in academia or industry), you will need to and maybe also want to directly collaborate1 a lot more with others. Different types of collaboration (e.g. meetings, brainstorming, real-time co-writing) form the basis for almost all research-based work and probably most non-research-based work.
More direct collaboration on a project quickly becomes unmanageable when using traditional academic “workflows” (emailing around). That’s when you need to start using tools designed for collaboration, like Git. But Git is just the starting point. There are many many other things to consider for workflows and processes to effectively collaborate with others. This session is about making use of more automated ways of structuring data analysis projects to ease collaboration.
The overall objective for this session is to:
More specific objectives are to:
{renv} and {usethis} R packages to implement these dependency management concepts.{styler} and RStudio’s canonical markdown mode to programmatically check and apply style guides to your project files.Time: ~10 minutes.
When you work on a project (for your thesis or a manuscript), how exactly do you and your collaborators contribute to the project? Is it mostly verbal contributions? Do you use a shared folder that the files are on? How do you keep track of who’s changed what? Do you mostly work on your own and contributions are largely verbal or written feedback (like in a meeting or through an email)? If you work directly on a project, how do you coordinate things? Does one collaborator work on one section or analysis, so your files are separate? Do you ever have to go in and contribute your own code to theirs (and vice versa)?
Note: This first session is more conceptual and is heavier on the reading and explanation, but is important for the next sessions.
One of the first things to consider when working collaboratively on a data analysis project (and probably other types of projects too) is what software to use for your project. This starts out at the highest level: Are you using R or some other software for the analysis? Since this is an R course, we’re assuming the software will be R! 😜
The next consideration is which packages your project depends on to produce the results. When working collaboratively with others, and yourself several months in the future, you need some way of knowing how to easily and quickly install or update these package dependencies.
Part of this approach requires that you follow a “project-oriented” workflow when working on, well, your project. In order to know how to track your project’s package dependencies, you need to first know, what is a “project” and how do we work around it? Since the introduction course’s first session on the Management of R Projects, we’ve consistently taught and used this workflow-style. In fact, it is embedded into the use of the R Projects via the .Rproj files and in the use of the {here} package. So we already are following this approach from the start, which will make it easier to track package dependencies of our project.
{here} package, compared to the common approach of using setwd(). Artwork by @allison_horst.Let’s start with the AdvancedR3 project that uses the lipidomics. We have code in the data-raw/nmr-omics.R file that uses some packages. Let’s assume that your project will be more complex than this and that you will eventually need some collaborators to contribute who are experts in for instance metabolomics data processing and in statistical analysis of high-dimensional data. You know you will end up needing to use other packages. You also know that you all need some way of tracking which packages are used so that when others join and contribute to the project, they can as seamlessly as possible install or update the packages your data analysis project needs. There are a few ways of “tracking” package dependencies.
The simplest, but most primitive way is to always make sure to use library() at the top of each R script for each package that the R script uses.
Advantage:
Disadvantages:
It doesn’t track project-level dependencies very well, since multiple scripts probably use similar packages across them. Which means you can’t easily and quickly install or update all the packages your project uses, since you will probably have to go through each R script manually and install each package manually. You might have seen some scripts with code that looks like this at the top:
if (!require("packagename")) {
install.packages("packagename")
}This code checks if a package exists, if not, it installs it. But! This is not an optimal method to track packages because require() won’t load the package if it doesn’t find it. Which means you would have to re-run the script probably a few times. Plus, sometimes you may need to restart the R session after installing a package in order for R to detect it properly.
It doesn’t track the versions of the packages your project depends on, so if a package gets updated and it breaks something, you might not be able to figure out how to quickly fix that issue, especially for those deadline crunches.
The most common form, at least based on R packages and projects found on GitHub, makes use of the DESCRIPTION file and usethis::use_package() to track if a package is used for a project or not. We covered this style of dependency in the intermediate course. We will also use this approach during this course, but expand a lot more on it.
Advantages:
Relatively easy to conceptually understand, since you can directly view the packages your project needs by opening the DESCRIPTION file and looking at the contents.
Because it is widely used, there are many processes already built around making use of tracking dependencies this way. For instance, you need to track package dependencies when creating R packages.
Installing packages is as easy as opening the project and running remotes::install_deps() in the Console, which will install all the packages listed in the DESCRIPTION file.
Adding packages that you need is as easy as writing usethis::use_package("packagename") in the Console.
Disadvantages:
Like the previous method, it doesn’t easily keep track of the versions of the packages you are using.
Your project might still rely on a package that is installed on your computer and that influences your project, but that might not be obvious as a dependency or that you forgot to include.
Before continuing to the exercise, we need to make sure to add and comment all the files from the project into the Git history. Open the Git interface by either typing Ctrl-Shift-M or by going to the Git pane and clicking the “Commit” button.
Time: ~10 minutes.
Since the DESCRIPTION file will be used later on for the more formal dependency management, let’s get it updated with the packages we are using in the data-raw/nmr-omics.R script. Open that file and complete these tasks:
library() and ::. It can help to use “Find / Replace Text” feature in RStudio for all the :: used in the script. Use the Command Palette (Ctrl-Shift-P, then “find replace”) to quickly select this option, or Edit -> Find and replace (Ctrl-Shift-J).?usethis::use_package to review how to use this function.usethis::use_package() for each package you find in data-raw/nmr-omics.R (from 1. above).Ctrl-Shift-M or go to the Git Pane and click the “Commit” button). What has been changed? Commit those changes to the Git history.
.Rbuildignore. We don’t really need this for this course, so let’s do usethis::use_git_ignore(".Rbuildignore").usethis::use_package("stringr")
usethis::use_package("readxl")
usethis::use_package("dplyr")
usethis::use_package("tidyr")
usethis::use_package("snakecase")
usethis::use_package("here")
usethis::use_package("fs")
usethis::use_package("usethis")Make sure that everyone has added the right packages, since it can be easy to miss some of the packages referenced using ::.
While the approach of managing package dependencies through the DESCRIPTION file is quite powerful, it has the major disadvantage of not keeping track of the version of each of your packages. So instead we are going to use a package dedicated to handling project dependencies, called {renv}.
{renv} does for R Projects, by making them more self-contained. Artwork by @allison_horst.{renv} is a package that manages package dependencies in a project by, in simple terms, creating a project-specific R “library”. You might think of library() when you hear R library, and you aren’t completely wrong. When you call, for instance, library("usethis"), R looks for the package {usethis} in your computer’s “library” of R packages. This library can be found by running this function:
.libPaths()[1] "/home/runner/work/r-cubed-advanced/r-cubed-advanced/renv/library/R-4.3/x86_64-pc-linux-gnu"
[2] "/home/runner/.cache/R/renv/sandbox/R-4.3/x86_64-pc-linux-gnu/5cd49154" Those file paths are where all R packages are installed to. If there is more than one path, R checks the first before continuing to the next. {renv} instead creates a file path for R packages to be installed within the project. This isn’t completely what it does, since there are a lot of very technical details to what {renv} does internally, but this is the basic concept.
In the end, this makes your project relatively self-contained in its package dependencies.
Advantages:
Installing all packages necessary for your project is as easy as running renv::restore(). {renv} even gives you helpful tips and instructions when things go wrong or if something is missing.
Every package, including the packages that your packages depend on, have their version tracked. So if a package gets updated on CRAN, it doesn’t affect you until you choose, using renv::update(). If a package update breaks your code and you have a deadline, you can easily go back to the older versions of the packages.
Because your project is now self-contained with its own R library, it becomes very obvious (through errors) when you might be missing some other dependency because your code wouldn’t run until you install or fix that package dependency.
From a reproducibility point of view, as long as your project is tracked by {renv}, it’s easier to independently have your data analysis be reproduced and verified.
Disadvantages:
It takes a fair amount of learning to conceptually understand what is going on.
When there are issues that come up, it can be difficult to figure them out.
Because all packages, including those packages that your packages depend on, are installed within your project, installation times can sometimes be a bit long.
It’s sometimes very annoying to debug these “virtual environments”. But thankfully you can turn it off with renv::deactivate()! It still is worth it to start considering and accounting for how your dependencies might influence your project results and collaboration.
Let’s start using {renv} in our AdvancedR3 project. In the Console, type out:
renv::init(bare = TRUE)The function renv::init() initializes the project to begin being managed by {renv}. By using the argument bare = TRUE we are telling {renv} to not search for dependencies in the project, since we want to do that ourselves.
When initilizing a {renv} instance using bare = TRUE we create a new .libPath() location that does not contain any of the packages we previously defined as project dependencies. You can notice this if you attempt to type usethis::, you’ll notice that nothing comes up. Therefore, before being able to continue our workflow we need to tell {renv} to install the packages we have already defined as dependencies. This can be done with renv::install():
renv::install()After doing this we should have several files in our project folder:
.
├── .Rprofile
├── renv
│ ├── .gitignore
│ ├── activate.R
│ ├── library
│ ├── settings.dcf
│ ├── settings.json
│ └── staging
└── renv.lock.Rprofile: This is the file that {renv} uses to build up its machinery. With an .Rprofile within the project folder, we can create a “project environment”. By having this project environment that is (mostly) self-contained, it allows us to be a bit closer to having a fully reproducible analysis and it makes it easier to collaborate, since we all than share the same project setup.
renv.lock: (Might not be created yet) This contains all the information about the packages your project depends on, including where it was installed from (CRAN or GitHub for example), what the version number is, and more. This is like a supercharged version of the DESCRIPTION file.
renv/: This folder contains several other files that make up the machinery of {renv}. For instance, there is the library/ folder that contains all the R packages necessary for the project. Then there are the activate.R script and settings.dcf file that both work to manage the dependencies, like installing, updating, and removing.
%%{init:{'theme':'forest', 'flowchart':{'nodeSpacing': 40, 'rankSpacing': 20}}}%%
graph LR
user["far:fa-folder User/"] --- root_docs
user --- root_desktop
user --- .Rprofile
subgraph global[Global fab:fa-r-project environment]
root_docs["far:fa-folder Documents/"] --- r["far:fa-folder R/ (global fab:fa-r-project library)"]
root_desktop["far:fa-folder Desktop/"] --- renv["far:fa-folder AdvancedR3/"]
root_docs --- 3["far:fa-folder other-projects/"]
.Rprofile
subgraph renv_env[Project fab:fa-r-project environment]
renv --- renv_folder["far:fa-folder renv/<br/>(fab:fa-r-project library)"]
renv --- proj_prof[.Rprofile]
end
end
linkStyle 0,1,2,3,4,5,6,7 stroke-width:1px;
classDef folderEnv fill:transparent,stroke-dasharray:5;
class renv_env,global folderEnv
Before we continue, let’s commit the new files to the Git history.
The general workflow for using {renv} while working on your project is described in more detail on the Introduction to renv webpage. However, unlike the general workflow, we also want to continue using the DESCRIPTION file. That’s because a lot of tools and workflows exist that make use of it, so we want to remain compatible with them.
As we work on the project and realize we need to use a specific package, we would normally use install.packages() and then add library() to the script or R Markdown file. Later on, we’d eventually run renv::snapshot() to update the renv.lock file with the new packages we’ve installed. renv::snapshot() as well as renv::init() usually rely on “implicit” dependencies, meaning {renv} will search throughout the project for any packages used and add them to the renv.lock file.
However, in science, we want to be more explicit rather than implicit. The way {renv} explicitly adds to the renv.lock file is by only scanning the DESCRIPTION file. In order for {renv} to always do this, we need to set an option for it. This option needs to be added to the project’s .Rprofile file.
We can quickly open this with:
usethis::edit_r_profile("project")Next, at the top of the file, add this code:
options(
renv.settings.snapshot.type = "explicit",
renv.config.auto.snapshot = TRUE
)These two options make it so that whenever you add a package with usethis::use_package() or install.packages(), {renv} will always run renv::snapshot() and the snapshot explicitly only look at the DESCRIPTION file. Let’s restart the R session so that the .Rprofile changes get activated.
Now our {renv} workflow will largely be automated for us, as long as we do usethis::use_package(). If we ever ever return to a project or collaborate on a project that uses {renv}, we can install all the necessary packages with:
renv::restore()And if we need to update packages, we use:
renv::update()Sometimes, working with {renv} can get annoying and you just need to finish working on a task. If that’s the case, you can always do renv::deactivate() to stop using {renv} and renv::activate() to reactivate it.
Let’s commit the changes made to the Git history.
It’s surprising how many issues can come up, from a reproducibility perspective, when it comes to managing package dependencies. You think something works well on your computer, but when you create a “virtual environment” like you do when using {renv}, you realize it might not work as well on other computers.
When this happens, there are several functions you can use to help debug the situation.
renv::diagnostics()List a lot of diagnostic information to look over. Sometimes its too much, but can help figure out what’s going on.
renv::clean()Installing packages can sometimes lead to issues in the files of the installed packages themselves or even left over temporary files. This function tries to clean up these issues for you. It can also clean up any unused packages
renv::repair()Because of the way {renv} works, the connection to where an installed package is actually found can get broken. So this function tries to fix that and reinstall these broken packages.
renv::rebuild()The last resort, use this to reinstall everything from scratch.
When you work on a research project that involves data analysis, you likely use packages in two different ways:
{dplyr} or {tidyr} would be build dependencies, since you use them for processing data.{renv} would be considered a workflow dependency.A good way to determine if a package is a build dependency for your project is by seeing if you write and use functions from the package within an R script that does something to the data or analysis. If you only ever use functions from the package in the Console, than it is likely a workflow dependency.
The way you add these packages is different depending on the type it is. For build dependencies, we use the function we’ve already used before: usethis::use_package("packagename"). For workflow dependencies, it’s the same function, but with a small difference. BUT! Before we cover it, let’s add a setting to our .Rprofile to make our life a bit easier. We will be using {usethis} functions many times throughout the course, so a simple quality-of-life fix is to make it so we don’t always have to do usethis::. Thankfully, there is a function that can help. Create and open the project .Rprofile with usethis::edit_r_profile().
usethis::edit_r_profile("project")Then copy and paste this code into the .Rprofile:
if (interactive()) {
suppressMessages(require(usethis))
}Let’s restart R (Ctrl-Shift-P then type “restart r”) before using use_package() to add {renv} as a workflow dependency.
usethis::use_package("renv", "suggests")Open the Git interface and see that under Suggests: in the DESCRIPTION file is renv. Let’s commit these changes, as well as the changes to .Rprofile.
For the information block below, mention it to the learners but you don’t need to go over it. Especially mention the second part of the tip.
When you come back to a project after a few months or if you start collaborating on a project, usually renv::restore() will be enough to install all types of dependencies. Sometimes though, workflow dependencies might not get installed. As long as they are tracked in the DESCRIPTION file though, you can force installation of them with:
renv::install()Depending on your operating system (Windows, MacOS, or Linux), using use_package() might actually install the package again, even though you already have it. In that case, you can use renv::install("packagename") first, before than using use_package().
When you’re working on your own and not needing to worry about anyone seeing your code, there’s a natural temptation to write your code like you might write notes to yourself… scribbled and scrawled down quickly. Even when you are working with others or try to write more readable code, it is very easy for code to “drift” towards being poorly formatted, especially when deadlines are close.
That’s when “linters” or “stylers” (types of “static code analysis” tools) become very useful. They will scan your code for common mistakes or syntax problems and either list them out for you to fix or fix them for you automatically. Linters are great when you are collaborating on a project with collaborators who are not as experienced in writing code or who only occasionally contribute so don’t know the workflow culture of your project. In this way, you might want to have automatic linting/styling checks that are independent of you having to run them yourself. This is where the {styler} package comes in!
Since we will use it for the project as a workflow dependency, let’s add it to the DESCRIPTION file.
use_package("styler", "suggests")Like {lintr}, there are only a few functions in {styler} that we need to use. The first is to style a single file, so let’s try it in the Console:
styler::style_file("data-raw/nmr-omics.R")Styling 1 files:
data-raw/nmr-omics.R ✔
────────────────────────────────────────
Status Count Legend
✔ 1 File unchanged.
ℹ 0 File changed.
✖ 0 Styling threw an error.
────────────────────────────────────────Since we already have the file all nice and tidy, it doesn’t do anything. But, this function is a bit much to type. We could also do it on all files in the project with:
styler::style_dir()You will probably be asked to install something, click “Yes”.
But, like the linting we did above, we only need to style all the files occasionally. Usually we would need to only style the file we are actually working. We can do that through the Command Palette (Ctrl-Shift-P) and typing “style file”, which should show the “Style active file” option. You’ll try it out in the next exercise.
The thing to note, though, is that {styler} isn’t perfect, so by using it together with {lintr} and sometimes manually running reformatting (Ctrl-Shift-A or with the Command Palette Ctrl-Shift-P then “reformat”) you will be able to fix most syntax issues.
Mention the callout block below, but don’t go into it at all.
You might be used to using 4 spaces for tabs instead of 2. The tidyverse style uses 2, so the default option in {styler}
options(
styler.addins_style_transformer = "styler::tidyverse_style(indent_by = 4)"
)Time: ~15 minutes.
Open your doc/lesson.qmd file, create a new code chunk (Ctrl-Shift-I or Ctrl-Shift-P followed by typing “chunk”) at the end of the file. Copy and paste the below code:
library(dplyr)
lipidomics <-
read_csv( here:: here("data/lipidomics.csv" )
)
lipidomics%>%select(gender , age)%>%filter(age>=25)Open the Git interface and commit the changes to the Git history. Then push the changes to GitHub. Go to the Actions tab and see what happens. It might take a few minutes to finish.
Go back to doc/lesson.qmd file in RStudio and run lintr::lint() with the Command Palette (Ctrl-Shift-P then type “lint file”). What happens? Don’t do anything just yet, simply see what things were identified as potential issues.
Run styler::style_file() using the Command Palette (Ctrl-Shift-P then type “style file”) while inside the doc/lesson.qmd file. What happens? Open the Git interface to see the changes made. Commit the changes and push to GitHub. Go into the Actions tab to see what is different now.
Re-run the lintr::lint() with the Command Palette. What’s changed since you did it last?
For multi-person collaborative projects, having some type of code styling and checker can really help with standardizing how the code looks, which ultimately will make it easier to read each other’s code contributions.
But what about for Markdown files? While there isn’t a package or function (yet) that styles the Markdown files, RStudio does have an option in their Tools to format Markdown into a “canonical form”. The reason for this option is because they added a “visual editor mode” to writing R Markdown files (which is great if you are more comfortable with apps like Word). Let’s test out this option. First, let’s make sure everything has been committed to the Git history.
Use this option only if you have your project under Git version control, since it will directly modify and overwrite the contents of the entire file.
There are two ways of doing this:
Tools -> Project Options -> R Markdown and changing the options “Automatic text wrapping” to “column” (with the default 72 width value) and “Write canonical visual mode markdown” to “true”._quarto.yml file (we will cover this in Chapter 9) or at the file-level in the YAML header.For right now, we will do the file-level YAML settings. Open the doc/lesson.qmd file and go to the top of the file. Right below the last ---, create a new line above it and paste this code in:
editor_options:
markdown:
wrap: 72
canonical: trueNow, when you save your file, RStudio should automatically reformat the Markdown into a standardized format. If you want to switch to using the Visual Mode, use Ctrl-Shift-F4 or the “Visual” button at the top of the Source Pane beside the bolding and italicizing buttons.
The instructors won’t be using the Visual Mode during the course, however you are welcome to. We will be using the “canonical” markdown mode though.
Let’s test it out. While in the doc/lesson.qmd file, go to the bottom of the file and type out:
## This is poorly formatted
- Definitely should have an empty space above this list.
- This isn't a list, why not?Save the file. What happens? Lists in Markdown must have an empty space above them to work properly. With this canonical mode on, we can get feedback right away that it isn’t right. We fix it by adding that empty space.
## This is poorly formatted
- Definitely should have an empty space above this list.
- This isn't a list, why not?Since this mode is on automatically, as we work through the sessions, we’ll get lots of experience using it.
Time: ~2 minutes.
There are many options inside the Global Options in RStudio that can help you work better and faster. There are a few that will help a lot, especially in this course and with the workflows we are showing:
Tools -> Global Options -> Code -> Saving.
Time: ~10 minutes.
Open up the README.md file and copy and paste these YAML metadata to the top:
---
editor_options:
markdown:
wrap: 72
canonical: true
---Then, start completing the TODO items. Save often and watch as the Markdown gets reformatted. After you are done, commit the changes you made to the Git history. Then delete both TODO.md, followed by committing these deletions in the Git history. Click the “Push” button to push the changes to GitHub.
Use this code to help synchronize with what the learners are doing. Copy and paste this code into the RStudio Console of the AdvancedR3 project.
editor_yaml <- "
---
editor_options:
markdown:
wrap: 72
canonical: true
---
"
readLines("README.md") |>
append(editor_yaml, after = 1) |>
writeLines("README.md")
gert::git_add("README.md")
gert::git_commit("Add editor options to README file.")
fs::file_delete("TODO.md")
gert::git_commit("Don't need this file anymore.")Time: ~25 minutes.
Since we will eventually connect our project Git repository to GitHub to display a website, we’ll do that right now.
Let’s complete these tasks to connect to GitHub.
First, commit the latest changes to the Git history.
If you haven’t yet, please create a GitHub account.
Add the {gitcreds} package as a workflow dependency. You’ll need it for the next item
use_package("gitcreds", "suggests")Read through and complete the tasks in the Connect to GitHub Guide.
Check your GitHub to make sure the project repository has been uploaded to it.
We’re at the end of the session and now need to fill in the survey. There is a function in r3, but r3 isn’t installed in our project library. However, because r3 has a few GitHub packages as dependencies, we need to tell {renv} to authenticate us to GitHub so we can install as much as we want (GitHub only allows a few unauthenticated installed, as a security feature against hackers and cyber attacks). We need to create a variable called GITHUB_TOKEN by running this function from {credentials}.
credentials::set_github_pat(TRUE)Then we can install the r3 package into our project with renv::install(). Because the r3 package is on GitLab, we need to tell {renv} about its location by adding gitlab:: followed by the “user” (which is rostools) and than the repository (r3).
renv::install("gitlab::rostools/r3")Now we can open the feedback survey:
r3::open_feedback_survey_advanced(){renv} and combine it with options() to automatically make snapshots so you can use the use_package() function.renv::restore() or renv::install().{styler}. Combine with the Command Palette (Ctrl-Shift-P) to quickly run their functions on code you are actively working on.Collaborate here meaning directly contributing to a shared project, rather than discussed or planning based collaborations (and definitely not emailing-files-around collaboration).↩︎