If you find any typos, errors, or places where the text may be improved, please let us know by providing feedback either in the feedback survey (given during class) or by using GitHub.
On GitHub open an issue or submit a pull request by clicking the " Edit this page" link at the side of this page.
Many of you probably work largely and most consistently on your own, but as your move through your career (in academia or industry), you will need to and maybe also want to directly collaborate1 a lot more with others. Different types of collaboration (e.g. meetings, brainstorming, real-time co-writing) form the basis for almost all research-based work and probably most non-research-based work.
1 Collaborate here meaning directly contributing to a shared project, rather than discussed or planning based collaborations (and definitely not emailing-files-around collaboration).
More direct collaboration on a project quickly becomes unmanageable when using traditional academic “workflows” (emailing around). That’s when you need to start using tools designed for collaboration, like Git. But Git is just the starting point. There are many many other things to consider for workflows and processes to effectively collaborate with others. This session is about making use of more automated ways of structuring data analysis projects to ease collaboration.
The overall objective for this session is to:
More specific objectives are to:
Time: ~10 minutes.
When you work on a project (for your thesis or a manuscript), how exactly do you and your collaborators contribute to the project? Is it mostly verbal contributions? Do you use a shared folder that the files are on? How do you keep track of who’s changed what? Do you mostly work on your own and contributions are largely verbal or written feedback (like in a meeting or through an email)? If you work directly on a project, how do you coordinate things? Does one collaborator work on one section or analysis, so your files are separate? Do you ever have to go in and contribute your own code to theirs (and vice versa)?
Note: This first session is more conceptual and is heavier on the reading and explanation, but is important for the next sessions.
One of the first things to consider when working collaboratively on a data analysis project (and probably other types of projects too) is what software to use for your project. This starts out at the highest level: Are you using R or some other software for the analysis? Since this is an R course, we’re assuming the software will be R! 😜
The next consideration is which packages your project depends on to produce the results. When working collaboratively with others, and yourself several months in the future, you need some way of knowing how to easily and quickly install or update these package dependencies.
Part of this approach requires that you follow a “project-oriented” workflow when working on, well, your project. In order to know how to track your project’s package dependencies, you need to first know, what is a “project” and how do we work around it? Since the introduction course’s first session on the Management of R Projects, we’ve consistently taught and used this workflow-style. In fact, it is embedded into the use of the R Projects via the .Rproj files and in the use of the here package. So we already are following this approach from the start, which will make it easier to track package dependencies of our project.
setwd(). Artwork by @allison_horst.Let’s start with the AdvancedR3 project that uses the lipidomics. We have code in the data-raw/nmr-omics.R file that uses some packages. Let’s assume that your project will be more complex than this and that you will eventually need some collaborators to contribute who are experts in for instance metabolomics data processing and in statistical analysis of high-dimensional data. You know you will end up needing to use other packages. You also know that you all need some way of tracking which packages are used so that when others join and contribute to the project, they can as seamlessly as possible install or update the packages your data analysis project needs. There are a few ways of “tracking” package dependencies.
Before continuing to the exercise, we need to make sure to add and comment all the files from the project into the Git history. Open the Git interface by either typing Ctrl-Shift-M or by going to the Git pane and clicking the “Commit” button.
Time: ~10 minutes.
Since the DESCRIPTION file will be used later on for the more formal dependency management, let’s get it updated with the packages we are using in the data-raw/nmr-omics.R script. Open that file and complete these tasks:
library() and ::. It can help to use “Find / Replace Text” feature in RStudio for all the :: used in the script. Use the Command Palette (Ctrl-Shift-P, then “find replace”) to quickly select this option, or Edit -> Find and replace (Ctrl-Shift-J).?usethis::use_package to review how to use this function.usethis::use_package() for each package you find in data-raw/nmr-omics.R (from 1. above).Ctrl-Shift-M or go to the Git Pane and click the “Commit” button). What has been changed? Commit those changes to the Git history.
.Rbuildignore. We don’t really need this for this course, so let’s do usethis::use_git_ignore(".Rbuildignore").usethis::use_package("stringr")
usethis::use_package("readxl")
usethis::use_package("dplyr")
usethis::use_package("tidyr")
usethis::use_package("snakecase")
usethis::use_package("here")
usethis::use_package("fs")
usethis::use_package("usethis")While the approach of managing package dependencies through the DESCRIPTION file is quite powerful, it has the major disadvantage of not keeping track of the version of each of your packages. So instead we are going to use a package dedicated to handling project dependencies, called renv.
renv is a package that manages package dependencies in a project by, in simple terms, creating a project-specific R “library”. You might think of library() when you hear R library, and you aren’t completely wrong. When you call, for instance, library("usethis"), R looks for the package usethis in your computer’s “library” of R packages. This library can be found by running this function:
[1] "/home/luke/.cache/R/renv/library/r-cubed-advanced-611f9650/R-4.2/x86_64-pc-linux-gnu"
[2] "/home/luke/Documents/teaching/r-cubed-advanced/renv/sandbox/R-4.2/x86_64-pc-linux-gnu/9a444a72"
[3] "/usr/local/lib/R/site-library"
[4] "/usr/lib/R/site-library"
[5] "/usr/lib/R/library" Those file paths are where all R packages are installed to. If there is more than one path, R checks the first before continuing to the next. renv instead creates a file path for R packages to be installed within the project. This isn’t completely what it does, since there are a lot of very technical details to what renv does internally, but this is the basic concept.
In the end, this makes your project relatively self-contained in its package dependencies.
Advantages:
Installing all packages necessary for your project is as easy as running renv::restore(). renv even gives you helpful tips and instructions when things go wrong or if something is missing.
Every package, including the packages that your packages depend on, have their version tracked. So if a package gets updated on CRAN, it doesn’t affect you until you choose, using renv::update(). If a package update breaks your code and you have a deadline, you can easily go back to the older versions of the packages.
Because your project is now self-contained with its own R library, it becomes very obvious (through errors) when you might be missing some other dependency because your code wouldn’t run until you install or fix that package dependency.
From a reproducibility point of view, as long as your project is tracked by renv, it’s easier to independently have your data analysis be reproduced and verified.
Disadvantages:
It takes a fair amount of learning to conceptually understand what is going on.
When there are issues that come up, it can be difficult to figure them out.
Because all packages, including those packages that your packages depend on, are installed within your project, installation times can sometimes be a bit long.
It’s sometimes very annoying to debug these “virtual environments”. But thankfully you can turn it off with renv::deactivate()! It still is worth it to start considering and accounting for how your dependencies might influence your project results and collaboration.
Let’s start using renv in our AdvancedR3 project. In the Console, type out:
renv::init(bare = TRUE)The function renv::init() initializes the project to begin being managed by renv. By using the argument bare = TRUE we are telling renv to not search for dependencies in the project, since we want to do that ourselves.
When initilizing a renv instance using bare = TRUE we create a new .libPath() location that does not contain any of the packages we previously defined as project dependencies. You can notice this if you attempt to type usethis::, you’ll notice that nothing comes up. Therefore, before being able to continue our workflow we need to tell renv to install the packages we have already defined as dependencies. This can be done with renv::install():
renv::install()After doing this we should have several files in our project folder:
.
├── .Rprofile
├── renv
│ ├── .gitignore
│ ├── activate.R
│ ├── sandbox
│ ├── settings.dcf
│ └── staging
└── renv.lock.Rprofile: This is the file that renv uses to build up its machinery. With an .Rprofile within the project folder, we can create a “project environment”. By having this project environment that is (mostly) self-contained, it allows us to be a bit closer to having a fully reproducible analysis and it makes it easier to collaborate, since we all than share the same project setup.
renv.lock: (Might not be created yet) This contains all the information about the packages your project depends on, including where it was installed from (CRAN or GitHub for example), what the version number is, and more. This is like a supercharged version of the DESCRIPTION file.
renv/: This folder contains several other files that make up the machinery of renv. For instance, there is the library/ folder that contains all the R packages necessary for the project. Then there are the activate.R script and settings.dcf file that both work to manage the dependencies, like installing, updating, and removing.
%%{init:{'theme':'forest', 'flowchart':{'nodeSpacing': 40, 'rankSpacing': 20}}}%%
graph LR
user["far:fa-folder User/"] --- root_docs
user --- root_desktop
user --- .Rprofile
subgraph global[Global fab:fa-r-project environment]
root_docs["far:fa-folder Documents/"] --- r["far:fa-folder R/ (global fab:fa-r-project library)"]
root_desktop["far:fa-folder Desktop/"] --- renv["far:fa-folder AdvancedR3/"]
root_docs --- 3["far:fa-folder other-projects/"]
.Rprofile
subgraph renv_env[Project fab:fa-r-project environment]
renv --- renv_folder["far:fa-folder renv/<br/>(fab:fa-r-project library)"]
renv --- proj_prof[.Rprofile]
end
end
linkStyle 0,1,2,3,4,5,6,7 stroke-width:1px;
classDef folderEnv fill:transparent,stroke-dasharray:5;
class renv_env,global folderEnv
Before we continue, let’s commit the new files to the Git history.
The general workflow for using renv while working on your project is described in more detail on the Introduction to renv webpage. However, unlike the general workflow, we also want to continue using the DESCRIPTION file. That’s because a lot of tools and workflows exist that make use of it, so we want to remain compatible with them.
As we work on the project and realize we need to use a specific package, we would normally use install.packages() and then add library() to the script or R Markdown file. Later on, we’d eventually run renv::snapshot() to update the renv.lock file with the new packages we’ve installed. renv::snapshot() as well as renv::init() usually rely on “implicit” dependencies, meaning renv will search throughout the project for any packages used and add them to the renv.lock file.
However, in science, we want to be more explicit rather than implicit. The way renv explicitly adds to the renv.lock file is by only scanning the DESCRIPTION file. In order for renv to always do this, we need to set an option for it. This option needs to be added to the project’s .Rprofile file.
We can quickly open this with:
usethis::edit_r_profile("project")Next, at the top of the file, add this code:
options(
renv.settings.snapshot.type = "explicit",
renv.config.auto.snapshot = TRUE
)These two options make it so that whenever you add a package with usethis::use_package() or install.packages(), renv will always run renv::snapshot() and the snapshot explicitly only look at the DESCRIPTION file. Let’s restart the R session so that the .Rprofile changes get activated.
Now our renv workflow will largely be automated for us, as long as we do usethis::use_package(). If we ever ever return to a project or collaborate on a project that uses renv, we can install all the necessary packages with:
renv::restore()And if we need to update packages, we use:
renv::update()Sometimes, working with renv can get annoying and you just need to finish working on a task. If that’s the case, you can always do renv::deactivate() to stop using renv and renv::activate() to reactivate it.
Let’s commit the changes made to the Git history.
When you work on a research project that involves data analysis, you likely use packages in two different ways:
A good way to determine if a package is a build dependency for your project is by seeing if you write and use functions from the package within an R script that does something to the data or analysis. If you only ever use functions from the package in the Console, than it is likely a workflow dependency.
The way you add these packages is different depending on the type it is. For build dependencies, we use the function we’ve already used before: usethis::use_package("packagename"). For workflow dependencies, it’s the same function, but with a small difference. BUT! Before we cover it, let’s add a setting to our .Rprofile to make our life a bit easier. We will be using usethis functions many times throughout the course, so a simple quality-of-life fix is to make it so we don’t always have to do usethis::. Thankfully, there is a function that can help. Create and open the project .Rprofile with usethis::edit_r_profile().
usethis::edit_r_profile("project")Then copy and paste this code into the .Rprofile:
if (interactive()) {
suppressMessages(require(usethis))
}Let’s restart R (Ctrl-Shift-P then type “restart r”) before using use_package() to add renv as a workflow dependency.
usethis::use_package("renv", "suggests")Open the Git interface and see that under Suggests: in the DESCRIPTION file is renv. Let’s commit these changes, as well as the changes to .Rprofile.
When you come back to a project after a few months or if you start collaborating on a project, usually renv::restore() will be enough to install all types of dependencies. Sometimes though, workflow dependencies might not get installed. As long as they are tracked in the DESCRIPTION file though, you can force installation of them with:
renv::install()Depending on your operating system (Windows, MacOS, or Linux), using use_package() might actually install the package again, even though you already have it. In that case, you can use renv::install("packagename") first, before than using use_package().
When you’re working on your own and not needing to worry about anyone seeing your code, there’s a natural temptation to write your code like you might write notes to yourself… scribbled and scrawled down quickly. Even when you are working with others or try to write more readable code, it still is very easy for “lint”2 to accumulate.
2 You know, the fluff you find in your pockets 😝
That’s when “linters” (a type of “static code analysis” tool) become very useful. A linter will scan your code for common mistakes or syntax problems and list them out for you to fix. They don’t directly fix them for you, though we will show how to do that later in this session. Linters are great when you are collaborating on a project with collaborators who are not as experienced in writing code or who only occasionally contribute so don’t know the workflow culture of your project. In this way, you might want to have automatic linting checks (or styling) that are independent of you having to run them yourself.
The main package for doing that is the lintr package. We’ll get a chance to try out using renv since we need to install it to use it in our project.
use_package("lintr", "suggests")Right now we don’t have much code to “lint”, but as we work through the course, we will make use of lintr before committing to Git. So, how do we use lintr? It’s actually pretty easy, since there are only a few functions that do the linting, even though there are dozens of customizations you can make to lintr. The basic is to lint a single file, don’t run this though:
lintr::lint("R/functions.R")A more encompassing function checks all files in the project (also, don’t run it yet, we’ll do that when we have more code in the project):
lintr::lint_dir()This might be useful if you want to go over everything to check things. But, if you are working on a single file and want to do some linting of that, the better solution is to use the Command Palette (Ctrl-Shift-P) and type “lint”. The first option is to lint the file you are working on. Let’s open up the data-raw/nmr-omics.R script and try it out. There should be at least one note that says:
Lines should not be more than 80 characters.Keep in mind, that not all notes need to be changed. A linter is meant to be used as a guideline, not as a rule. It helps you to follow best practices and have some consistency between collaborators, but not to stress about making sure all notes are addressed. We can suppress lint warnings by adding a # nolint comment.
Since we will eventually connect our project Git repository to GitHub to display a website, we can actually make use of a really powerful service called GitHub Actions to also run lintr there for us. But before we can do that, we need to connect to GitHub. So before doing the next exercise, let’s all commit the changes to the Git history.
Time: ~25 minutes.
Let’s complete these tasks to connect to GitHub.
Now that your AdvancedR3 project is connected and up on GitHub, we can make use of the GitHub Actions. The RStudio team has a repository that lists all R “Actions” that you can use to help streamline checks on your project. Specifically, we want to add the one for lintr, which we can add with:
use_github_action("lint-project")This creates a file in .github/workflows/lint-project.yaml. In order for GitHub to start using it, we need to commit it to the Git history and than push it to GitHub.
Once you’ve done that, switch to your project on GitHub and click the “Actions” tab at the top. Then click the “lint-project” link on the sidebar. This lists the checks that lintr has done on your project in an independent environment.
An advantage of lintr is that it does not make the changes it detects for you, but the disadvantage of lintr is that it does not make the changes for you. Often checks that require manual action are the best way of finding and fixing things. But sometimes, especially with very common syntax formatting mistakes, like not using spacing properly, it is so much easier to have the computer fix it for you. Thankfully there is the styler package!
Since we will use it for the project as a workflow dependency, let’s add it to the DESCRIPTION file.
use_package("styler", "suggests")Like lintr, there are only a few functions in styler that we need to use. The first is to style a single file, so let’s try it in the Console:
styler::style_file("data-raw/nmr-omics.R")Styling 1 files:
data-raw/nmr-omics.R ✔
────────────────────────────────────────
Status Count Legend
✔ 1 File unchanged.
ℹ 0 File changed.
✖ 0 Styling threw an error.
────────────────────────────────────────Since we already have the file all nice and tidy, it doesn’t do anything. But, this function is a bit much to type. We could also do it on all files in the project with:
styler::style_dir()You will probably be asked to install something, click “Yes”.
But, like the linting we did above, we only need to style all the files occasionally. Usually we would need to only style the file we are actually working. We can do that through the Command Palette (Ctrl-Shift-P) and typing “style file”, which should show the “Style active file” option. You’ll try it out in the next exercise.
The thing to note, though, is that styler isn’t perfect, so by using it together with lintr and sometimes manually running reformatting (Ctrl-Shift-A or with the Command Palette Ctrl-Shift-P then “reformat”) you will be able to fix most syntax issues.
Time: ~15 minutes.
Open your doc/lesson.Rmd file, create a new code chunk (Ctrl-Shift-I or Ctrl-Shift-P followed by typing “chunk”) at the end of the file. Copy and paste the below code:
Open the Git interface and commit the changes to the Git history. Then push the changes to GitHub. Go to the Actions tab and see what happens. It might take a few minutes to finish.
Go back to doc/lesson.Rmd file in RStudio and run lintr::lint() with the Command Palette (Ctrl-Shift-P then type “lint file”). What happens? Don’t do anything just yet, simply see what things were identified as potential issues.
Run styler::style_file() using the Command Palette (Ctrl-Shift-P then type “style file”) while inside the doc/lesson.Rmd file. What happens? Open the Git interface to see the changes made. Commit the changes and push to GitHub. Go into the Actions tab to see what is different now.
Re-run the lintr::lint() with the Command Palette. What’s changed since you did it last?
For multi-person collaborative projects, having some type of code styling and checker can really help with standardizing how the code looks, which ultimately will make it easier to read each other’s code contributions.
But what about for Markdown files? While there isn’t a package or function (yet) that styles the Markdown files, RStudio does have an option in their Tools to format Markdown into a “canonical form”. The reason for this option is because they added a “visual editor mode” to writing R Markdown files (which is great if you are more comfortable with apps like Word). Let’s test out this option. First, let’s make sure everything has been committed to the Git history.
Use this option only if you have your project under Git version control, since it will directly modify and overwrite the contents of the entire file.
There are two ways of doing this:
Tools -> Project Options -> R Markdown and changing the options “Automatic text wrapping” to “column” (with the default 72 width value) and “Write canonical visual mode markdown” to “true”._quarto.yml file (we will cover this in Chapter 9) or at the file-level in the YAML header.For right now, we will do the file-level YAML settings. Open the doc/lesson.Rmd file and go to the top of the file. Right below the last ---, create a new line above it and paste this code in:
editor_options:
markdown:
wrap: 72
canonical: trueNow, when you save your file, RStudio should automatically reformat the Markdown into a standardized format. If you want to switch to using the Visual Mode, use Ctrl-Shift-F4 or the “Visual” button at the top of the Source Pane beside the bolding and italicizing buttons.
The instructors won’t be using the Visual Mode during the course, however you are welcome to. We will be using the “canonical” markdown mode though.
Let’s test it out. While in the doc/lesson.Rmd file, go to the bottom of the file and type out:
## This is poorly formatted
- Definitely should have an empty space above this list.
- This isn't a list, why not?Save the file. What happens? Lists in Markdown must have an empty space above them to work properly. With this canonical mode on, we can get feedback right away that it isn’t right. We fix it by adding that empty space.
## This is poorly formatted
- Definitely should have an empty space above this list.
- This isn't a list, why not?Since this mode is on automatically, as we work through the sessions, we’ll get lots of experience using it.
Time: ~2 minutes.
There are many options inside the Global Options in RStudio that can help you work better and faster. There are a few that will help a lot, especially in this course and with the workflows we are showing:
Tools -> Global Options -> Code -> Saving.
Time: ~10 minutes.
Open up the README.md file and copy and paste these YAML metadata to the top:
---
editor_options:
markdown:
wrap: 72
canonical: true
---Then, start completing the TODO items. Save often and watch as the Markdown gets reformatted. After you are done, commit the changes you made to the Git history. Then delete both TODO.md, followed by committing these deletions in the Git history. Click the “Push” button to push the changes to GitHub.
We’re at the end of the session and now need to fill in the survey. There is a function in r3, but r3 isn’t installed in our project library. However, because r3 has a few GitHub packages as dependencies, we need to tell renv to authenticate us to GitHub so we can install as much as we want (GitHub only allows a few unauthenticated installed, as a security feature against hackers and cyber attacks). We need to create a variable called GITHUB_TOKEN by running this function from credentials.
credentials::set_github_pat(TRUE)Then we can install the r3 package into our project with renv::install(). Because the r3 package is on GitLab, we need to tell renv about its location by adding gitlab:: followed by the “user” (which is rostools) and than the repository (r3).
renv::install("gitlab::rostools/r3")Now we can open the feedback survey:
r3::open_feedback_survey_advanced()options() to automatically make snapshots so you can use the use_package() function.renv::restore() or renv::install().Ctrl-Shift-P) to quickly run their functions on code you are actively working on.