Many of you probably work mostly on your own, but as you move through your career (whether in academia or industry), you will need to—and perhaps want to—directly collaborate1 much more with others.
1 Here, “directly collaborate” means contributing to a shared project directly (like editing the same files), rather than discussion- or planning-based collaborations (and definitely not the “emailing-files-around” type of collaboration).
Even though different types of collaboration (e.g. meetings, brainstorming, real-time co-writing) form the basis for almost all research-based work (and probably most non-research-based work), direct collaboration quickly becomes unmanageable when relying on traditional academic “workflows” (like emailing files around). That’s when you’ll need tools designed for collaboration, such as Git and GitHub. But that’s just the starting point; many other factors related to workflows and processes must be considered to collaborate more effectively.
This session focuses on using more automated ways to structure data analysis projects and facilitate smoother collaboration.
This session’s overall learning outcome is to:
Specific objectives are to:
Time: ~10 minutes.
Reflect on when you work on a project (for your thesis or a manuscript), how exactly do you and your collaborators contribute to the project:
Consider these questions as we do the following steps.
Note: This first session is more conceptual and is heavier on the reading and explanation, but is important for the next sessions.
One of the first things to consider when working collaboratively on a data analysis project (and probably other types of projects too) is what software to use for your project. This starts out at the highest level: Are you using R or some other software for the analysis? Since this is an R course, we’re assuming the software will be R! 😜
The next consideration is determining which packages your project will depend on to produce the results. When working collaboratively with others (or even with your future self), you’ll need a clear way to track the project’s package dependencies, and, ideally, have a simple and efficient method to install or update them.
Part of the approach covered in this course requires that you follow a “project-oriented” workflow when working on, well, a project. In order to know how to track your project’s package dependencies, you need to first know, what a “project” is and how do we work around it. Since the introduction course’s first session on the Management of R Projects, we’ve consistently taught and used this workflow-style. In fact, it is embedded into the use of the R Projects via the .Rproj files and in the use of the here package. So, we’re already following a approach, which will make it easier to track package dependencies of our project.
setwd(). Artwork by @allison_horst.Let’s start with the AdvancedR3 project, created in Chapter 3, that includes the lipidomics dataset. We have code in the data-raw/nmr-omics.R file already that uses some packages. Let’s assume that your project will be more complex than this and that you will eventually need some collaborators to contribute who are experts in for instance metabolomics data processing and in statistical analysis of high-dimensional data. You know you will end up needing to use other packages. You also know that you all need some way of tracking which packages are used so that when others join and contribute to the project, they can as seamlessly as possible install or update the packages your data analysis project needs. There are a few ways of “tracking” package dependencies.
The simplest, but most primitive way is to always make sure to use library() at the top of each R script for each package that the R script uses. We can call this “informal” dependency management.
Let’s review the advantages and disadvantages of this form of dependency management:
Advantage:
Disadvantages:
It doesn’t track project-level dependencies very well, since multiple scripts might use the same packages, while some might use different ones. This means that you can’t easily and quickly install or update all the packages used in your project. You will probably have to go through each R script manually and install each package manually.
It doesn’t track the versions of the packages your project depends on, so if a package gets updated and it breaks something, you might not be able to figure out how to quickly fix that issue, especially for those deadline crunches.
You might have seen a similar “informal” approach where scripts starts with code that looks like this:
if (!require("packagename")) {
install.packages("packagename")
}This code checks whether a package exists (i.e., is installed), and if not, it installs it. But! This is not an optimal method to track packages because require() won’t load the package if it doesn’t find it. As a result, you will have to rerun the script a few times if you don’t have that package installed; first, to install the package, then to load it. Plus, sometimes you may need to restart the R session after installing a package in order for R to detect it properly. So, this is not a very efficient way to track dependencies either.
What’s the alternative, then? One alternative is a more formal way of managing your dependencies, which is what we’ll cover in the next section.
Before moving on to the exercise, be sure to add and commit all the files in the AdvancedR3 project to save the current state of the files to the Git history. Open the Git interface by either typing with Ctrl-Alt-MCtrl-Alt-M or with the Palette (Ctrl-Shift-PCtrl-Shift-P, then type “commit”) or by going to the Git pane and clicking the “Commit” button. Then, write a commit message and click the “Commit” button in this window.
Time: ~10 minutes.
Let’s update the DESCRIPTION file with the packages we depend on in the data-raw/nmr-omics.R script. Open nmr-omics.R file and complete the following tasks:
library() or ::. It can help to use the “Find in files” feature in RStudio to look for all places that have either :: or library. Use either Edit -> Find in Files ..., or with Ctrl-Shift-FCtrl-Shift-F or with the Palette (Ctrl-Shift-PCtrl-Shift-P, then type “find files”) to open up the search popup.?usethis::use_package to review how to use this function.usethis::use_package(packagename) for each package you find in data-raw/nmr-omics.R (from 1. above).Console
usethis::use_package("stringr")
usethis::use_package("readxl")
usethis::use_package("dplyr")
usethis::use_package("tidyr")
usethis::use_package("snakecase")
usethis::use_package("here")
usethis::use_package("fs")
usethis::use_package("readr")When you work on a research project that involves data analysis, you’ll likely use packages in two different ways:
A good rule of thumb to determine whether a package is a build or a workflow dependency is:
So, now we know that the dependencies we added in the exercise are actually build dependencies, since we use them directly in our project to process the data.
The way you add packages to the DESCRIPTION file is slightly different depending on what type of dependency it is. For build dependencies, we use the function we’ve already used before: usethis::use_package("packagename"). For workflow dependencies, it’s the same function with a small twist. Our first workflow dependency is usethis, which we will use many times throughout the course.
But before we get into adding usethis to the DESCRIPTION file, let’s tweak our personal (also called the global) .Rprofile to make our lives a bit easier. We’ll add a setting so we don’t have to type usethis:: before each function.
Run the usethis::use_usethis() function to open your own (user) .Rprofile:
Console
usethis::use_usethis()Then, copy and paste the code below into your .Rprofile, at the bottom of the file.
if (interactive()) {
suppressMessages(require(usethis))
}Let’s restart R with Ctrl-Shift-F10Ctrl-Shift-F10 or with the Palette (Ctrl-Shift-PCtrl-Shift-P, then type “restart”). Now, we’re ready to use use_package() to add usethis as a workflow dependency. The only difference between adding a build and a workflow dependency is that for workflow dependencies, we need to add a second argument to use_package() to specify that it is a workflow dependency. The second argument is "suggests", like so:
Console
use_package("usethis", "suggests")If you open the help file for use_package() you’ll see that the second argument we used above is called type, and “Imports” is its default value. So, if you don’t specify the second argument, the package will be added under “Imports” in the DESCRIPTION file. This is exactly what we did in the exercise in Section 5.5!
So, “imports” corresponds to build dependencies, while “suggests” are equivalent to what we call workflow dependencies.
Open the Git interface and see that under Suggests: in the DESCRIPTION file is usethis. Let’s commit these changes, with Ctrl-Alt-MCtrl-Alt-M or with the Palette (Ctrl-Shift-PCtrl-Shift-P, then type “commit”).
When you come back to a project after a few months or if you start collaborating on a project, you can run pak::pak() to install all types of dependencies, both workflow and build. Neat! 🎉
Time: ~25 minutes.
Since we will eventually connect our project Git repository to GitHub to practice working with GitHub, we’ll connect our project to GitHub right now.
Remember that GitHub is a platform for sharing your projects in a way that is transparent (if you add all the relevant files and keep a clean history of your files) and makes it easy to collaborate with others. It’s a great way to share your project with others, get feedback, and even contribute to other projects.
Let’s complete these tasks to connect to GitHub.
First, commit the latest changes to the Git history with Ctrl-Alt-MCtrl-Alt-M or with the Palette (Ctrl-Shift-PCtrl-Shift-P, then type “commit”).
If you haven’t yet, please create a GitHub account.
Add the gitcreds package as a workflow dependency. You’ll need it for the next item
Console
use_package("gitcreds", "suggests")Read through and complete the tasks in the linking your project to GitHub of the Connect to GitHub Guide.
Check your GitHub to make sure the project repository has been uploaded to it.
Since we will use styler for the project as a workflow dependency, let’s add it to the DESCRIPTION file:
Console
use_package("styler", "suggests")There are only a few functions in styler that we can to use. The first is used to style a single file with R code in it using styler::style_file(). However, an easier way to style the file we’re currently working on is the “style active file” RStudio addin. We use that through the Command Palette (Ctrl-Shift-PCtrl-Shift-P) and typing “style file”, which should show the “Style active file” option. This is what we will do frequently throughout the course.
Let’s try it out. While inside the data-raw/nmr-omics.R file, use the Palette (Ctrl-Shift-PCtrl-Shift-P, then type “style file”) to style the file. Notice that no changes were made since the file is already tidy.
You will probably be asked to install a package called {miniUI}, click “Yes”.
If you wanted to run styler on all the files, we can use:
Console
styler::style_dir()Before we move on, let’s make sure everything has been committed to the Git history with Ctrl-Alt-MCtrl-Alt-M or with the Palette (Ctrl-Shift-PCtrl-Shift-P, then type “commit”).
For multi-person collaborative projects, having some type of code styling and checker can really help with standardizing how the code looks, which ultimately will make it easier to read each other’s code contributions.
But what about for Markdown files? While there isn’t a package or function (yet) that styles the Markdown files, RStudio does have an option in their Tools to format Markdown into a “canonical form”. The reason for this option is because they’ve added a “visual editor mode” to writing R Markdown / Quarto files (which is great if you are more comfortable with apps like Word). Let’s set up the project to automatically reformat Markdown files.
Use this option only if you have your project under Git version control, since it will directly modify and overwrite the contents of the entire file.
There are two ways of doing it:
Tools -> Project Options -> R Markdown and change the options “Automatic text wrapping” to “column” (with the default 72 width value) and “Write canonical visual mode markdown” to “true”._quarto.yml file or at the file-level in the YAML header.For right now, we will do the project option settings so that as long as we are using RStudio and in the R Project, it will automatically reformat the Markdown files as you write in them. Follow the instructions in item 1 above to set the options.
Now, when you save your file, RStudio should automatically reformat the Markdown into a standardized format. If you want to switch to using the Visual Mode, use Ctrl-Shift-F4Ctrl-Shift-F4 or the “Visual” button at the top of the Source Pane beside the bolding and italicizing buttons.
The instructors won’t be using the Visual Mode during the course, however, you are welcome to do so. We will be using the “canonical” Markdown mode in the “source” (default) mode though.
Let’s test it out. While in the doc/learning.qmd file, go to the bottom of the file and type out:
## This is poorly formatted
- Definitely should have an empty space above this list.
- This isn't a list, why not?Save the file. What happens? Lists in Markdown need to have an empty space above them to work properly (except for when they are below a header, but in all other cases it needs a space above). With the canonical mode on, we can get feedback right away that it isn’t right. It gets automatically fixed by adding that empty space.
## This is poorly formatted
- Definitely should have an empty space above this list.
- This isn't a list, why not?Since this mode is on automatically now, as we work in the doc/learning.qmd file through the sessions, we’ll get lots of experience using it.
Time: ~10 minutes.
Open up the README.md file and do all the TODO items found inside the file as well as inside the TODO.md file. Save often and watch as the Markdown gets reformatted.
After you are done, commit the changes you made to the Git history with Ctrl-Alt-MCtrl-Alt-M or with the Palette (Ctrl-Shift-PCtrl-Shift-P, then type “commit”).
Then delete the TODO.md, followed by committing these deletions in the Git history. Click the “Push” button to push the changes to GitHub.
In this chapter, we have covered how to do the following in a project for smoother collaboration (and nicer-looking code and documents):
DESCRIPTION file and usethis.pak::pak().