Introduction

The breadth and sophistication of software development skills required to build and maintain research software projects are increasing at an unprecedented pace. Researchers thrust into the role of developing software, with little or no software development experience or training, often employ ad-hoc, potentially detrimental development methods. It has become clear in recent years that the level of effort and required skills to keep pace with computer and programming tools is not in the repertoire of the average researcher (Merali, 2010). When novices develop software or when researchers are more focused on research publications than on producing quality software, problems can arise that limit its usability, sustainability, reproducibility, and even accuracy (Miller, 2006; Bhandari Neupane et al., 2019). Researchers often lack both the time to refine initial software implementations and the incentives to use best practices while focusing on domain-specific advances. As a result, it is not uncommon for software tools and research code to become unusable after a project ends or the primary developer leaves.  

This has led to the emergence of the Research Software Engineer (RSE), a role that is relatively new to both Princeton and the broader research community. Many researchers have not had experience working with an RSE, and because there are not many defined RSE roles, most are hired into the role without having held an RSE position before.

This document defines the RSE role at Princeton, introduces the mission of the RSE group, and summarizes best practices for a successful partnership with the RSE group.

What is an RSE?

Coined in 2012, the term Research Software Engineer has been broadly used as an inclusive title to describe anyone who understands and cares about good software and research (Baxter et al., 2012). Specifically, an RSE views the development of research software as the primary output of their work efforts. This distinguishes RSEs from domain researchers who view research publications as the primary focus of their work.

By combining an intimate knowledge of research with the skills of a professional software engineer, RSEs can transform traditional computational research by directly compensating for a domain researchers’ lack of software development expertise. An experienced RSE has the tools and knowledge to work collaboratively with domain researchers in a manner that ensures the quality, performance, reliability, and sustainability of the software.

In addition to software development and technical expertise, RSEs provide two other critical advantages in the research software ecosystem. First, RSEs serve as leaders and mentors to novice software developers, including undergraduates, graduate students, and postdoctoral researchers. By providing mentorship on research software projects, RSEs elevate domain scientists’ development through exposure to professional best practices. Second, RSEs are recruited to design and deliver software training programs to students and researchers. These training programs are uniquely beneficial as RSEs understand the technology, audience, and requirements of research software.

The RSE Group at Princeton

The Research Software Engineering group is led by the Director of Research Software Engineering for Computational and Data Science and reports to the Associate CIO for Research Computing.

RSE Group Mission

The Princeton RSE group’s mission is to help researchers create the most efficient, scalable, and sustainable research codes possible to enable new scientific advances.

We do this by working as an integral part of traditional academic research groups, providing leadership in the design and construction of complex and highly customized software systems. 

Our group is committed to creating a collaborative environment in which best software engineering practices are valued, and to sharing and applying cross-disciplinary computational techniques to new and emerging areas.

RSE Partnerships & Funding

Funding for the RSE program is provided by the Offices of Information Technology, the Provost, and the Dean for Research. Two partnership models are available within the RSE group: Co-Funding and Full Partner Funding. 

Co-Funding: These RSEs have a home department in Research Computing and report formally to the Director for Research Software Engineering for Computational and Data Science. Each RSE is assigned to a partnering department, institute, center, consortium, or PI (referred to as "partner"). These partners agree to fund between 0%-50% of the position for a finite term (typically 1-3 years) and Research Computing provides the remainder. Potential partners may submit short proposals for review by the RSE Steering Committee who award available RSE partnerships. The RSE’s Annual Performance Review (APR) is completed by the Director of Research Software Engineering for Computational and Data Science. 

Full Partner Funding: These RSE positions have a home department outside of Research Computing. A partner fully funds an RSE position and commits to the RSE's participation in the RSE group with the same expectations as for other RSE members (including those who are co-funded). This includes meeting the expectations listed in the next section (RSE Expectations). The partner and RSE group leadership “co-supervise” the RSE. The RSE’s home department and reporting structure will be formally within the partner department/PI and the details of the co-supervision will be agreed upon at the time of hire. The partner covers all costs on items such as computer & peripherals, conference travel & fees, etc. The RSE’s Annual Performance Review (APR) will be completed by someone in the funding department with input from the RSE group leadership. 

Regardless of funding method, the expectation is an open and collaborative evaluation between partners to ensure a successful engagement and as such, feedback from partners and all participating PI’s is regularly sought.

RSE Expectations

Regardless of funding, each member of the RSE group is expected to work on the research software projects as prioritized by the partner with an effort of roughly 85% of their time. The RSE group’s leadership team will support the RSE in ensuring that the priorities laid out by the partner are followed.

The remaining 15% of resource time is allocated to RSE group activities and professional development. The RSE, along with the RSE group leadership, will track effort to ensure that the 15% target is met as closely as possible. Too much would take away from project work, too little limits growth and minimizes the ability to leverage the collective RSE group knowledge.

These activities include, but are not limited to:

• One-on-one meetings with their direct manager from the RSE group leadership team 
• Participation in all RSE group meetings. This includes:
  • 15-minute standup meetings once a week
  • One 90-minute RSE group meeting every other Wednesday
  • One 60-minute Research Computing department meeting once a month
• Occasional (2-4 times per year) participation in RSE group events such as:
  • RSE project sprints - These short 1-3 day events provide a small team (2-4 RSEs) an opportunity to collaborate on a particularly challenging or technically edifying task. This leverages group expertise and provides one project the opportunity to benefit from others as well as an opportunity for RSEs to learn new techniques and strategies. 
  • RSE code reviews - The best way to improve code quality is through code review, and one of the best ways to learn new programming approaches is to read other people’s code. RSE peers can take turns reviewing each other’s code for specific feedback and advice. Both the reviewer and reviewee benefit, as does the project whose code is the subject of the review. 
• Professional development. As largely independent professionals, RSEs must spend time and effort to maintain knowledge on advancing technologies. Research changes rapidly. Software development best practices change even faster. Learning is essential in these positions. Sometimes this is facilitated by participation in RSE group conversations. However, a number of other options should be explored on a regular basis:
  • Attending training, tutorials, conferences. 
  • Contribute to open source projects and other project-based learning opportunities.
  • Collaborate on other Princeton RSE group member projects
• Deliver Training (optional)
  • As mentioned in the RSE overview, RSEs have knowledge of software engineering best practices that are extremely useful to share with researchers. One of the most impactful approaches to sharing such information is through tutorials and workshops. PICSciE oversees a number of such events. When interested RSEs are encouraged to deliver workshops and mini-courses to share their knowledge with the broader community. 

In many cases, RSEs are asked to track their time. This is done on a case-by-case basis depending on the specific situation of each RSE. RSEs who are supporting more than one initiative typically track their time to ensure that each project is getting an appropriate effort.

RSEs are typically classified in the University as Exempt AIT professionals with a 36.25-hour work week (7.25-hr work day). They accrue 2 vacation days each month. RSEs are committed professionals who are dedicated to their projects’ success. RSEs are expected to be independent and capable of managing their own time and efforts. That being said, awareness of an RSE’s expected weekly effort by the partner and the RSE group leadership will contribute to continued success, and as such, performance evaluations are based on impact and production rather than timekeeping.

RSE Project Intake, Assignment & Management

Allocating individual RSE time to specific projects can be managed in a manner best suited for all parties involved. Requirements tend to be unique and therefore the structure of project intake and assignment should be organized in a way that addresses the needs of a partner organization while maximizing the ability of an individual RSE to make a meaningful impact. An established process for RSE project intake is important, especially when multiple parties are involved. RSE project intake typically falls into one of the following models:

PI project proposal & review

For co-funded positions, this is the most typical form of project intake and assignment, especially when the partner is a department or center and has many potential constituents. At regular intervals, an email request for software project proposals is sent to associated faculty in the partner department/center. Interested faculty respond with a short description of a potential project, and a meeting with the RSE and RSE group leadership is scheduled. During the meeting, goals and deliverables of the project, an estimated scope, and the expected time commitment are discussed while guidelines for productive collaboration are established. After spending time reviewing each proposal, the partner leadership (department chair/manager, etc.) meets with the RSE and RSE group leadership to select successful proposals and define the project commitment. The goal is to balance the strategic priorities of the partner and the feasibility of the project with the interests/skills of the individual RSE. This commitment is typically for a fraction of the RSE’s availability within a fixed project cycle (typically 6-9 months). Regular project check-ins are scheduled throughout the project cycle to ensure project goals are being met. Renewals and small maintenance projects are possible through subsequent calls for software projects cycles. 

Single project partnership

For fully-funded partnerships, this is the most common and simplest form of project assignment. As the RSE is funded in alignment with a single project, that project is the sole focus of the RSE’s effort. As such, there is no need for a process to request, review and select new projects. Occasional project check-ins and regular communication are still important to ensure project goals are being met. 

Ad-hoc projects

Occasionally, an RSE will take on a project by direct request or through a meeting. Together with the RSE group leadership and/or the partner representative, they manage the commitment and timeline. While this model advantageously allows for flexibility and autonomy of the RSE, maintaining this model increases difficulty with predicting workloads which challenge the ability to advertise to potential PIs and can lead to an overwhelming amount of “walk-ins” and requests.

RSE Partner Expectations

As outlined above, RSE partnerships are flexible and can be managed in a manner best suited for the partner. Partnerships focused on a single project should rightfully be managed differently than a departmental partnership targeting multiple projects led by multiple PIs.

A primary goal of the RSE program is to develop a cohort of RSE professionals who are committed to both the success of research projects and the intellectual growth of the RSE group. A strong connection with their RSE peers will ensure that best practices are shared. RSEs are actively encouraged to have physical office space in both Lewis Library (to be near other RSE and Research Computing professionals) as well as in the home/partner department. In some cases, this can be difficult, or may not be possible. The recommendation is for 50-80% of an RSE’s time to be spent in the partner’s space nearest the research groups while the remaining time would be spent in Lewis Library near the other RSEs and Research Computing professionals. 

A primary benefit of collaborating with an RSE is the production of high-quality, sustainable research software. In some cases, this can mean software that remains useful and relevant for years, in other cases it can mean performance improvements (10-100x speedups are not uncommon) that accelerate research. To achieve this level of work, clear communication and joint understanding of the target lifecycle is crucial. To ensure the long-term success of a project, regular communication about the pace of development and prioritization is recommended between the partner, the RSE, and the RSE group leadership. 

As previously mentioned, RSEs are focused on producing high-quality, robust research software. This is valuable and has significant long-term benefits making the RSE role critical to producing maintainable and sustainable software projects. The focus on quality and best practices requires careful balance with research and scholarly output timelines to limit the pressure to speed up development. From a software engineering perspective, building robust software will always take longer than the development of the initial functional proof-of-concept. One of the hallmarks of an experienced RSE is their understanding of both research publication incentives and research software requirements. RSE’s balance anticipated project scope and research deadlines to produce software that not only meets the short-term research needs but is sustainable, maintainable, and extensible. Projects are most successful when timelines and project scope are developed collaboratively and refined continuously to best balance priorities. We strongly encourage communication between partners and both the individual RSEs and the RSE group leadership in the case of questions about the pace of development or the prioritization of effort. Several best practices initially slow down the addition of “new science” but facilitate the development of new aspects of a project while ensuring correctness, reproducibility, and the long-term success of the project. 

The demand for experienced RSEs continues to grow exponentially, placing Princeton in a challenging position of having to compete with a market that values research experience, mathematical and statistical software development, and experience with software engineering best practices at a salary structure we cannot always match. To retain our excellent RSEs, we want to ensure our work environment is as positive and rewarding as possible. Enabling intellectual challenges, providing work-life balance, and promoting valued collaboration are all aspects of working at Princeton that keep us competitive. Together we need to prioritize the achievement of these benefits to increase job satisfaction and retain our talented RSEs.

OIT, PICSciE, and each RSE partner need to work together to create a positive and rewarding work environment for the RSEs or we will risk losing highly talented RSEs. Communication between all parties involved in the partnership is critical. If expectations are not being met, then it behooves everyone to quickly address the problem to collaboratively work towards a solution. 

Publications & Authorship

RSEs sit at the intersection of scholarly research and software development. A common and extremely important output of scholarly research is the journal/conference publication. RSEs, by design, are not directly measured or graded based on their publication output, however many RSEs, especially those from a pure research background, still value such publications. 

We encourage RSEs, PIs, and/or partner managers to discuss authorship as soon as possible. It is typically recommended that if a publication would not have been possible without the RSE, then the RSE should be included as a co-author. 

RSEs will occasionally be interested in publishing their work in a manner appropriate for the field and individual. Please refer to the separate document on RSE publishing guidelines for details about how the RSE group handles potential RSE publications.