Proposal Selection Process

Distributed Peer Review

All submitted proposals during a Cycle are peer-reviewed. It is expected that all participants in the review process will behave in an ethical manner. Proposals are reviewed within the Science Review Panels that are set up based on broad science topics, e.g., stars, galaxies, exoplanets. etc. During Cycle 1 the Panels met virtually online. Proposals are assigned to non-conflicted reviewers based on the expertise of each reviewer and are subject to a two-stage review process. All reviewers submit an asynchronous preliminary grade on each proposal prior to the panel discussion. The grading scale is 1 (weakest) to 5 (strongest). All non-triaged proposals are discussed and re-graded during the panel review. (The triage process removes lower-ranked, non-competitive proposals from consideration and ensures that discussion time is focused within a panel on the higher-ranked, competitive proposals.) Average grades for all non-triaged proposals will then be sorted to determine an overall ranked list. That list is then further discussed, with considerations of the variety of science potentially selected by and the number of observing programs and research funding allocated to each Panel.

Reviewers are also required to provide detailed written comments on each proposal that they have been assigned as a designated reviewer, either primary or secondary. The comments are intended to provide justification and explanation for the final assigned grade of a proposal, and to provide feedback that could potentially help investigators make improvements on a proposal not selected.

Evaluation Criteria

The Science Review Panels and the Time Allocation Committee will base their evaluation of proposals on the criteria listed below. The criteria are listed in descending order of importance.

Proposals are evaluated according to these criteria:

  1. The overall scientific merit of the proposed investigation, its potential contribution to the advancement of scientific knowledge, and its potential for enabling new and important types of scientific investigations.
  2. The extent to which the proposed investigation leverages the unique capabilities of Roman or demonstrates the relevance to Roman science.
  3. Well-developed analysis and implementation plans describing how the scientific objectives will be realized and the results distributed to the community.
  4. For proposals requesting new observations only: the technical feasibility of the proposed observations, and the extent to which the observations can be accommodated within routine Roman operations. Proposals for Cycle 1 were expected to provide a compelling justification for why the particular survey should be executed in the first two years of the mission.

Final Ranking and Time/Funding Allocation

The proposal ranking for a Panel uses the average final grades from all panelists to produce a rank-ordered list of proposals. The Panels will review the rank-ordered list to determine whether highly ranked proposals above the nominal cutoff represent an appropriate science balance for each panel. A consensus may exist for a Panel that some science areas may have been unduly favored. A situation may also exist for which the Panel Chair has identified highly ranked proposals that have science overlap with highly ranked proposals in another panel. Panel members can make a consensus decision to re-rank proposals to provide a more appropriate reflection of the various science topics reviewed by that panel. The Large category non-observing proposals do not have an allocation of money at the Science Review Panel level, but will be ranked together with the Small and Medium category proposals. Similarly, a Panel will promote 1–2 observing proposals (these can be of any size category) to the TAC for consideration for approval.

Panels are provided with a per-panel allocation of funding for Small and Medium General Investigator programs, consisting of analysis of Roman WFI data, theoretical or laboratory astrophysics research, and analysis of publicly accessible ancillary data with direct relevance to Roman science, and number of observing programs for new General Astrophysics Surveys (GAS). Note that not all proposal Cycles will include calls for both General Investigator and GAS programs --- the SSC expects to release three solicitations for new GAS proposals among the currently-planned five calls for proposals during Cycles 1, 3, and 4. The allocations are determined by the relative time and funding pressure across science areas and proposal types and are, in principle, different for each panel. Proposals “above the line” for funding or observing time will likely be recommended for approval; those “below the line” will likely not be recommended; and the few that are “on the line” may or may not be recommended.

Panels generally will be asked to rank proposals down to twice the number of proposals within each funding level. This is done in case any changes need to be made to those proposals above the line, allowing lower-ranked proposals to ascend in the ranking. Panels can also set a do-not-approve line, if they deem it appropriate. Panels can re-rank two or more proposals that are judged to have very similar science in pair-wise fashion to the extent that a Panel may recommend approving only one of the proposals. Panelists without a conflict will then be asked to vote toward a comparative re-ranking, until a final ranked list of proposals has been established within the panel.

Time Allocation Committee

Proposals are further reviewed by the Time Allocation Committee (TAC), which consists of the Panel chairs, at-large TAC members, and the TAC chairs, following the Science Review Panel discussions and rankings. During Cycle 1 the TAC met in-person at the Roman Science Support Center (SSC) at Caltech/IPAC in Pasadena, CA. The TAC is responsible for discussion and ranking of the Large General Investigator proposals, as well as the final selection of GAS observing programs. The GAS programs will account for approximately 25% of the total observing time during the nominal five-year Roman mission. As a result, a maximum amount of observing time is imposed for each of the three Cycles when a call for GAS proposals will be issued, which results in an anticipated maximum number of approved GAS programs. For Cycle 1, for instance, it is anticipated that the maximum number will be six programs. Any changes that are required to be made to proposals above the line in the various panels are made by the TAC. The TAC is generally responsible for the full list of recommended programs of all categories. Final approval of this list is the responsibility of the Roman Project Science Office (PSO). The actual level of NASA funding available for both General Investigator and GAS programs may not be known at the time of the Science Review Panel and TAC deliberations. It is therefore the PSO, working together with NASA program management, which ultimately decides which programs are fully approved for each Cycle.

Dual Anonymous Peer Review

The SSC is responsible for ensuring that the community has equal opportunity access to use the Roman Space Telescope and that the best science will be selected, given the limited resources available. The SSC places a high value on the equity and integrity of the proposal review process. The SSC has therefore adopted Dual Anonymous Peer Review (DAPR) to reduce bias and ensure reviewers focus on the scientific merit, rather than the identity of the proposer or proposing team. DAPR has been successfully used in various NASA proposal reviews. Guidelines for DAPR will be provided to both proposers and reviewers for each proposal Cycle. Any proposal that contains significant violations of DAPR guidelines may be rejected. Proposers will be provided with feedback regarding any detected violations to avoid similar violations in future proposal Cycles. For more details on DAPR, please refer to the NASA ROSES guidelines.

Conflicts of Interest

The goal of the proposal review is an informed, unbiased discussion of each proposal's science case as described in the Selection Criteria and Scoring System section:

  • Science Review Panelists should have neither direct nor indirect interest vested in the outcome of the review;
  • The subset of the panelists discussing a proposal should have sufficient knowledge to assess the science.

It is considered critically important that conflicts of interest, or the appearance thereof, are avoided during the selection process. Proposal reviewers are asked to identify potential conflicts of interest as part of the recruitment and proposal assignment process. Panelists with any conflicts on proposals in their panel do not provide grades for or comment on a proposal with which they have a conflict. During the proposal discussion, conflicted panelists are asked to leave a panel meeting during the discussion and ranking of a conflicting proposal.

Confidentiality

Proposals submitted in response to a Call for Proposals will be kept confidential to the extent allowed by the proposal review process. For approved investigations only, the SSC will make the titles, investigator names, and abstracts publicly available after selected proposals have been announced. The remainder of the approved proposals, and the entirety of proposals not selected, shall remain confidential.

Restriction on Investigator Nationality

The Roman General Investigator program is open to investigators of any nationality or institutional affiliation. However, submitted proposals involving only investigators based in the People’s Republic of China (PRC) or involving only PRC-based and U.S. investigators are subject to the NASA restriction on bilateral activity with China (see Section 1340 of the Public Law 112-10). Such investigations, whether funded or performed under a no-exchange-of-funds basis, shall be ineligible for an award. See the NASA ROSES PRC-related Frequently Answered Questions for further details. For purposes of this provision, although Hong Kong is considered part of the PRC, Taiwan is not.