Before 1996 NSF used four criteria to determine if a proposal would be awarded. The four criteria were: (I) Researcher Performance Competence; (II) Intrinsic Merit of Research; (III) Utility or Relevance of the Research and; (IV) Effect of the Research on the Infrastructure of Science and Engineering. These criteria were first adopted in 1981 and used until 1996.
The re-assessment and subsequent replacement of these four criteria resulted from a combination of events that happened at the national level and with NSF. Congress passed the Government Performance and Results Act (GPRA) in 1993. This now meant that NSF’s goals and strategies needed to be linked to the outcomes of the investments it had made throughout the country.
By the mid-1990s NSF had expanded its portfolio to include several things like broad education initiatives. NSF had also adopted a new strategic plan in 1995 (NSF 95-24). NSF 95-24 embraced some new long-term core strategies and goals. One of the aims of these long-term strategies and goals was about promoting knowledge in service or “benefit” to society. This strategic plan was called “NSF in a Changing World”. Meaning that NSF specifically needed to exemplify, participate, provide proof, acknowledge, and measure the societal benefit of its agency’s mission. “They needed to have a broader impact”! All of which was consistent with the trend happening internationally.
Official re-assessment of these four criteria were performed by a combined National Science Board (NSB) and NSF Task Force. The potential implementation of this term as a criterion was first provided in the National Science Board (NSB) and NSF Staff Task Force on Merit Review Discussion Report. A discussion about NSF’s Intellectual Merit criterion was also provided in this summary, NSB and NSF 1996 Task Force Report.
After the task force’s findings, NSB and NSF were convinced that the effective utilization and implementation of the broader impacts criterion would better allow NSF to connect its investments to societal value and benefit. NSF thought that it would allow them to better demonstrate and more clearly align their goals to their strategic plan. In addition, the use of this term, as a criterion, would allow NSF to combine and replace at least two out of the four criteria used from 1981 to 1996. A result that would also allow NSF to streamline its review process.
In 1997 NSF officially introduced broader impacts to the nation via notice 121. The two 1981-1996 criteria, (labelled III and IV above), were replaced by the Broader Impacts Criterion. NSF stated that broader impacts would be used as one of their criteria for determining if a proposal would be awarded. This new term was defined as encompassing the potential to benefit society and contribute to achievement of specific, desired societal outcomes. The term that replaced the other two 1981-1996 criteria, (labelled I and II above), was called Intellectual Merit.
In 1998 and 1999 Congress told NSF to work with the National Academy of Public Administration (NAPA) to determine the effects of their new merit review criteria. The 2001 NAPA report discussed the academic and research community’s views and challenges of the broader impacts criterion from 1997-2000. As J. Britt Holbrook wrote in his National Science Foundation: Second Merit Review Criterion editorial, “NAPA stated that NSF needed to address a host of philosophical issues raised by the broader impacts criterion”.
After the NAPA report, NSF began to try to address these philosophical issues. NSF continues to make efforts to address many of these challenges. This has been in part due to the apparent confusion about broader impacts and how it was inconsistently applied as a criterion. This sentiment has continued to be expressed by many across the nation.
Several events that followed the NAPA report began to shape NSF’s current version of the Broader Impacts Criterion. During the mid-2000’s many started to question the efficacy of and issues surrounding NSF broader impacts. This led to the American Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science Act of 2007. This is commonly known as the American COMPETES Act.
In the years following there were also other major attempts to address NSF’s Broader Impacts Criterion. Major events that have shaped or been influential in the response to the criterion between 2010-2017 are provided below. Note: the specific nuances outlining how each of these events historically developed will not be discussed.
First, Congress approved the American Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science (COMPETES) Reauthorization Act in 2010. This legislation mandated broader impacts and encouraged Institutes of Higher Education (IHEs) to assist investigators in achieving the Broader Impacts criterion. It also required investigators to provide evidence of institutional broader impacts resources.
In the year that followed, the National Science Board (NSB) provided statements supporting the Broader Impacts Criterion. The NSB stated that NSF broader impacts would continue. NSB also stated that IHEs should provide support for investigators because there was still confusion about the criterion.
Between 2011 and 2012, NSF made some procedural changes which provided nine over-arching long-term outcomes as it related to broader impacts. This is discussed further in Jeffery Mervis’s write-up in Science Mag News online. Some refer to this as Broader Impacts 2.0.
In 2013, the National Alliance for Broader Impacts (NABI) was formed. The National Alliance for Broader Impacts (NABI) received funding through a NSF Research Coordination Network (RCN) grant. NABI is comprised of about 200 institutions, companies, colleges, and organizations across the nation.
The goal of NABI is to create a community of practice that fosters the development of sustainable and scalable institutional capacity and engagement in the broader impacts of STEM research activity. NABI holds a national summit every year. In 2018 it will be hosted by Brown University.
In early 2015, NABI convened a working group to develop a guiding document for the NSF Broader Impacts Criterion. The purpose of this document was to assist program managers, proposal reviewers, and review panels in their evaluation of broader impacts components of NSF proposals. The guiding document was the first nation-wide/community attempt to begin the process of normalizing the way review panels evaluated and rated proposed NSF broader impacts plans. This document was released at the end of 2015 and has been nationally vetted and used across the nation.
In 2016, the American Innovation and Competitiveness Act (AICA) was passed by the 114th Congress. The AICA was considered the successor to the America COMPETES Acts of 2007 and 2010. A brief discussion and the implications of AICA was provided by Mitch Ambrose in the American Institute of Physics’ (AIP) Science Policy News.
Also in 2016, a workshop about NSF broader impacts was organized by Sheldon Jacobson (from University of Illinois), Jerome Hajjar (from Northeastern University), Dawn Tilbury (from University of Michigan), and Andrew Johnson (from Texas A&M University). This workshop was called, “Setting a Broader Impacts Innovation Roadmap”. It took place in Arlington, Virginia and was supported by NSF (CMMI-1629955). At this workshop, researchers and administrators, mostly from the Mechanical, Industrial, and Civil Engineering disciplines, deliberated about how investments made by NSF could be enhanced by broader impacts contributions.
There have been other workshops, meetings, and seminars on broader impacts. However, this appears to be the first time publicly any group of specific disciplines, specifically several engineering disciplines came together to provide answers to questions about the NSF Broader Impacts Criterion and begin to articulate aspects of the concept. This will be discussed more in a future paper.
Lastly, in 2017, approximately a month after their yearly summit, NABI held a two-day meeting to discuss the Future Directions of Broader Impacts. This meeting brought together NSF officers, policy professionals, university administrators, broader impact and extension professionals, and representatives from two- and four-year institutions. This included Historically Black College and Universities (HBCUs), Minority Serving Institutions (MSIs), and Hispanic Serving Institutions (HSIs). During this time, there were discussions on developing needs assessments, strategies for creating consensus, and how to move broader impacts forward professionally and logistically. One topic that began to surface in this meeting centered on the Research and Scholarship of Broader Impacts (SoBI), see footnote six.