NSF; Method of Selection and Evaluation, Chapter 4

National Science Foundation
Small Business Innovation Research and Small Business Technology Transfer
1999 SBIR/STTR Phase I Program Solicitation and Phase II Instruction Guide

CHAPTER 4.0 Method of Selection and Evaluation Criteria

[back to Table of Contents]

4.1 SBIR/STTR Phase I
4.2 Evaluation Selection and Criteria
4.3 Debriefing of Unsuccessful Proposers

All Phase I and II proposals will be evaluated and judged on a competitive basis. NSF may elect to fund several or none of the proposers to the same subtopic.

4.1 SBIR/STTR Phase I

Proposals judged to be responsive to this solicitation will be evaluated on a competitive basis by peer review.

4.1.1 Administrative Screening.

Proposals will be screened to determine responsiveness to the specific requirements of the Solicitation. NSF will review each proposal to determine that it satisfies all administrative requirements, described on the STOP Page. Proposers are advised that failure to satisfy any one of these administrative requirements will render a proposal nonresponsive to this Solicitation. Nonresponsive proposals will be returned to the proposer without further consideration.

4.1.2 Technical Screening.

The following technical screening criteria will be applied to proposals. If the answer to any of the questions below is "NO", the proposal will be returned to the proposer without further consideration.

Does the proposal provide sufficient technical substance to enable review?

Does the proposal fall within the scope of the topic/subtopic as delineated in the topic/subtopic description?

Is appropriate research proposed in science, engineering or education?

The proposal will be returned if the research proposed is for any of the following purposes:

Weapons research;
Biomedical research (except bioengineering research , as discussed in Chapter 1.0, Section 1.2.2, Program Purposes); or
Classified research

The proposal will also be returned if it is principally for demonstration, technical assistance, literature survey or market research. Patent application and patent litigation costs are not supported under SBIR awards.

4.2 Evaluation and Selection Criteria

Proposals that are found to be responsive to this Solicitation will be competitively evaluated in a process of external merit review by scientists, engineers, or educators knowledgeable in the appropriate fields and by individuals familiar with commercial product development. Most reviewers are employed by universities or by the Federal Government. Others may be employees of nonprofit research laboratories, recent retirees from industrial firms, and, on occasion, employees of industrial organizations, including small business concerns. In all instances, proposals will be handled on a confidential basis and care taken to avoid conflicts of interest. Evaluations will be confidential to NSF, to the proposed Principal Investigator, and to the submitting small business concern, to the extent permitted by law.

Normally, more proposals will be found technically meritorious than can be supported. Evaluations by external reviewers are advisory to the cognizant program officer for the topic or subtopic, who makes recommendations on each proposal. Other factors that may enter into consideration include the following: the balance among NSF programs; past commercialization efforts by the firm where previous awards exist; excessive concentration of awards in one firm or with one principal investigator; participation by women-owned and socially and economically disadvantaged small business concerns; distribution of awards across the States; importance to science or society; and critical technology areas. The SBIR/STTR Program then makes its recommendations for awards to the Division of Grants and Agreements (DGA).

4.2.1 SBIR and STTR Phase I and II Proposals.

In the merit review process, reviewers will consider the following criteria:

Criterion 1.

What is the intellectual merit of the proposed activity? This criterion addresses the overall quality of the proposed activity to advance science and engineering through research and education.

How important is the proposed innovation to advancing scientific and engineering knowledge and understanding?
Is the proposed plan a sound approach for establishing technical and commercial feasibility?
To what extent does the proposal suggest and explore unique or ingenious concepts or applications?
What will be the significant contribution of the project to the research and knowledge base of the field?
Is there sufficient access to resources (equipment, facilities, etc.)?
How well qualified is the team (the Principal Investigator, other key staff, consultants, and subawardees) to conduct the proposed activity?
For Phase II proposals: As a result of Phase I, did the firm succeed in providing a solid foundation for the proposed Phase II activity?

Criterion 2.

What are the broader impacts of the proposed activity? This criterion addresses the overall impact of the proposed activity.

What is the potential use of the product or process developed in the project to advance NSF goals in research and education?
What may be the commercial and societal benefits of the proposed activity?
Does the proposal lead to enabling technologies (instrumentation, software, etc.) for further discoveries?
How well does the proposed activity broaden the participation of underrepresented groups (e.g. gender, ethnicity, disability, geography, etc.)?
Does the outcome of the proposed activity lead to a marketable product or process?
Evaluate the competitive advantage of this technology vs. alternate technologies that can meet the market needs.
How well is the proposed activity positioned to attract further funding from non-SBIR sources once the SBIR project ends?

Additionally, the following factors are also considered in making an award:

Integration of Research and Education
One of the principal strategies in support of NSF's goals is to foster integration of research and education through the programs, projects, and activities it supports at academic and research institutions. These institutions provide abundant opportunities where individuals may concurrently assume responsibilities as researchers, educators, and students and where all can engage in joint efforts that infuse education with the excitment of discovery and enrich research through the diversity of learner perspectives. Principal Investigators should address this issue in their proposal to provide reviewers with the information necessary to respond fully to both NSF merit review criteria. NSF staff will give it careful consideration in making funding decisions.
Integrating Diversity into NSF Programs, Projects, and Activities
Broadening opportunities and enabling the participation of all citizens - women and men, underrepresented minorities, and persons with disabilities - is essential to the health and vitality of science and engineering. NSF is committed to this principle of diversity and deems it central to the programs, projects, and activities it considers and supports. Principal Investigators should address this issue in their proposal to provide reviewers with the information necessary to respond fully to both NSF merit review criteria. NSF staff will give it careful consideration in making funding decisions.

NSF considers that commercial potential can probably be best demonstrated by the small business concern's record of commercializing SBIR/STTR or other research, and the existence of acceptable third-phase follow-on funding commitments from private sector or non-SBIR/STTR funding sources. NSF will recognize the distinct issues faced by a start-up or young company, which does not have a track record as compared to an older, more seasoned operation. NSF also recognizes issues such as a company's ability to retain control over the products, processes, or techniques that can ultimately be developed as a result of research. However, it is incumbent upon the proposer to make a persuasive case for a significant probability of commercial success.

SBIR Phase IIB Proposals. Reference Chapter 10, Section 10.3.

4.2.2 Evaluation of Follow-On Funding Commitment(s) and Commercialization Plan.

In order to succeed in the SBIR/STTR Program, the small business must convert the research results into innovative, competitive technology. A top-notch management team is obviously a necessary requirement for this part of the process. The commercialization plan and the follow-on funding commitment play a key role in NSF's evaluation.

The NSF will consider the following points when it assesses a company's Follow-On Funding Commitment and Commercialization Plan:

The company's financial plan for commercialization:

Does the Follow-On Funding Commitment have the potential to become a serious commitment?

If actual funds are committed in Phase II, how "real" is the commitment and to what extent will the additional funds enhance the project?

If actual funds are not committed, is the company negotiating with potential partners to enable commercialization? Has this been shown to NSF in the information provided by the proposer?

The company's planned commercialization activities:

Does the innovation meet existing or emerging customer needs?

Will the proposed innovation be cost-effective?

What are the non-market benefits (e.g. social, education, or enabling scientific knowledge)?

What does the innovation provide to the Nation in the form of competitive advantages?

How realistic is the marketing plan to impact the market within five years of completion of the Phase II project.

Has the small company mapped out a strategic plan for maintaining its intellectual property?

Has the small company fully investigated a sound method of producing and bringing the innovation to market?

Has the small company successfully converted government supported research projects into commercial products?

4.2.3 Conditions of Follow-On Funding Commitments.

A few clearly defined and measurable technical objectives should be stated in the commitment agreement which, if achieved, would justify private investment. The objectives do not have to be the same as those stated in the Phase II proposal but they must be attainable within the scope of the proposed SBIR government-funded research.

A company should be in serious negotiations with potential third party investors early in the SBIR/STTR process. This can be demonstrated through letters of intent or signed commitment letters from third party investors that Phase III funds will be made available. The commitment may be contingent upon:

Receipt of the Phase II award;

Phase II achieving a few stated key technical objectives, which are agreed upon between the company and the prospective third party provider;

The resulting technology not being bypassed in the marketplace during Phase II; and/or
The potential technology continuing to appear economically viable at the end of Phase II.

Small businesses with NSF SBIR/STTR Phase II awards should continue to foster and implement third party commitments. In order to obtain the goals of the NSF SBIR/STTR Program, supplemental Phase IIB funding is available to those companies able to secure and demonstrate third party financial support. For more details on the Phase IIB option reference Chapter 10, Section 10.3

The Phase II or Phase IIB financial commitments usually must be in the form of money to the SBIR/STTR company. Commitments for a third party to spend the money directly or to provide in-kind services are not acceptable. However, instruments, computers, software, equipment, etc., provided to the proposing firm at fair market value to the SBIR/STTR company are acceptable.

Each follow-on funding commitment involving a third party should contain the following certification: "The undersigned certify that they agree to this funding commitment and that this information will be used by NSF in evaluating the commercial potential of the company's innovation and, therefore, that information will be a significant factor in determining whether the SBIR/STTR Phase II proposal will be funded. They further understand that willfully making a false statement or concealing a material fact in this commitment or any other communication submitted to the NSF is a criminal offense." (U.S. Code, Title 18, Section 1001)

A number of companies have requested a sample follow-on funding commitment letter. Although commitments are highly case dependent, two suggested formats solely for guidance may be found in Attachment H. One is for investment in your company, and one is for licensing to others. (Investment commitments are normally for the SBIR company to pursue commercialization itself, including product development, manufacturing, and marketing with private or venture capital investments from a third party). We suggest you involve your lawyer in any final agreement.

4.3 Debriefing of Unsuccessful Proposers.

When an award or declination is made, verbatim copies of reviews, excluding the names of the reviewers, summaries of review panel deliberations, if any, a description of the process by which the proposal was reviewed, and the context of the decision (such as the number of proposals and award recommendations, and information about budget availability) are mailed to the Principal Investigator. The company officer/organization representative is also notified, but that individual only receives information as to whether an award or a declination was made.

Phase I proposals that have been declined or returned by NSF are not eligible for reconsideration under the same Program Solicitation, however, they can be resubmitted after suitable revision, under subsequent Solicitations. Phase II proposals that have been declined are not eligible for resubmission.