Start Submission Become a Reviewer

Reading: High-Risk, High-Impact Health Services Research: Developing an Innovation Initiative

Download

A- A+
Alt. Display

Program Profile

High-Risk, High-Impact Health Services Research: Developing an Innovation Initiative

Authors:

Jolie N. Haun,

James A. Haley Veterans Hospital, US
X close

Naomi Tomoyasu,

Veterans Health Administration, US
X close

Courtney Paolicelli,

Veterans Health Adminstration, US
X close

Christine Melillo ,

James A. Haley Veterans Hospital, US
X close

David Atkins

Veterans Health Administration, US
X close

Abstract

As health systems strive towards innovative solutions to healthcare problems, research programs should be a crucial resource to support the evidence base of innovation. Unfortunately, current research funding processes and infrastructure often produce incremental improvements over an extended timeline. Within the Veterans Health Administration (VHA), the Health Services Research & Development (HSR&D) service used the need for greater innovation to address healthcare problems to examine possible changes in how research is solicited, reviewed, funded, and implemented. Towards this goal, HSR&D has launched the “Innovation Initiative” which includes a new funding mechanism that supports research that has potential for high impact. The aim of this paper is to describe this new funding mechanism for supporting scientific innovation and impact in health services research.
How to Cite: Haun, J.N., Tomoyasu, N., Paolicelli, C., Melillo, C. and Atkins, D., 2020. High-Risk, High-Impact Health Services Research: Developing an Innovation Initiative. Journal of Veterans Studies, 6(1), pp.171–177. DOI: http://doi.org/10.21061/jvs.v6i1.169
29
Views
4
Downloads
1
Twitter
  Published on 18 May 2020
 Accepted on 17 Mar 2020            Submitted on 13 Jan 2020

Necessity is the Mother of Innovation Project Background

New medical products and interventions are increasingly based on scientific knowledge generated through research (Ahmadpoor & Jones, 2017). Researchers depend more and more upon federally funded research (as opposed to industry funded research) as a source of scientific knowledge (Fleming et al., 2019). Since World War II, federally funded research has increasingly generated new scientific knowledge that improves our lives, especially our medical care (Fleming et al., 2019). Though federally funded research is important for bringing advances to healthcare, it is often time consuming and difficult to fund new ideas that challenge our beliefs and processes-- innovative ideas.

Applying for federal research funding is typically a conservative process which often discourages bold ideas or risks in research methods. Review panels scrutinize methods to ensure rigor and usually require preliminary data to ensure the research investment is highly likely to succeed. With revisions and resubmissions, up to two years can elapse before an innovative idea is tested. This process creates barriers and disincentives to scientists pursuing new ideas and innovative approaches (Bindman et al., 2018; Sensmeier, 2012). If research is to contribute to innovative solutions, then the scientific community needs to embrace ideas that challenge current research or practice paradigms by using novel concepts, approaches, or interventions. For example, a study engaging local firing ranges to teach suicide prevention would involve supporting the use of firearms and acknowledging this is a part of military and veteran culture. This may be “too risky” under traditional mechanisms but would be embraced by more risk tolerant mechanisms.

The Veterans Health Administration (VHA) Health Services Research & Development (HSR&D) program reviews and funds proposals based on assessing the importance and innovative nature of the research questions, as well as the feasibility and quality of both the study design and research team (Atkins et al., 2018). Although specific review criteria exist (e.g., scientific merit, potential to improve health care), there are no clear criteria to assess a proposal’s innovation. As a result, peer reviewers often give more weight to easily assessed criteria (e.g., study team experience and rigor of prior research and study design) rather than innovation in the research question or methods.

The work involved in preparing an investigator-initiated research proposal, which usually involves up to two resubmission rounds, may deter investigators from submitting proposals that use novel interventions or methods such as applying proven interventions from one disciplines to a new discipline (Atkins et al., 2018). Additionally, successful funding of an application is limited, pushing researchers to invest their time and submit the application they feel is most likely to be funded (“HSR&D Merit Review Success Rates,” 2018). As a result, research ideas pursuing a well-established pathway and building incrementally on current knowledge are more often submitted and funded. Ideas that break from traditional approaches, draw from different disciplines, have limited early data, or appear to pose a higher risk of failure are less likely to be submitted and funded.

Other barriers may also impede funding innovative ideas. These include a limited number of funding mechanisms, more weight given to traditional perspectives by established reviewers, and extensive three to four year funding cycles. These obstacles may impede the ability of health services’ research to produce meaningful clinical care improvements in a rapidly evolving learning healthcare system (Bindman et al., 2018; Riley et al., 2013; Sensmeier, 2012). If research is to claim a central role in generating “learning” and new ideas to sustain VHA as a learning healthcare system, then research organizations, such as HSR&D need to build capacity to rapidly adopt innovation without abandoning rigorous evaluation (Asch & Rosin, 2015; Bindman et al., 2018; Lyon et al., 2015; Lyon & Koerner, n.d.). Study designs often submitted for funding (e.g., incremental advances in protocol aims and hypotheses tested) are not sufficiently flexible to support innovative interventions that require early adaptation in dynamic and pragmatic healthcare settings. These issues warrant funding mechanisms that support innovative and impactful studies designed to respond to current healthcare issues.

This paper presents a brief overview of a quality improvement project conducted by HSR&D leadership and innovation experts. The project aimed to rigorously identify recommendations for a new funding process to effectively support innovative, high-risk, high-impact research. This research will contribute to meaningful real-world transformations in veteran care and health outcomes. These transformations will be foundational to substantial contributions to health services research.

Project Structure and Procedures

The project team used a mixed-methods approach (Creswell, 2007) with representative and snowball sampling. The team collected and collated data for current funding mechanisms for innovative, high-risk, high- impact studies. Data were collected using a focus group (n = 5), individual interviews (n = 10), and follow-up interviews (n = 3) to allow a breadth of data on relevant topic areas. The focus group participants then described current funding mechanisms and highlighted how innovation was expressed in those mechanisms. Next, we conducted semi-structured interviews both in-person and by telephone with remaining participants. These interviews asked participants to define innovation, describe current funding mechanisms, and highlight how innovation was expressed in those mechanisms. Participants included Department of Veterans Affairs (VA) scientists (n = 12), innovation experts outside of the VA (n = 9), VHA HSR&D program administrators and managers (n = 7), HSR&D representatives (n = 4), VA central office administrators (n = 3), National Institutes of Health program administrators (n = 2), a VA Center for Innovation representative (n = 1), and a VA Diffusion of Excellence Program representative (n = 1). Although this project was not designed as a research project requiring an institutional review board submission, we made every effort to provide confidentiality to all project participants.

Thematic analysis was used to analyze data collected from the focus groups and interviews. Thematic analysis is an “iterative and reflective process that develops over time and involves a constant moving back and forward between phases” (Lorelli Nowell S. et al., 2017). The six phases are a) familiarizing yourself with the data; b) generating initial codes; c) searching for themes; d) reviewing themes; e) defining and naming themes; f) producing the report (Braun & Clarke, 2008). One team member conducted data collection and the initial generation and search for themes, and as a team, we reviewed and defined themes and then jointly produced a report of the findings.

We conducted an environmental scan to identify documents (e.g., publications, correspondence, funding announcements) relevant to innovation in research in the current climate. Search terms included “innovation”, “what is innovation”, “adopting innovations”, “innovation implementation”, “disruptive innovation”, “innovation in healthcare”, and “innovation sustainability”. Search engines included PubMed, OVID MEDLINE, Cochrane Systematic Reviews, Google Scholar, and the Joanna Briggs Institute Evidence Based Practice Database. We reviewed resultant titles and abstracts for appropriateness of content. This review generated a subsequent grouping of literature to be further reviewed for this project. We excluded literature speaking to non-research innovation and published ten or more years previous to time of review. We believed articles older than ten years would not capture the intent of this project phase which was to define innovation within the research context. Published books were also examined using Google and Amazon, from which a collection of representative texts was selected. Data collected and collated from these sources were triangulated and ultimately produced the following findings and recommendations.

Project Results

This project developed a current definition of innovation within the VA research context. That definition is further explained in the following sections. We also discovered how innovation is used within various funding mechanisms. Innovation and its use were used to develop the new HSR&D Innovation Initiative.

Defining Innovation

Based on the environmental scan as well as interview data, we identified the following three themes relating to innovation:

  • Innovation is often defined as novel—something new and different--a new product, process, approach, and/or idea.
  • Innovation should have a positive impact (Anthony, 2017). “Impact” could be defined as societal impact, affecting a sizable portion of a population, improving quality of life, having positive financial results, and/or improving systems-level efficiency and performance.
  • Innovation involves risk; sometimes even a quantum leap that presents a greater risk of failure. This differs significantly from the incremental approach traditionally used in research (Poirier et al., 2017).

We understand innovation to be a complex construct with multiple dimensions. This understanding is key to identifying and funding innovative studies. These three contingent characteristics combined (novel, positive impact, risk) work synergistically to set innovation apart from its component characteristics. To reiterate, it is the combination of these characteristics that generates innovation.

Common Features of Funding Mechanisms that Encourage Innovation

In addition to identifying characteristics of innovation, focus groups and interviews with the experts described above examined best practices in grant review and funding decision making from previous VA and non-VA funding mechanisms. Mechanisms contained many of the following features to support funding of innovative and impactful research, though few used all recommended features.

Create a Specialized Review Panel and Process

A specialized review panel is essential to identify and fund innovative, impactful research. Traditionally, reviewers and funders focus on potential risks as funds are perceived as a limited resource. Experts suggest a specialized and diverse panel should consist of reviewers who are sufficiently flexible and forward-thinking with regard to a big picture view of research. These reviewers should also permit greater risk in funding innovative, impactful health services research. Lastly, the pool of reviewers must represent individuals from diverse fields, including operations.

Increase the Role of the Funder

Several administrators reported using a cooperative agreement model to ensure operational partners, researchers, and the funder were actively engaged in the development and implementation of the project. This model allows all stakeholders to be engaged throughout the research process. Additionally, this method for engagement allows for seamless communication and cooperation that facilitate goal setting, study monitoring, and efficiency in troubleshooting.

Provide Phased Funding

Phased funding allows a fail fast, fail often approach by which funds can be allocated in phases to accommodate the complex nature of innovative research. A phased approach permits funds to be dispersed incrementally to: a) assess the innovative concept separately from proof of concept; b) test the feasibility and operational practicality of the innovative idea; and c) account for the funds invested. When using a phased approach, co-monitoring by investigators and funders is essential.

Establish Pre-defined Benchmarks

When co-developed by the investigator, operational partners, and funder, pre-defined benchmarks facilitate a smooth process for monitoring the progress of a project and making decisions that enhance the success of the project. Specifically, pre-defined benchmarks adjust funding to accommodate unexpected needs. This cooperative agreement approach permits modifications while maintaining scientific rigor.

Provide Successive Opportunities for Additional Funding Based on Success in Meeting Milestones

In their final year of funding, allowing successful projects to compete for transition funds can help facilitate implementation into practice. At the same time, there may be other mechanisms beyond traditional research funded that take innovative ideas to scale (e.g., partnerships with industry or health systems).

Consolidating all the findings from our project, we developed the HSR&D “Innovation Initiative” program. Using a specialized panel of reviewers, this new funding mechanism is intended to adapt to the complex nature of innovative research, establish cooperative agreements between funder and researcher, and ultimately expedite implementation of research into practice.

HSR&D Innovation Initiative Program Description

The new HSR&D Innovation Initiative program is an evidence-based, funding mechanism that employs innovation and a specialized application and review process.

Innovation. The request for applications emphasizes four specific features to distinguish high-impact, innovative studies: (a) Novelty: Applicants will be asked to show how their approach differs substantially from past approaches; (b) Potential for significant positive impact: Applicants will be asked to demonstrate why new approaches offer better chances of meaningful impacts; (c) Risk: While risk is not itself a desired feature, the solicitation phase of our study makes clear that in pursuit of greater innovation, we were willing to tolerate a higher risk of failure; and (d) Accountability: Proposals must specify how researchers will use the planning period to show that they could overcome the greatest challenges of their approach--i.e. to “de-risk” their innovation. To promote iteration and recalibration as both the funder and investigator explore the proposed innovation, proposals need to include a plan for regularly reassessing progress and the likelihood of success.

Specialized Application and Review Process. We therefore modified our usual review process in the following ways to align with recommended approaches for encouraging innovation. Solicitation and portions of the review were done off cycle from our usual semi-annual research solicitations. This purposive timing allowed us to differentiate this initiative from our regular solicitations, thus emphasizing uniqueness of the initiative, encouraging new applicants, and creating a break from culture of our traditional review. Applications were limited to three pages in order to encourage teams and reviewers to focus on interesting ideas rather than detailed research plans. Review criteria and scoring were simplified to emphasize only two factors--evel of innovation and potential for impact. Panels contained a higher proportion of outside (non-VA) reviewers and more interdisciplinary expertise including subject matter experts, experts in innovation, methodologists, and end users (e.g. veterans, clinicians, administrators). Applications were blinded to give greater emphasis to the proposed idea than to the individual or institution.

Given the unique nature of this initiative, we sought to create a review panel that was diverse in terms of research, operational, and administrative experience. We selected individuals from across the United States who had backgrounds ranging from technology and start-ups to military healthcare and research administration. Criteria for selection included expertise in the content areas of submitted grants as well as expertise in innovation (e.g., previous innovative work, publications and review panels). Panelists were both VA and non-VA experts. To prepare the review panel, we conducted a synchronous, virtual training to orient reviewers to the goal of the initiative. Reviewers used specific evaluation criteria that were unique to the goals of our “request for application” (RFA) The two critical elements evaluated for each concept proposals were innovation and impact.

Innovativeness of the Proposed Research and Strategies to Address the Questions Related to the Innovation. Reviewers consider whether the proposed work:

  • Challenges or seeks to shift current research or clinical practice paradigms by using novel theoretical concepts, approaches or methodologies, or interventions;
  • Involves the creation of a new, original, or unusual process or product or the adoption of an existing process or product in a new or unusual way that transforms current approaches;
  • Contributes to an area of practice or science that is ready for change;
  • Focuses on broader VA healthcare system rather than within a specific subpopulation;
  • Addresses a significant problem in terms of its prevalence, severity, urgency etc., for the VA and the general public.

Impact of the Proposed Innovation Plan. Reviewers consider whether the proposed work has high potential impact that results in major transformation in healthcare delivery system. The level of impact is based on whether the proposed innovation has the potential to:

  • Dramatically impact VA clinical practice, health policies, and/or advance the field of health services research, or
  • Impact the existing VHA healthcare infrastructure and/or veterans’ quality of health.

Approach to Reviewing and Funding Proposals

The HSR&D Innovation Initiative consists of two phases. Phase 1 is a planning phase wherein proof of concept is tested, and phase 2 is a more traditional, research-testing phase. HSR&D solicited applications for phase 1 in late 2018 (see Appendix 1). Proposals underwent a double-blinded review (i.e. reviewers and funders did not know the applicants’ identity), and the top proposals were discussed during an in-person review panel in early 2019. In spring 2019, ten projects were awarded phase 1 funding. This program is iterative and requires response to reviewer and researcher experiences. As such, subsequent RFAs have been revised to address earlier reviewer and researcher experiences (see Appendix 2).

Only projects that received phase 1 funds will be able to compete for phase 2 funding. Successful phase 1 projects may use their initial funding to develop the full proposal for phase 2, which will be submitted approximately ten months after the phase 1 award was made. Unlike phase 1, the review process for phase 2 will not be blinded. Phase 2 projects will receive four to five years of funding to complete a study in order to test the studied innovation with health outcomes. The timeline for both phases is illustrated in Table 1.

Table 1

Characteristics of Innovation Initiative Phased Funding.

Phase 1 Phase 2

Application Requirements 3-page concept paper Full 15-page proposal
Number of Applications Funded Up to 10 Up to 3
Funding Duration Up to 18 months Up to 5 years
Award Amount Up to $200,000 per proposal Up to $500,000 per year
Progress Evaluation Progress assessed using the timelines and benchmarks developed by the awardee and approved by HSR&D Progress assessed using the timelines and benchmarks agreed upon by the investigator, HSR&D, and partner

Inaugural Innovation Initiative Funding Results

We released the final RFA for the Innovation Initiative on September 19, 2018 (Appendix 1). A total of 121 applications were submitted across the five priority areas. Due to the high number of applications, only the top third based on their preliminary scores were discussed in the review panels. This allowed the full panel to review all of the concept papers, rather than relying on the opinions of lead reviewers. The greater engagement of all panel members on all proposals allowed them to reach a shared understanding of what they were looking for in the best proposals. Ultimately, ten proposals were funded; two from each priority area. Awarded projects demonstrated innovation in at least one of four categories (a) new use of data sources not previously utilized in VA research; (b) new policies to change established practice within the organization; (c) new partnerships between groups within and outside the VA for a shared purpose; and (d) new use of technology not yet established in the VA. Descriptions of several funded projects are illustrated in Table 2.

Table 2

Examples of Funded Projects, Their Purpose, Innovation and Impact.

Innovation Project Title Project Purpose, Innovation, and Impact

Use of data sources Reduction of Postoperative Opioids Dispensed in Surgical Patients Purpose: Develop and implement a single “procedure agnostic” postoperative opioid prescribing protocol that can be used across surgeries performed in the VA system.
Innovation: Development and implementation of a “procedure agnostic” postoperative opioid prescription protocol using opioid prescription and refill data, emergency department visits, and readmissions for surgical pain or complications.
Potential Impact: (a) Reduction of opioids dispensed post-surgery; (b) reduction in attendant harms of overprescribing opioids; (c) ability to implement for all surgical procedures across VA system.
New policy and partnerships Does Protecting Service-Connected Disability Income Motivate Return to Work in Veterans with TBI and PTSD? Purpose: Determining if protection of disability benefits is associated with return to work or increase in work hours among veterans with Traumatic Brain Injury (TBI) and Post Traumatic Stress Disorder (PTSD).
Innovation: Development of intra-agency (e.g. the Veterans Health Administration and the Veterans Benefits Administration) and interagency (e.g. the Social Security Administration and the Internal Revenue Service) partnerships and new policies to ensure disability benefits protection for veterans with TBI and PTSD, while allowing them to maintain employment-based income.
Potential Impact: (a) Eliminate veterans’ risk of losing financial safety net and motivate veterans to more fully engage in recovery; (b) increase in veterans’ work hours and income over and above their disability compensation, while promoting positive benefits of employment for veterans and contributing to the greater economy.
New partnerships Development and Evaluation of a Veteran-Informed Means Restriction Intervention for Suicide Prevention Purpose: Improving acceptance and uptake of means restriction counseling (i.e. approaches to prevent at-risk veterans from accessing firearms) to reduce suicides by firearms.
Innovation: Public health approach to change the culture around gun safety messaging by collaborating with gun sellers and owners to deliver means restriction counseling and materials in community-based settings.
Potential Impact: (a) Establish a network of trained individuals in restriction counseling and safety messaging; (b) equip stakeholders with access to means restriction programs and materials; (c) engender trust and mutual understanding between VA and community-based gun safety stakeholders; (d) reduce suicide by firearms.
Use of data sources and technology Patient Incentives for Reducing No-shows, Accommodating Walk-in Visits, and Improving Primary Care Workflow Purpose: Developing predictive models and implementing incentives to improve primary care.
Innovation: Use of machine learning techniques to predict no-shows and combining predictive models with targeted social incentives based on behavioral economic principles.
Potential Impact: (a) Increase efficient use of existing clinical resources; (b) increase access to services for veterans in need; (c) promote cost containment; (d) increase the use of technology to provide access to care.
New use of technology Mobile App for the Prevention of Suicide Purpose: Develop and create a framework for implementation and dissemination of a suicide prevention mobile app.
Innovation: (a) Personalized mobile intervention to support at-risk veterans and allow clinicians to monitor changes in patient risk over time; (b) facilitate patient/provider communication to reduce suicide risk and identify periods of increased risk.
Potential Impact: (a) Provide veterans with a way to identify and address suicide risk; (b) identify periods of acute need to supplement current care; (c) provide resource to veterans in and outside the healthcare system.

Lessons learned: Challenges and Solutions in Funding Innovation

The first-round initiative shed light on challenges and possible solutions. Initiatives with a non-traditional application require training to acclimate reviewers to the program expectations. Despite efforts to create explicit definitions of innovation, judgments are inherently subjective and reviewers’ scores reflected this variation. Non-traditional review panels produce non-traditional reviews. While panels generally reached consensus on the best proposals, feedback provided to unsuccessful applications was more abbreviated than the detail provided in our usual review process. The line between “high-risk” and “impossible” is subjective. Some reviewers found ideas to be intriguing, but felt they could not succeed in the VA, either because of policy or resource issues.

Not all innovative ideas fit this solicitation. Some applicants proposed interesting and innovative approaches, but because they had sufficient preliminary data to support their ideas, it was felt that their proposal could fit comfortably in our traditional application process (i.e. they did not need phase 1 funding to show their idea was viable). To continue championing innovation in health services research, HSR&D plans to publish another Innovation Initiative solicitation. Based on our recent experience, though, we do plan to adjust our process in the following ways:

  • provide additional guidance to reviewers through virtual trainings and written instruction;
  • provide additional guidance to investigators to focus proposals more on innovation and impact and less on methods and aims;
  • require reviewers to assign numeric scores to the two criteria as opposed to one overall score;
  • expand the page limit beyond three pages to allow investigators to thoroughly articulate their ideas;
  • allow reviewers to view references as part of the review process.

Implications for Future Research

Improvements in healthcare require a combination of incremental advances building on established knowledge and the ability to test bold ideas that might offer faster progress. For research funders, encouraging new ways of thinking requires new models for review and funding. Our early experience with the Innovation Initiative, and the unprecedented response to it, suggest there is a strong, previously untapped interest in innovative health services research. We will learn much more over the next eighteen months about which ideas succeed or fail and whether our planning period is sufficient. The ultimate proof of this concept is years away when we will learn whether a sufficient number of these ideas advance to definitive testing and result in improved healthcare for veterans. Nonetheless, our experience suggests that a new process can successfully serve a range of creative ideas that might never have been submitted or supported through traditional review processes.

Additional Files

The additional files for this article can be found as follows:

Appendix 1

HSR&D Innovation Initiative Program (Planning/Start Up Funds) 2018. DOI: https://doi.org/10.21061/jvs.v6i1.169.s1

Appendix 2

HSR&D Innovation Initiative Program (Planning/Start Up Funds) 2019. DOI: https://doi.org/10.21061/jvs.v6i1.169.s2

Competing Interests

JNH acted as a member on the initial Reviewer Panel. NT is Deputy Director of HSR&D. CP has no competing interests. CM has no competing interests. DA is Director of HSR&D.

References

  1. Ahmadpoor, M., & Jones, B. F. (2017). The dual frontier: Patented inventions and prior scientific advance. Science, 357(6351), 583–587. DOI: https://doi.org/10.1126/science.aam9527 

  2. Anthony, S. D. (2017). The little black book of innovation, with a new preface: How it works, how to do it. Harvard Business Review Press. ISBN-13: 978-1633693401 

  3. Asch, D. A., & Rosin, R. (2015). Innovation as discipline, not fad. New England Journal of Medicine, 373(7), 592–594. DOI: https://doi.org/10.1056/NEJMp1506311 

  4. Atkins, D., Tomoyasu, N., & Paolicelli, C. (2018). Innovation initiative program RFA: High risk, high impact health services research. VA HSR&D Cyberseminar, Virtual. https://www.hsrd.research.va.gov/for_researchers/cyber_seminars/archives/3535-notes.pdf 

  5. Bindman, A., Pronovost, P., & Asch, D. (2018). Funding innovation in a learning health care System. JAMA, 319(2), 119–120. DOI: https://doi.org/10.1001/jama.2017.18205 

  6. Braun, V., & Clarke, V. (2008). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. DOI: https://doi.org/10.1191/1478088706qp063oa 

  7. Creswell, J. (2007). Designing and conducting mixed methods research. SAGE Publications. 

  8. Fleming, L., Greene, H., Li, G., Marx, M., & Yao, D. (2019). Government-funded research increasingly fuels innovation. Science, 364(6446), 1139–1141. DOI: https://doi.org/10.1126/science.aaw2373 

  9. HSR&D Merit Review Success Rates. (2018, February 5). [Government]. Health Services Research & Development. https://www.hsrd.research.va.gov/for_researchers/merit_review/success_rates.cfm 

  10. Lyon, A. R., & Koerner, K. (n.d.). User-centered design for psychosocial intervention development and implementation. Clinical Psychology: Science and Practice, 23(2), 180–200. DOI: https://doi.org/10.1111/cpsp.12154 

  11. Lyon, A. R., Lewis, C. C., Melvin, A., Boyd, M., Nicodimos, S., Liu, F., & Jungbluth, N. (2015). Health Information Technologies—Academic and Commercial Evaluation (HIT-ACE) methodology: Description and application to clinical feedback systems. Implementation Science, 11(1), 128. DOI: https://doi.org/10.1186/s13012-016-0495-2 

  12. Nowell, L. S., Norris, J. M., White, D. E., & Moules, N. J. (2017). Thematic analysis: Striving to meet the trustworthiness criteria. International Journal of Qualitative Methods, 16, 1–13. DOI: https://doi.org/10.1177/1609406917733847 

  13. Poirier, V., Schwartz, L. H., Eddy, D., Berman, R., Chacour, S., Wynne, J. J., Cavanaugh, W., Martin, D. F., Byrne, R., & Sanberg, P. R. (2017). Thoughts on improving innovation: What are the characteristics of innovation and how do we cultivate them? Technology & Innovation, 18(4), 319–330. DOI: https://doi.org/10.21300/18.4.2017.319 

  14. Riley, W. T., Glasgow, R. E., Etheredge, L., & Abernethy, A. P. (2013). Rapid, responsive, relevant (R3) research: A call for a rapid learning health research enterprise. Clinical and Translational Medicine, 2(1), 1. DOI: https://doi.org/10.1186/2001-1326-2-10 

  15. Sensmeier, J. E. (2012). Disruptive innovation and the changing face of healthcare. Nursing Management, 43(11), 13–14. DOI: https://doi.org/10.1097/01.NUMA.0000421681.71712.86 

comments powered by Disqus