ResearchPulse

ResearchPulse is an AI-powered framework designed to automate the generation of Research Impact Assessment Framework (RIAF) use case studies.

It uses large language models (LLMs) and a multi-source data retrieval system to analyse academic publications, web content, and institutional data to produce comprehensive reports aligned with the RIAF guidelines.

ResearchPulse directly addresses the shortcomings of traditional frameworks by implementing the RIAF, which moves beyond simple productivity metrics to consider a wider range of impact indicators.

The RIAF evaluates research across four key criteria:

• problem significance,
• research outputs,
• delivered impact, and
• future impact. 

This multifaceted approach provides a more comprehensive understanding of research impact, encompassing both academic contributions and real-world outcomes.

ResearchPulse helps researchers and funding bodies generate comprehensive use-case studies and research impact assessments.

It was developed in a partnership between the Faculty of Medicine and Health, and the Sydney Informatics Hub,

ResearchPulse employs a Retrieval-Augmented Generation (RAG) pipeline.

It retrieves data from various sources such as academic databases (PubMed, arXiv, Semantic Scholar), web content via a Bing Search API, local documents, and structured health data catalogue (AIHW, state health reports).

It then processes this information through a context engine, vector index, query engine, and review system.

Finally, it generates a comprehensive research Impact Case Study, with accompanying references, source documents and context analysis.

ResearchPulse uses RAG to ground all outputs in verifiable sources, ensuring that generated content is consistently backed by evidence.

This approach is reinforced by a multi-stage review process that validates the accuracy of all assessments.

To address potential biases, ResearchPulse integrates multiple diverse data sources and employs intelligent filtering mechanisms.

The system maintains a human-in-the-loop review process, allowing researchers to validate and customise assessment criteria based on their specific field and requirements.

Transparency and accountability are maintained through extensive process documentation and logging.

Data sources are clearly attributed, ensuring users can trace any claim to its origin.

Regarding security and user control, ResearchPulse operates within strict network security restrictions and processes only publicly available research data. 

The tool runs on secure University of Sydney infrastructure, and is currently only available to university staff and affiliates. 

ResearchPulse recognises the importance of context in impact assessment, and follows the RIAF domains and sub-domains for developing impact case studies.

The platform’s Context AI Agent analyses the research problem, identifies missing information, and searches for complementary evidence.

This contextual analysis allows for a more nuanced understanding of research impact, considering factors such as:

• Research environment: The research environment within NSW’s health and medical research ecosystem, considering factors that may influence research outcomes.
  
  
• Problem significance and urgency: The critical nature of the research problem, its scale, and the potential consequences of inaction, providing context for impact assessment.
  
  
• Knowledge generation metrics: Assessment of novel intellectual property, methodological innovations, and contributions to the field's theoretical foundations.
  
  
• Capacity development impact: Analysis of workforce development, skill enhancement, and research capability building within the healthcare sector.
  
  
• Healthcare technology advancement: Evaluation of technological innovations, medical device development, and digital health solutions that enhance patient care.
  
  
• Health system integration: Assessment of research impact on healthcare delivery, system efficiency, and service quality improvements.
  
  
• Population health indicators: Analysis of broader public health outcomes, disease prevention effectiveness, and community health improvements.
  
  
• Economic value creation: Evaluation of commercial opportunities, cost-effectiveness, and economic benefits to the healthcare system.
  
  
• Collaborative strength: Assessment of research partnerships, network development, and institutional collaborations that enhance impact.
  
  
• Policy influence: Analysis of research contribution to policy formation, clinical guidelines, and public health recommendations.

ResearchPulse is a useful tool whether you’re preparing a funding application, documenting research impact, creating use-case studies, or assessing research outcomes.

The tool provides automated, evidence-based analysis while maintaining academic rigour and relevance. Additionally, it is also:

• Time-efficient: the AI tool is able to draft a case study within minutes, that would take a researcher or medical writer 20+ hours to complete
  
  
• Evidence-based: ResearchPulse can automatically gather and validate evidence from multiple sources. All references are included in the case study and can be verified
  
  
• Customisable: Users can adapt the assessment criteria to a specific research program, tailor the impact date range, and change the style of impact case study as needed