EbD Metrics Framework

Users of the metrics framework will likely differ in the level of detail with which they wish to engage. Therefore, the qualitative metrics have been separated into three successive levels of specificity: STRATEGIC, TACTICAL, and OPERATIONAL. Please note that that all suggested qualitative metrics are guideposts and are intentionally non-specific to any given stakeholder within the clinical research enterprise. Users of the metrics framework can adapt them to the specific nature of their organization and work.

  • STRATEGIC: Some users may wish to look only at the 3-4 strategic “must have” qualitative metrics per theme, which are in bold grey font, and designated by the theme letter and strategic metric number (e.g., D.1).
  • TACTICAL: Other users may wish to go beyond the strategic level “must have” qualitative metrics and look additionally at the 2-3 suggested tactical level qualitative metrics. These are listed under each strategic metric with the theme letter, the strategic metric number, and the tactical metric number (e.g., D.1.1).
  • OPERATIONAL: Operational level metrics are not listed directly under the tactical metrics, because it is anticipated that users who wish to consider in more detail how to measure efforts on the tactical qualitative metrics would create their own operational level metrics. However, to assist those wishing to develop operational level metrics, approach examples in the format of logic model flowcharts have been included on the Qualitative Operational Measures page. These are available for 1-2 tactical level metrics for each theme and can be accessed by clicking on the “(see operational example…)” that is in blue underlined font next to the tactical measure, or by going to the Qualitative Operational Measures page.

A

Commitment, leadership, and resources to support DEI in CR (“Commitment”)

Theme A focuses on leadership actions to make Diversity, Equity, and Inclusion (DEI) a priority within and across institutions/organizations. It encompasses processes such as creating a statement of commitment and clearly communicating that statement, and institutional support for commitment to DEI, both internally and externally. Theme A includes actions to set the stage for institutional accountability to DEI in Clinical Research (CR), including partnering with external advisors; designating and empowering institutional teams, committees, and positions to lead work in DEI; and committing to evaluation of institutional performance in DEI. Please note: the conception of metrics in A.2 relates to institutional (or study) performance and are reflected in the proposed quantitative and qualitative measures in this framework. However, the conception of metrics in A.3 relates to evaluating individual performance toward specific tasks involving DEI and, as they are specific to the individual, are beyond the scope of this document and are not enumerated here. Finally, Theme A also involves the generation or re-allocation of resources to support leadership, partner, and team commitment to DEI in CR.

  1. Internal communication of commitment to DEI in CR
  2. Public communication of commitment to DEI (see operational example A.1.2)
  1. Assessments of the institution’s financial, human, and physical resources to support DEI in CR
  2. Processes to generate/re-allocate financial, human, and physical resources to support DEI in CR
  3. Criteria for vendor selection and contracting to support DEI in CR (see operational example A.2.3)
  1. Institutional/organizational partnerships with external advisors that coordinate or conduct CR
  2. Processes for monitoring of institutional performance for DEI in CR
  1. Dedicated teams, committees, and positions for DEI (in CR and departments impacting CR)
  2. Demarcated roles, tasks, and supervisory/reporting structures for teams, committees and positions for DEI
  3. Processes for monitoring of team, committee, and individual position performance for DEI

B

Institutional and study metrics for DEI in CR, data collection, and analysis (“Data”)

Theme B centers on the processes for setting up metrics and data collection systems to evaluate institutional/organizational and/or study (i.e., clinical trial) performance on DEI in CR. This involves consideration of the purpose(s) for the metrics and the question(s) that are driving data collection and analysis, in addition to which variables, data sources, and forms of analysis would be needed to complete the analysis. Although some quantitative and qualitative metrics at the institutional level and study level may not always align, DEI success at the institutional and at the study level are interdependent. Therefore, the qualitative measures in Theme B attempt to be sufficiently broad to capture both institutional and study processes for developing metrics and data collection/analysis structures, but also specific enough so that the contextualization necessary for studies is not overlooked. Please note: Theme B does not yet include quantitative metrics. Qualitative metrics on analyses of data or review of results to plan for corrective action are covered in Theme F (Review).

  1. Statement of purpose to capture, measure, and report DEI metrics in clinical trials from the President, Executive Leadership Team, and Board
  2. Training, resources, and capacity to support the consideration, selection, and reporting of metrics
  3. Defined purpose(s) for the metrics, targets for success, and timeframes to assess the metrics (see operational example B.1.3)
  1. Institutional and/or study question(s) defined in terms of the epidemiology of the condition(s) of concern and the population(s) of concern
  2. Map of existing internal and external data sources that collect data from participants, patient records, electronic systems, and/or research records and the flow of data within the data infrastructure
  3. Data source audit to identify data variables (e.g. race) and values (e.g. White, Black or African American, Asian) that can be utilized to meet data needs for the defined questions
  1. Data dictionary that aligns terminology and formatting for DEI metrics, variables, and values to be utilized, in compliance with the latest regulatory guidelines
  2. Standardized data collection and reporting practices in place for departments or site(s)
  3. Analytic methodologies to optimally answer the institutional or study question(s) with available data sources, variables, and values

C

360-degree partnerships, communication, and engagement (“Engagement”)

Theme C begins with recognizing that partnership with the community is critically important at the outset of clinical research and leads to the co-development of communication and engagement activities. Planning for respectful communication involves the actions to understand how internal and external audiences wish to be engaged. CR institutions/organizations can then develop necessary plans for physical, cultural, and linguistic accessibility (e.g., translation) of preferred modes of communication and participant-facing study materials. Theme C also encompasses processes to understand the lived experiences of individuals working and participating in clinical research and to build trust for initial and sustained engagement with community partners. Please note: Partnerships with external advisors who coordinate or conduct clinical trials are covered under Theme A (Commitment). Mentorship and career development is included in Theme D (Workforce). In addition, while it is important to prepare for 360-degree Partnerships, Communication and Engagement through clinical research staff training in areas such as implicit bias and cultural competency, all clinical research training is considered under Theme D (Workforce). Review processes and accountability measures, including community review of partnership sustainability, are covered in Theme F (Review).

  1. Defined process(es) for partnership with community sites, community groups, and their trusted members
  2. Plan/s for community partner relationships to be supported and sustained
  1. Terminology/imagery and communication that respect the preferences of intended audiences
  2. Communication platforms and materials that are physically accessible and usable for intended populations (see operational example C.2.2)
  3. Communication platforms and materials that are culturally and linguistically accessible and comprehensible to intended populations
  1. Internal: Lived experiences and related perspectives on clinical research are heard and understood
  2. External: Lived experiences and related perspectives on clinical research are heard and understood
  3. External: Connections fostered with community sites, trusted community members and potential trial participants

D

Educational and career opportunities to support DEI in CR (“Workforce”)

Theme D focuses on actions to recruit and retain a diverse workforce that is representative of participant populations. It begins with recruitment, pipeline, and cohort programs for underrepresented clinical research staff, then moves to training and mentorships to support new hires. Finally, workforce retention also involves workplace environments, benefits, resources and support plans that enable all employees to remain physically and mentally healthy, performing at their best, and feeling valued. Please note: The term “clinical research staff” is not limited in scope to principal investigators and research coordinators, but also encompasses research nurses/assistants/monitors, patient navigators, pharmacists, medical writers, IRB chairs/members, scientists, technicians, regulatory professionals, etc. The term “equitable opportunities” implies equitable availability of, accessibility to, consideration for, and functioning of programs that will appropriately meet the training needs and learning styles of diverse clinical research staff. “Targeted opportunities” are those that specifically support underrepresented clinical research staff in surmounting structural barriers to participation.

  1. Equitable and targeted opportunities to access institutionally supported pipeline/recruitment/cohort programs such as CR leadership development, internships/fellowships, or CR workforce training programs
  2. Vacancies for professional/volunteer positions in CR accessible to diverse audiences and applications fairly assessed
  1. Equitable and targeted opportunities for training in clinical research, grant applications, and professional development (see operational example D.2.1)
  2. Equitable and targeted opportunities for mentorship programs, training partnerships, or networking
  1. Required training for all employees on cultural humility, implicit bias, and accessibility by design
  2. Equitable and targeted benefits, flexible work policies, and opportunities to create/participate in support groups
  3. Retention strategies, individualized plans, and support available through human resources

E

Strategies for inclusion of diverse participant populations in CR (“Inclusion strategies”)

Unlike the other themes in this framework that could be applicable to institutional/organizational, partnership, project, or study processes, Theme E is specific only to study processes. The measures under Theme E act in a way as a “backstop” to the measures in the other themes. That is because the measures in Theme E check that DEI recommendations have been included and specified throughout study plans, communication plans, and participant-facing materials. Please note: The qualitative measures in Theme E assume that implementation will follow in the manner specified in the study plans, and therefore separate measures are not listed for implementation. However, the quantitative measures in Theme E, and the qualitative measures in Theme F (Review) and Theme G (Dissemination), can be utilized to assess implementation of strategies for inclusion of diverse participant populations in the clinical research.

  1. Study question(s) identified based on relevance for intended populations
  2. Study designed to consider inclusion of intended populations (e.g., language, accessibility, frequency of visits, etc.)
  3. Eligibility criteria drafted for inclusivity (see operational example E.1.3)
  1. Feasibility assessments and selection of sites conducted with attention to DEI
  2. Supports to facilitate accessibility to clinical research by people of all abilities
  1. Planning documents, including overall study and site-specific communication plans, communication platforms, and participant-facing materials are respectful and accessible
  2. Recruitment and retention plans that have been developed through community engagement and consider participants’ time, resources, and family/life circumstances

F

Review, accountability, and course adjustment (“Review”)

Theme F involves actions to clean and analyze data (both quantitative and qualitative), and to review methodologies and results and of data analysis. Theme F enables stakeholders to begin drawing a picture of how well the institution/organization, team, or study progressed with their DEI efforts in CR and to evaluate what worked well and/or needs improvement. Collating and assessing all lessons learned will subsequently inform decision-making for course adjustment. This [institutional or study] course adjustment presents an opportunity to revise metrics and performance criteria, policies, tools, and to create mechanisms to hold institutions/staff responsible for outcomes. Please note: Theme F does not yet include quantitative metrics. In addition, as in other themes, careful consideration of all perspectives and forms of expertise necessary for these processes, and thus whom to involve, will be important. For example, the people ideally responsible for cleaning and analyzing data may not be the same as those who review results, generate lessons learned, and/or make decisions about corrective action. Also note that outputs from course adjustment in this theme loop back into Theme A (Commitment, specifically A.3.2), determination of metrics to be utilized in the future in Theme B (Data), and components of Theme C (Engagement), Theme D (Workforce), and Theme E (Inclusion strategies). Theme F also relates to Theme A (Commitment) in that F.3.3 assess if the commitment to develop metrics, embed them in review processes, and utilize them to hold institutions and staff accountable has been upheld. Theme F does not include dissemination of results, or advocacy for course adjustment or revised guidelines that are external to the institution or study (covered in Theme G– Dissemination).

  1. Cleaned and analyzed data for diversity, equity, and inclusion metrics in each theme (see operational example F.1.1)
  2. Review of the methodologies and results of the data analyses
  1. Collation of lessons learned; Assessment of any unintended consequences
  2. Comparison of data analyses and lessons learned to goals and performance criteria
  1. Accountability for DEI performance by institutions and staff
  2. Revision of policies, standard operating procedures, investigator and IRB staff human participant education requirements, data forms, tools, and checklists
  3. Revision of metrics and performance criteria

G

Dissemination and expansion of work (“Dissemination”)

Theme G focuses upon the dissemination of results, lessons learned, and course adjustments generated in Theme F (Review) for the purposes of transparency and accountability. Theme G includes soliciting diverse views in communication preparations for return of results, grey- and peer-reviewed publications, planning documents, tools, and other materials for information dissemination. Regardless of the topic being communicated, principles of transparency such as timely reporting (to all intended audiences internal and/or external), use of plain language, clear descriptions of methodologies, and communication of favorable and unfavorable results should be upheld. Finally, Theme G encompasses advocacy for the work to be continuously evaluted, shared, and improved. While DEI metrics and goals for CR may be achieved, progress toward health equity and justice in CR are likely still needed. Theme G does not indicate an end of the road, but rather the opportunity for reflection and expansion of the work, for which stakeholders can revisit and (re)apply other themes and measures in this framework. Please note: Theme G does not yet include quantitative metrics.

  1. Diverse clinical research staff involved in review and authorship
  2. Patients, caregiver/family, patient advocates, community members/organizations, community and institutional providers involved in review and authorship
  1. Internal communication of results, analyses, and plans (see operational example G.2.1)
  2. External communication of results, analyses and plans
  1. Institutional leaders and research teams active within institutional, local, national, or global networks/forums/conferences to share and/or promote practices, policies, and/or legislation
  2. Institutional leaders and research teams supporting others to access or generate forums within institutional, local, national, or global forums to promote DEI policies and/or legislation

Was this helpful?

Thanks for your feedback!