July 2015 • Vol. 4, Issue 3

HT TOPIC

Joint Review Committee on Education in Radiologic Technology Shares Plan

Stephanie Eatmon, EdD, R.T.(R)(T), FASRT

This article is based on content previously published by the American Society of Radiologic Technologists and is used with permission of the ASRT.

Assessment: Working Backward

Developing a plan to assess student learning for an imaging, radiation therapy, or medical dosimetry program can be a daunting task. Starting backward from the ideal graduate and deciding what to measure makes the entire process easier and provides data that is essential for program improvement. By working backward, assessment becomes more than a difficult, meaningless task; it becomes an exciting, informative process. Examining the ideal graduate helps identify exactly what skills, attitudes, behaviors, and values are important to measure.

Similar to terminology used in computer technology, the adage of “garbage in, garbage out” is also valid for assessment. If the objectives and student learning outcomes (SLOs) do not measure the skills, attitudes, behaviors, and values of the ideal graduate, no amount of data or analysis will provide the program with information for continuous quality improvement. If the ideal graduate is one who is skilled at producing a PowerPoint presentation, then that skill should be measured. If that skill is not important for an entry-level radiographer, magnetic resonance technologist, radiation therapist, or medical dosimetrist, then measuring that skill provides meaningless data and will never lead to program improvement. This article provides a different lens through which to view assessment and make the process easier and more useful to the assessment team and the program.

The Joint Review Committee on Education in Radiologic Technology developed standards for accredited educational programs with input from the medical imaging, radiation therapy and medical dosimetry communities.^{1, 2, 3, 4} The JRCERT Standards require, at minimum, for programs to set goals and measure SLOs in 4 areas: clinical competence, critical thinking, professionalism, and communication. When working assessment backward, a program should ask itself these questions:

·         What skills, attitudes, and behaviors do we want our graduates to possess in relation to the 4 goal areas?

·         How will our graduates be differentiated from graduates of other medical imaging, radiation therapy, and medical dosimetry educational programs?

·         How will we know if our educational program has been successful?

Answering these questions provides a more concrete idea of what assessment data is needed to ensure the program is reaching its established goals. Each program is unique, so the assessment plan will be unique as well. Although the 4 goal areas might be the same, the objectives and SLOs differ depending on the type of program, its structure, and degree offered. For example, at the certificate and associate degree levels, the program might not require students to write research papers for publication; however, at the baccalaureate and master’s degree levels this might be appropriate. One program might place extra emphasis on communication with patients from diverse populations, while another might focus on adaptability to new technologies. Each program determines what unique qualities it desires its graduates to possess. Continuous quality improvement via assessment is the only way to determine if goals are being met, and if not, what changes need to be made to improve the program.

Desirable Skills, Attitudes, and Behaviors

Considering the 4 goal areas, there are many desirable skills, attitudes, and behaviors for the graduate. Box 1 contains possible statements a program might consider depending on its goals. These are not necessarily SLOs, but merely statements a program might make regarding what graduates could achieve or demonstrate in relation to each goal. With further refinement, these statements can become the basis for SLOs.

Important to note is the absence of any reference to scores or pass rates on the credentialing examination or completion of specific classroom or laboratory assignments. Although all graduates are expected to pass the national credentialing exam, it is a minimum expectation and does not demonstrate a graduate’s ability to function with entry-level skills in the clinical setting. Consistent with the JRCERT Standards, however, programs must collect and report credentialing examination pass rates as one measure to assure program effectiveness. Regarding specific classroom or laboratory assignments—unless a program determines, for example, that it is vital the graduate be skilled at presentations using PowerPoint software, developing and gathering data for this attribute provides less than meaningful assessment data for the program.

Differentiation

To create differentiation, the program and the respective communities of interest should focus on a unique set of values that are well-integrated into the curriculum. For example, a program might state something similar to the below in reference to its values:

In addition to meeting all of the learning outcomes, our students will be identified by their demonstration of excellence in regard to:

·         Competence in addressing cultural diversity.

·         Excellence in patient care.

·         Technical expertise.

·         Professional activities.

·         Adaptability.

·         Community service.

Each value is well-communicated to students because the topic is reinforced in almost every class and evaluated throughout the program.

Putting It All Together

Once a program is clear on the skills, attitudes, behaviors, and values associated with the ideal graduate, it is time to put it all together. Further delineation is needed to hone in on exactly what each statement means in terms of measurable skills, attitudes, and behaviors. Although some statements seem clear, it is important that everyone associated with the program knows exactly what the requisite skills, attitudes, behaviors, and values are and how they are to be measured and assessed. Consider the statement from the critical thinking example (see Box 1): Our graduates are adaptable to new procedures, technologies, and situations. The specific behaviors, skills, and attitudes associated with the statement must be identified to determine how to use this desired trait of a graduate in an assessment plan. Skills, attitudes, and behaviors associated with adaptable need to be determined, and specific new procedures, technologies, or situations of focus for the program specified. Once these steps are completed, SLOs can be developed for a particular goal (see Box 2).

Measuring Success

When the program and all of the communities of interest have agreed on the ideal graduate’s skills, behaviors, attitudes, and values and appropriate SLOs have been developed and measured, the final step is to analyze the assessment results to determine program success. If benchmarks are continually being met, it might be time to raise the benchmarks or find another area to examine and assess for program improvement. However, if the established benchmarks are not being met over multiple assessment periods, the program should reassess benchmarks, measurement tools, and perhaps curriculum. It is possible the content area covered by the SLOs needs to be enhanced or the benchmarks might have been set to high. The program can analyze the data for patterns. Data might show a specific group of students not meeting the benchmark or it might show all students struggling with the SLOs. Trends leading to decreasing results might be found by studying data over several years. When examining data in relation to the curriculum, any changes in teaching methodology or instructors should be taken into account. Careful analysis of data will show if the measurement tools really measure the SLOs and should help the program identify opportunities for incremental improvement, which is the exact purpose of assessment.

Conclusion

Assessment is the most effective means by which a program can make evidence-based decisions for program improvement. Working assessment backward involves the program and communities of interest working together to make the process more meaningful. Asking questions to determine what skills, attitudes, behaviors, and values are most important to the program and communities of interest provides valuable insight for the program and a roadmap for the assessment process. Once the meaningful traits are identified, determining what to measure is simple. With measurable SLOs, multiple measurement tools can be identified or developed for effective measurement of student learning.

Because communities of interest are involved in the assessment process, there is buy-in from the clinical sites. When program faculty, clinical instructors, and clinical staff are all on the same page in regard to the expectations for student learning, the evaluations and measurement tools are more effective and the data more reliable. The data that is collected is truly meaningful, and when analyzed, it is a powerful tool for program improvement. No program is perfect, but with incremental changes programs can continue to improve.
 

Standards for Accredited Educational Program in (Radiography, Magnetic Resonance, Radiation Therapy and Medical Dosimetry)

Standard Five
Assessment

Standard Five: The program develops and implements a system of planning and evaluation of student learning and program effectiveness outcomes in support of its mission. 

Objectives:
            In support of Standard Five, the program: 

Student Learning 

5.1       Develops an assessment plan that, at a minimum, measures the program’s student learning outcomes in relation to the following goals: clinical competence, critical thinking, professionalism, and communication skills.

Stephanie Eatmon, EdD, R.T. (R)(T), FASRT, is a member of the JRCERT Board of Directors and an advisor for the National University Radiation Therapy Program. She can be contacted at sleatmon15@gmail.com.

References

1.   Joint Review Committee on Education in Radiologic Technology. Standards for an accredited educational program in radiography. JRCERT Web site. http://www.jrcert.org/sites/jrcert/uploads/documents/2014_Standards/Standards_2014-Radiography.doc. Adopted October 2013. Effective January 1, 2014. Accessed January 16, 2015.

2.   Joint Review Committee on Education in Radiologic Technology. Standards for an accredited educational program in magnetic resonance. JRCERT Web site. http://www.jrcert.org/sites/jrcert/uploads/documents/2014_Standards/Standards_2014-Magnetic_Resonance.doc. Adopted October 2013. Effective January 1, 2014. Accessed January 16, 2015.

3.   Joint Review Committee on Education in Radiologic Technology. Standards for an accredited educational program in radiation therapy. JRCERT Web site. http://www.jrcert.org/sites/jrcert/uploads/documents/2014_Standards/Standards_2014-Radiation_Therapy.doc. Adopted October 2013. Effective January 1, 2014. Accessed January 16, 2015.

4.  Joint Review Committee on Education in Radiologic Technology. Standards for an accredited educational program in medical dosimetry. JRCERT Web site. http://www.jrcert.org/sites/jrcert/uploads/documents/2014_Standards/Standards_2014-Medical_Dosimetry.doc. Adopted October 2013. Effective January 1, 2014. Accessed January 16, 2015.

Box 1

Possible Statements for a Program to Consider

Clinical Competence

·         Our graduates are able to competently run a general radiography room upon graduation.

·         Our graduates take quality general radiographs in multiple settings.

·         Our graduates are able to select appropriate scans and protocols.

Critical Thinking

·         Our graduates are adaptable to new procedures, technologies, and situations.

·         Our graduates identify errors and are able to provide suggestions for correction.

·         Our graduates participate actively in quality assurance and safety procedures.

·         Our graduates provide appropriate resources needed to meet patient needs.

Professionalism

·         Our graduates are reliable and ethical.

·         Our graduates participate in continuous learning.

·         Our graduates are actively involved in professional presentations at the local, state, and national levels.

·         Our graduates conduct research to expand knowledge of the profession.

·         Our graduates are exemplary in their attitudes and behavior toward patients. 

Communication

·         Our graduates use appropriate medical terminology.

·         Our graduates appropriately discuss care strategies for side effects with patients under treatment.

·         Our graduates gather accurate patient histories.

·         Our graduates discuss procedures with patients in a manner appropriate to the patient’s age, gender, ethnicity, and general ability to understand.

Box 2

Example Goal 1:  Our students/graduates will utilize critical thinking skills.

·         SLO 1.1  Students will analyze pretreatment imaging and make treatment adjustments – a possible measurement tool is an imaging rubric.

·         SLO 1.2  Students will produce treatable plans for areas that have previously received radiotherapy – a possible measurement tool is a treatment plan.

·         SLO 1.3  Students will adjust positioning to accommodate the patient’s condition – possibilities for measurement tools are clinical evaluations and  simulation evaluations.

Example Goal 2: Our students/graduates will model/demonstrate professionalism.

·         SLO 2.1 Students will demonstrate ethical behaviors – possibilities for measurement tools are clinical evaluations and employer surveys.

·         SLO 2.2 Students will be prepared for the employment interview process – a possible measurement tool is to produce a professional résumé.

·         SLO 2.3 Students will be able to analyze career opportunities – a possible measurement tool is to develop a career plan.

Signals is a publication produced four times per calendar year by the International Society for Magnetic Resonance in Medicine for the benefit of the SMRT membership and those individuals and organizations that support the educational programs and professional advancement of the SMRT and its members. The newsletter is the compilation of editor, Julie Strandt-Peay, BSM, RT (R)(MR) FSMRT, the leadership of the SMRT and the staff in the ISMRM Central Office with contributions from members and invited participants.