OAITH Foundations of Violence Against Women (VAW) Online Training Program Evaluation
Evaluation and Report Prepared by:
Linda L. Baker, Ph.D., Learning Director, Centre for Research & Education on Violence Against Women & Children
Nicole A. Etherington, Ph.D, Research Associate, Centre for Research & Education on Violence Against Women & Children
Graphic Design by:
Krista Cipriani, Digital Media Support Assistant, Centre for Research & Education on Violence Against Women & Children.
Suggested Citation:
Etherington, Nicole A. & Baker, Linda L. (2016). OAITH Foundations of Violence against Women (VAW) Online Training Program Evaluation. London, ON: Centre for Research & Education on Violence Against Women & Children.
OVERVIEW
The Ontario Association of Interval & Transition Houses (OAITH) is a provincial coalition founded by women’s shelter advocates in 1977. Membership includes first stage emergency shelters for abused women and their children, second stages housing programs and community-based women’s service organizations. OAITH works with member agencies to educate and promote change in all areas that abused women and their children identify as important to their freedom from violence.
OAITH developed an online training course for individuals working or planning to work in the violence against women (VAW) sector, entitled the Foundations of VAW Online Training Course. OAITH requested an external evaluation of the Foundations course to be carried out by the Learning Network.
The Learning Network is a provincial knowledge translation and exchange initiative at the Centre for research & Education on Violence Against Women & Children funded by the Government of Ontario through the Ontario Women’s Directorate. With the guidance of its Provincial Resource Group, the Learning Network’s mandate is to build knowledge on gender-based violence, including the enhancement of supports for survivors, training for professionals, public education, and evaluation.
The evaluation contained in this report includes both quantitative and qualitative assessments of participants’ perceived and actual learning from the Foundations course across two groups of participants. Results point to the value of the course in its positive impact on a variety of participant outcomes (e.g. knowledge, attitudes), as well as recommendations for future course development and evaluation.
EVALUATION HIGHLIGHTS
- This evaluation was designed to reflect the core competencies for domestic violence training (see Broll et al., 2012), making it one of the few applications of these key criteria for assessing program impact.
- 187 participants registered for the OAITH Foundations of VAW Online Training Course, with 108 participants completing the entire course. This meets or exceeds industry standards for external, non-mandatory online training and evaluation surveys (Long, Dubois, & Faley, 2009; Archer, 2008).
- Participant demographic characteristics were consistent across both offerings of the course, enabling comparisons between groups (i.e., there were no statistically significant differences between groups).
- The course attracted participants of all age categories as well as those with various lengths of time worked and experience in the VAW sector.
- Most participants had at least some college or university education and had previous VAW training or learning experiences.
- The vast majority of participants rated their comfort, abilities, and knowledge as being enhanced by the course (>90%).
- Participants’ knowledge and attitudes were assessed objectively through a series of questions testing understanding of feminist principles and the nuances of VAW work. Participants’ scores significantly increased after completing the training, compared to their scores prior to taking the course. This provides support to participants’ subjective ratings of improved abilities and knowledge.
- The above findings held when scores of those who had completed the training were assessed relative to a comparison group, increasing confidence that the Foundations course effectively improves the knowledge and attitudes of those who take it.
- 97% of participants reported that the skills learned as a result of this training would be useful in their daily work, and 100% indicated that the training would make a positive difference in the way they do their jobs. Participants also identified that they would encourage others to change their practice to become more in line with the principles taught in the Foundations course.
- Qualitative assessments further contextualized these findings, with participants citing numerous examples of what was particularly helpful about the training (e.g. harm reduction component, discussion forums) and how their practice would change as a result of the training (e.g. language used, enhanced capacity to support women).
- The course evaluation also included asking participants to identify what would enhance the learning of future participants in this course (e.g. more case studies) and what additional training would be helpful (e.g. more information on risk and safety planning skills).
- The 79 participants who did not complete the Foundations course did not significantly differ from the participants who remained in the course on any of the demographic variables assessed (e.g. level of education, age, time worked/experience in VAW sector).
- Participants who did not complete the course were contacted to inquire about the factors influencing noncompletion. Participants who responded to this non-completion survey felt the course was valuable but time was the primary issue inhibiting completion; however, these comments may not be generalizable to all those who did not complete the course.
- Overall, the evaluation results provide evidence demonstrating training effectiveness as well as support for the core competencies of domestic violence training (e.g. recognizing the characteristics of violence, risk reduction).
DESCRIPTION OF COURSE
The Foundations of VAW Online Training Course is a four week course, covering a wide range of topics which are important for building a strong VAW practice based in feminist anti-violence principles and values. The course supports the development of a portfolio for VAW counsellors and advocates, and incorporates moderated discussions with peers. Course objectives are listed below. A detailed description of course content can be found in Appendix A.
Foundations Course Objectives:
- Support frontline staff to develop a professional portfolio that can be used in professional practice settings.
- Enhance participants’ knowledge of feminist intersectional theory and practice.
- Develop a feminist intersectional analysis of: documentation, risk assessment, harm reduction and mental health to inform participants’ practice.
- Encourage participants to engage in self-reflective practice.
- Facilitate networking and relationship-building among learners.
Course Promotion
The Foundations course was promoted via email to all shelters in Ontario as well as partner organizations (e.g. Ontario Coalition of Rape Crisis Centres, Aboriginal Shelters of Ontario). Materials promoting the course were made available at the Building a Bigger Wave Forum and provincial OAITH events in the fall of 2015.
The course was promoted to student and adult learner populations using social media (Twitter and Facebook) and the OAITH website. In addition, a waitlist had been in operation on the OAITH website since March 2014. These individuals were contacted once registration for the course opened.
To participate in the course, individuals had to create an account in OAITH’s online learning portal and self-enroll in the course. Enrollment for both sessions (January and February) opened at the same time, giving individuals the option as to which course they would enroll in.
A waitlist was also generated for the February offering of the course once the January course was underway in order to have participants that could fill spots should any of those individuals who signed up for the second offering of the course no longer be interested.
PARTICIPANTS
A total of 187 participants registered for the online training program, across the January (N=85) and February (N=102) offerings of the course. The January and February cycles will be referred to as Course 1 and Course 2, respectively.
There were 79 participants in total that did not complete the course (35 in January, 44 in February). These individuals were excluded from the evaluation analyses, resulting in a final sample size of 108 participants. Importantly, there was no relationship between whether a participant did not complete the course and their demographic characteristics. This means that with some confidence the conclusions drawn from the analyses can be said to be representative of the original group. Participants in Course 1 also did not significantly differ from participants in Course 2 on any characteristic, meaning the two groups are comparable.
Summary of Participant Demographic Characteristics (N=108)
A summary of participant demographic characteristics is provided below. For a detailed breakdown of demographic information, see Figures 1 through 5.
- 44% of participants were 25-34 years old.
- 56% of participants have worked in the violence against women (VAW) sector for 3 years or less.
- 22% of participants had experience in the VAW sector through school programs/practicums or volunteering only.
- 64% of participants had a college/university undergraduate degree.
- 87% of participants had been involved in a least one other form of VAW training (e.g. workshop) prior to completing this online course.
Figure 1. Age group of participants.
Figure 2. Time worked in violence against woman (VAW) sector.
Figure 3. Education level of participants.
Figure 4. Previous training or learning experiences in violence against women (VAW) sector.
Figure 5. Hours spent on course per week.
EVALUATION SURVEY
The evaluators developed survey questions designed to reflect a series of core competencies for domestic violence training programs (see Broll, Crooks, Baker & Straatman, 2012). These competencies cover such aspects of VAW knowledge and skills such as the characteristics and impacts of violence, intervention, reflective practice, professional roles in the workplace, and collaboration within and between organizations.
These competencies can be used to understand the impact of training programs on the real-world experiences of participants (Kahan, 2008; UNIFEM, 2010). In addition, these competencies represent key aspects of VAW work identified by VAW stakeholders and domestic violence training developers and deliverers (CREVAWC, 2011). Questions were also designed to reflect broader levels of evaluation, such as actual (i.e. reaction level) and perceived (i.e. learning level) knowledge gained, attitude change, and skills acquired, based on a widely accepted evaluation approach (Kirckpatrick, 1996; Thackwray, 1997), useful in identifying information for developing, evaluating, and revising training programs (Gramckow et al., 1997).
Last, participants were asked to provide qualitative responses on what they felt was the most helpful aspect of the training, how their practice would change as a result, and what changes could be made to improve the course overall.
Further details on evaluation metrics and survey questions can be found in Appendices B and C, respectively.
EVALUATION METHODS
Survey responses were scored so that higher scores indicate greater knowledge; more favourable attitudes; and higher self-ratings of comfort, knowledge, and abilities.
Maximum scores for each component of the survey are listed in Table 1.
In Course 2, participants rated how much they felt their comfort, abilities, and knowledge was enhanced by the training course.
Changes in participants’ average pre- and post- scores were assessed using paired t-tests.
Further detail on variables and methods used in the evaluation analyses can be found in Appendix D.
RESULTS: QUANTITATIVE DATA
Participants’ mean pre- and post-test scores are summarized in Figure 6. It should be noted that pre- and post-scores for comfort/abilities/knowledge were available for Course 1 only.
There was a significant increase in scores for knowledge, attitudes, and comfort/abilities/knowledge (p<0.001) but no significant change in application scores.
Pre- and post-scores did not vary by participant characteristics, with the exception of the following:
- Knowledge: An increased post-test score was also associated with belonging to the age category 45-54.
- Attitudes: More favourable attitudes pre-test were associated with having worked in the VAW sector for 1-3 years.
- Application: A lower score post-test was associated with having worked in the VAW sector for 7-10 years.
- Comfort/Abilities/Knowledge:
- Pre-test: a higher rating of comfort, abilities and knowledge was associated with having worked in the VAW sector for 4-6 years, 7-10 years, and 10 or more years.
- Post-test: a higher rating of comfort, abilities and knowledge was associated with having worked in the VAW sector for 1-3 years.
Comparison Group
The post-test scores of participants in Group 1 were compared with the pre-test scores of participants in Group 2 to further determine whether apparent increases could be attributed to the online training course (see Figure 7). This comparison could be made as participants in each group did not differ on any demographic characteristics indicating that the participant groups were similar (e.g., same proportions in school, working in VAW settings). The major difference between the groups was that one had completed the course while the other had not. The group which had not yet completed the course (Group 2) is considered the comparison group in this analysis with the group that finished the course (Group 1) considered the Foundations intervention group.
The knowledge and attitude scores of the Foundations intervention group were significantly higher than the scores of the comparison group. This provides a greater degree of confidence in the impact of the Foundations course on improving knowledge and attitudes.
Participants in the comparison group rated their comfort, abilities and knowledge higher than the Foundations intervention group. This could be attributed to the realization by participants that their initial assessment of their comfort, abilities, and knowledge was inaccurate as completion of the course made them aware of aspects of VAW work they may have previously lacked a complete understanding of.
Discrepancies in application scores are likely the result of problematic question design as indicated by the pre-/post-test results. Further discussion of these issues can be found in the recommendations section of this report.
Rating Change in Comfort, Abilities, and Knowledge (Course 2)
Results for participants’ reports of the degree of enhancement in their comfort, abilities, and knowledge in VAW work are presented in Table 2. These were obtained from participants in Course 2 only.
General Evaluation Metrics: Reaction, Learning, and Performance (Course 2)
Participants’ reports of perceived changes in skills, attitudes, and knowledge as a result of participating in the training course are listed in Table 3. These were obtained from participants’ in Course 2 only.
QUALITATIVE DATA: IDENTIFYING GAPS AND LESSONS LEARNED
What was particularly helpful about the training?
Participants identified the harm reduction component of the training (28%) and discussion aspect of the training (23%) as the most helpful. See Figure 8 for additional components identified as helpful and examples of quotes given by participants which are both representative of the range of comments provided and speak to the value and impact of the training course.
What would you recommend changing about the training?
The majority of participants indicated that no changes to the training were necessary (31%). 23% recommended including more practical application examples, such as safety planning with diverse clients and applying feminist theory to practice. See Table 4 for more suggested changes and examples.
What additional training would be helpful?
The majority of participants (46%) indicated that additional training on risk and safety planning skills would be helpful, followed by counselling skills (18%) and an intersectionality focus (11%). Examples of risk and safety planning skills included training on additional assessment tools and handling high risk situations. Identified counselling skills included trauma-focused training, disclosure training, and feminist counselling skills. See Figure 9 for further suggesting training by participants.
How will your practice change as a result of this training?
Many participants indicated that their practice would change to reflect their increased understanding of harm reduction and enhanced ability to respond (25%). This change in understanding and approach is best reflected by one participant, who comments: “My argument against keeping women who are actively using drugs in the shelter has been completely diminished as a result of this training.” Participants also identified an enhanced capacity to support women (19%), such as through the use of an intersectionality framework, being more comfortable and/or confident in working with women, and possessing greater knowledge of issues relevant to VAW work. See Figure 10 for additional changes to practice identified and representative quotes.
NON-COMPLETION SURVEY RESULTS
To obtain greater insight into participants’ reasons for leaving the course, OAITH emailed a follow-up survey to those participants who did not complete the course (see Appendix E). Eleven responses were received. Of the 11 participants who responded to the non-completion survey, 8 indicated they did not remain in the course because they did not have the time to complete everything. The 3 other participants did not provide a response to this question.
Participants suggested that a longer timeframe be provided in which to complete the course, as often school or work requirements could take away from the number of hours they could spend on the course each week. In addition, participants commented that there was often a lot of information in each unit, which can take considerable time to work through and absorb. As one participant noted, some sections (e.g. portfolio development) could be an entire course in themselves.
Participants also indicated that flexibility in the course design itself would be helpful. That is, allowing participants to move ahead if they completed a section early and had time to do another, allowing participants to go back and complete a unit they did not have time for during a previous week, or allowing participants to enter/exit at various points in the course. For example, one participant indicated:
“I knew it was going to be a busy week so I was going to get it done beforehand but couldn’t because it would not allow me to get into the final unit until the Wednesday night which did not work for my schedule to complete… Now I need to re-do the entire course to finish the final week and get the certificate of completion.”
It was also noted that some “homework” exercises or discussions were not relevant to participants’ work and did not make sense for them to complete given their limited time. Participants commented that had they been able to skip these portions, they might have been able to complete other sections.
Importantly, participants expressed that they still valued the course overall, and that it was “definitely not a waste of time.” Participants also noted that they “enjoyed the material” and “learned quite a bit.” Many requested that the course be offered more often.
The extent to which these findings are representative of the others who did not complete the course is unknown. However, it is interesting that some of the same themes of time restraints and lack of flexibility in the course design were also reported by participants who completed the Foundations course.
SUMMARY OF RESULTS: SUPPORT FOR CORE COMPETENCIES
Participants demonstrated increased knowledge both objectively through multiple choice/true false questions and subjectively through self-reported enhanced knowledge as a result of the training course. In addition, the core competencies of recognition, response, and risk reduction were achieved given participants’ increased understanding of such items as the characteristics of violence, an intersectional framework, the consequences of violence, the impacts of mental health and addiction, risk management, documentation practices, confidentiality, and self-care. For example, 97% of participants indicated that their ability to apply a feminist intersectional approach in their practice was enhanced as a result of this training, and 93% reported an enhanced understanding of the ways in which violence affects women’s health.
Participants’ perceived gains and objective measures of increased knowledge were also attained through demonstrated attitude improvement (i.e. attitudes more reflective of feminist principles, assessed by level of agreement with a series of statements on the nuances of VAW work) and perceived attitude change. Attitudinal scores reflecting an understanding of the dynamics of woman abuse and the broader societal context in which it occurs increased following completion of the training course. Participants also reported increased confidence in skills and their understanding of violence and its impacts, quantitatively and qualitatively. These outcomes further relate to the core competency of recognition. One of the most impactful aspects of the training which lead to constructive attitude change and increased awareness was that of harm reduction. For example, 95% of participants reported an enhanced ability to engage in and/or support harm reduction efforts for women who use drugs as a result of the training course. Also, 28% of respondents specifically identified the harm reduction component of the training as the most helpful. Related to skill development, 93% of participants reported that they felt better equipped to support women who have experienced violence as a result of this training and that their skills in working with these women were enhanced.
The significant improvement in scores in the Foundations intervention group (i.e. group who had completed the training) compared to the comparison group (i.e. group that had yet to complete the training) provides support to the above findings and suggests this change is due to the training course itself, rather than other potential contributing factors. The fact that the observed improvement in knowledge and attitudes among the treatment group is similar to that which was observed in the overall pre-/post-test analyses strengthens conclusions of training effectiveness and evaluation measures. Additionally, the training course appears to beof particular importance given that most (87%) participants indicated that they had previous VAW training, and still demonstrated learning. The training may therefore be a valuable complement to existing initiatives outside of the organization.
Importantly, 100% of participants reported that this training will make a difference in the way they do their jobs. Changes to practice as a result of the raining included a greater understanding and enhanced approach to harm reduction, an enhanced capacity to support women, improvement in risk and safety planning abilities, improved documentation practices, more informed use of language, engagement in self-care, promoting training development, improve collaboration amongst coworkers and between organizations, and the ability to identify violence as a continuum.
Related to training impact, participants identified the following components of the course as particularly helpful: harm reduction, discussion forums, general content, relevance, structure of the course, risk and safety planning, format, and resources. While 31% of participants indicated no changes were necessary to the training, others identified more practical application examples, more videos/visuals, resolving technical issues, improving the course design, and allowing more time for completion as recommendations for future training. Indeed, the most notable barrier to completion was not having the time to complete everything in the course. Additional training on risk and safety planning skills, counselling skills, intersectionality, addiction and mental health, compassion fatigue/vicarious trauma, advocacy, and intervention with abusive partners was also identified as potentially helpful.
LIMITATIONS
This evaluation is not without its limitations. Specifically, there was no randomized control group available to be used in the current evaluation to determine if results were truly the result of the training course or a result of external factors (e.g. other educational opportunities, increased work experience). Nevertheless,
these variables were controlled for and a change in scores from pre- to post-test was still observed. Further, participants in Course 2 were asked to specifically rate changes which occurred as a result of the training. Finally, this evaluation included a wait-list comparison group that allowed for the results of the Foundations intervention group to be compared with the results of the group that was waiting to take the course the following month (Group 2). Still, a quasi-experimental design that randomly assigns participants to treatment and control groups is recommended for future evaluations in order to determine with greater confidence that positive gains are the result of the Foundations course rather than confounding factors. This evaluation suggests only that the course appears to promote beneficial gains for VAW practice, but randomly assigned groups would solidify this observation and enhance the generalizability of the findings. The use of a randomized control group, in addition to a follow-up survey of participants several months after the course was completed, would speak to both desired gains in and retention of knowledge.
Challenges also exist related to sample size and retention of participants. Although results were not biased due to attrition and sample size provided adequate power for analyses conducted, an increased sample size with greater retention would strengthen the reliability, generalizability, and validity of results.
RECOMMENDATIONS FOR COURSE MODIFICATION
Based on participant feedback, the following recommendations for course modification may be useful for OAITH to consider:
- Allowing more time for participants to complete the course.
- Increased flexibility in course design (i.e. allowing participants to move ahead if they finish a unit early, making discussion optional).
- Ensuring all units have clearly detectable audio.
- Providing more examples of application in practice for complex topics such as intersectionality (e.g. safety planning with diverse clients).
- Including more information on creating safe environments for and responding to disclosures of violence.
- Providing optional additional training on identified topics of interest such as risk and safety planning, addictions and mental health, counselling skills, intersectionality with links to these training opportunities (e.g., linking participants to other available online trainings).
NEXT STEPS FOR COURSE EVALUATION
Based on results shown here, the following next steps are recommended as important directions for OAITH to consider for subsequent course evaluations, which will contribute to strong evidence for the effectiveness of training programs:
- Run course with quasi-experimental design:
Half of registered participants would be assigned to Course 1 and the other half assigned to Course 2. All participants would complete the pre-test prior to Course 1 and the post-test at the end of Course 1. If the first group of participants improves on the post-evaluation and the second group does not, positive gains could more confidently be attributed to the Foundations Course rather than other confounding factors. All participants would also take the post-test at the end of Course 2, with the responses of the Course 1 participants serving as a follow-up measure to provide information on knowledge retention. This design would further promote the Foundations course as an evidence-based curriculum of value to employers and educators, in addition to positioning OAITH as leaders in Ontario and Canada in developing and evaluating training in this area. OAITH would also become more competitive for additional training grants.
- Develop marketing strategies to increase participation:
- Improve course description on organization website
- Advertise course as supported by emerging evidence and clearly describe benefits to participation
- Create a visually appealing flyer that is easily shareable and printable (both in JPEG and PDF format)
- Promote course on social media (e.g. paid social media posts; Facebook, Twitter) to target audiences
- Reach out to member organizations, asking them to promote on websites, social media, print advertisements
- Email broadcast to subscription members
- Include print flyer whenever distributing resources
- Promote training at all OAITH events
- Make connections with relevant college/university programs to distribute information on the course
- Include a link to the training and brief description in email signatures
- Develop strategies to improve retention of participants:
- Modify course to be more flexible and to allow appropriate time for completion
- Send weekly email reminders to complete course, emphasizing how course is a valuable asset for working in VAW sector
- Emphasize certificate of completion awarded at end of course (and awarded only if both pre- and post- questionnaires are completed)
- Offer an additional incentive for participants who complete the course (e.g. entrance into a prize draw)
- •Revise application test questions to reflect more practical and relevant examples, as suggested by participants:
- Develop scenarios in consultation with shelter workers and organization leads
- Draw upon existing domestic violence case studies
- Ensure scenarios require participants to draw upon existing guidelines for assessing domestic violence
- Provide fewer scenarios to that each can include more detail (i.e. to avoid ambiguity)
- Pilot developed questions with small test group
- Include additional questions asked in the Course 2 post-test for all courses evaluated.
These next steps would represent an important follow-up to the current evaluation.
REFERENCES
Archer, T. (2008). Response rates to expect from web-based surveys and what to do about it. Journal of Extension, 46(3).
Broll, R., Crooks, C., Baker, L., & Straatman, A. (2012). Evaluating Violence against Women Training Initiatives: A Primer. London, ON: Centre for Research & Education on Violence against Women & Children.
Centre for Research and Education on Violence against Women and Children (CREVAWC). (2011). Knowledge exchange workshop on domestic violence training, June 16-17, 2011: Report of the meeting. London, ON: Centre for Research and Education on Violence against Women and Children.
Gramckow, H.P., Nady Sigmon, J., Gaboury, M.T., & Burt, M.R. (1997). Measuring the impact of training. In M.R. Burt, A.V. Harrell, L.C. Newmark, L.Y. Aron, L.K. Jacobs et al. (Eds.). Evaluation guidebook for projects funded by S.T.O.P. formula grants under the Violence Against Women Act (pp. 201-220). Washington, DC:
Urban Institute.
Kahan, B. (2008). Review of Evaluation Frameworks. Regina, SK: Saskatchewan Ministry of Education. Available Online
Kirkpatrick, D.L. (1996). Evaluating training programs: The four levels. San Francisco: Berrett-Koehler Publishers, Inc.
Long, L., Dubois, C., & Raley, R. (2009). A case study analysis of factors that influence attrition rates in voluntary online training programs. International Journal on E-Learning, 8(3), 347-359.
Thackwray, B. (1997). Effective evaluation of training and development in higher education. London: Kogan Page.
United Nations Development Fund for Women (UNIFEM). (2010). Monitoring, evaluation and knowledge management framework. New York: UNIFEM.
APPENDIX A: FOUNDATIONS VAW ONLINE TRAINING COURSE UNITS
Unit 1: Foundational Frameworks
1.1 Intersectional Feminism and Violence Against Women
- Explores theories and examines concepts that can be applied to VAW practice, such as intersectionality.
1.2 Portfolio Development
- Looks at how to develop a portfolio and how it can be used to support growth and development throughout practice.
Unit 2: Ethics
2.1 Ethical Decision-Making
- Explores guidelines and tools for developing an ethical model for your VAW practice.
2.2 Record-Keeping: Risks and Strategies
- Explores the different obligations for keeping records, outlines benefits and risks or keeping records, and identifies strategies for applying an intersectional feminist framework to record-keeping practices that emphasize keeping women safe.
2.3 Self-Reflective Practice
- Defines the purpose and approach of self-reflective practice, with guiding questions and strategies for implementing the concepts into regular practice.
- Includes developing an understanding of Compassion Fatigue and Vicarious Trauma, and provides strategies for recognizing and addressing these.
Unit 3: Unpacking Constructs around Mental Health and Substance Use
3.1 Feminist Analysis of Mental Health Issues
- Explores concepts behind mental health labels to understand how they impact women who have experienced violence with a focus on recognizing how the tendency to problematize the individual’s behaviour can obscure the circumstances that are impacting an on individual’s well-being.
3.2 Harm Reduction
- Explains what the Harm Reduction model and why we choose to use it, and provides concrete information about how to reduce harms associated with substances and substance use, including strategies for supporting women who use.
Unit 4: Risk Assessment and Safety Planning
- Explains different approaches to Risk Assessment and Safety Planning, including both benefits and limitations.
- Identifies strategies and tools for supporting women working with the Criminal Justice System and applying feminist principles and approaches to conducting Risk Assessment and Safety Planning with women in VAW practice.
APPENDIX B: EVALUATION MEASURES
B1. General Evaluation Metrics
Participants were asked a set of questions designed to reflect a broader set of metrics based on the evaluation levels identified by Kirkpatrick (1996), listed in Table B1 as well as performance measures required of OAITH. Kirkpatrick’s (1996) evaluation approach is widely accepted and credited with “revolutionizing the evaluation of training programs” (Thackwray, 1997; Broll et al., 2012, p. 8).
Note: the other two evaluation levels (behaviour change and impact on VAW) were not relevant to the goals of the current training and were therefore excluded.
The performance measure included assessed the percentage of participants who reported that, as a result of the training, they are better equipped to support women and their children who have experienced violence. Participants were also asked to identify changes that would have improved the training, lessons learned, and training gaps that still exist, further speaking to the impact of the program, another organizational requirement.
B2. Core Competencies for Domestic Violence Training Programs
The core competencies adapted by Broll and colleagues (2012) were developed based on a comprehensive review of the literature on VAW training initiatives and the role of evaluations in VAW training programs, in addition to a knowledge exchange workshop hosted by the Centre for Research & Education on Violence
Against Women & Children. Core competencies are an important addition to VAW training evaluations in that it extends evaluation beyond the reaction or learning levels (Broll et al., 2012). A detailed list of these competencies and corresponding knowledge and skills is summarized in table B2.
B3. Survey Operationalization of Core Competencies & Evaluation Metrics
Knowledge, attitudinal, and application measures were included pre- and post-test for both courses.
Knowledge: A set of multiple choice and true/false questions on VAW knowledge was included to assess reaction and learning, and individual questions can be linked to each of the three core competencies (recognition, response, risk reduction).
Attitudes: Participants were presented with a series of statements on VAW with which they had to indicate their level of agreement. These statements were designed as an attitudinal measure, mapping onto the core competency of recognition and the evaluation levels of reaction and learning.
Application: Learning as well as the core competencies of recognition and response were further assessed through a series of application questioned requiring participants to identify whether a given scenario could be classified as intimate partner violence.
Course 1 piloted the following question:
Reaction: Participants in Course 1 (January) were asked a series of questions asking participants to rate their comfort level, abilities, and knowledge on aspects of VAW practice was used to assess the reaction evaluation level. This item also captured elements of the three core competencies (response, recognition, and risk reduction).
Course 2 tested some additional post questions, listed below:
Reaction: Participants in Course 2 (February) were asked to indicate how much of a change (diminished, enhanced, no change) in their comfort levels, abilities, and knowledge could be attributed to the training course.
Performance measure: Participants in Course 2 were asked whether they felt better equipped to support survivors of violence in addition to other questions on the impacts of the training course (e.g. whether skills useful in daily work, topics applicable to practice/service setting). See Appendix B2, question 29 for further detail.
Impact: Participants in Course 2 were asked to provide qualitative responses on what was particularly helpful about the course, how their practice would change as a result of the course, what improvements could be made to the course, and what additional training might be helpful. Detailed questionnaires and links between individual questions and specific core competencies/general evaluation metrics can be found in Appendix C.
APPENDIX C: PRE/POST QUESTIONNAIRES
Pre- and post- questionnaires are included below. The core competency/general evaluation metric that each question is linked to is identified in purple in brackets after the question.
C1. Pre-training Questionnaire
Thank you for agreeing to take part in this evaluation of the Foundations of Violence Against Women (VAW) online training program. Your responses to the following questionnaire will provide valuable feedback as OAITH aims to continuously improve its training programs. The questionnaire should take approximately 20 minutes to complete. All responses are voluntary and anonymous.
Part 1:
1. What is your age?
a. 17-24
b. 25-34
c. 35-44
d. 45-54
e. 55-64
f. 65 years or older
2. How long have you worked in the violence against women sector?
a. Less than 1 year
b. 1-3 years
c. 4-6 years
d. 7-10 years
e. 10 or more years
f. Placements/practicums through my school program only
g. Volunteer experience only
3. What is the highest degree or level of school you have completed?
a. Secondary school
b. Some college courses
c. Some university courses
d. Some graduate courses
e. College diploma
f. Undergraduate (university) degree
g. Graduate degree
h. Other (please specify): _______________
4. Please indicate any previous training/learning experiences you have had on intimate partner
violence (check all that apply).
a. Attended a workshop on VAW
b. Work/school placement/practicum
c. Volunteered at a VAW organization
d. Took a course on VAW in school/continuing education
e. Other (please specify): ____________________
f. None
Part 2:
5. Please rate the following questions in regards to working in the VAW sector on the scale provided.
(REACTION)
6. Please indicate your level of agreement with the following statements. (1/2)
7. Oppression, or the systematic mistreatment of individuals based on a belief in the innate superiority of one group over another, operates: (1d)
a. At the interpersonal level
b. At the institutional level
c. At the social/cultural level
d. At the interpersonal, institutional, and social/cultural levels
e. At the institutional and social/cultural levels
8. According to intersectional analysis… (1d)
a. Responses to VAW framed in sexism and patriarchy reflect the experiences of all women
b. Women do not occupy one identity at a time and their experiences reflect multiple social stratifications that interact with each other
c. Women are the most disadvantaged social group
d. A and B
e. None of the above
9. Asking yourself questions such as: “What did I do or say?” “How might I do or say this differently?” “What have I learned?” is an important part of… (Pick the best answer) (7)
a. Being a professional
b. Professional portfolio development
c. Self-reflective practice
d. Ethical decision making
e. Self-care
10. Documentation and record-keeping can be harmful to service users. (4d)
a. True
b. False
c. Not sure
11. Written permission to turn over a client’s records is not required when ordered to do so by a court of law. (4d)
a. True
b. False
c. Not sure
12. A feminist and anti-oppression approach to documentation and record-keeping involves promising a client that you will never turn over her records to anyone. (6b)
a. True
b. False
c. Not sure
13. Which of the following options is NOT true: When working with women who have experienced violence, reframing experiences of violence: (3c)
a. Can help women understand that the violence is not their fault
b. Can help to identify mental health conditions
c. Can involve identifying actions that women engage in as coping strategies
d. Can Identify women’s responses to violence as forms of resistance
e. All statements are true
14. Being diagnosed with a mental health condition can be both helpful and problematic for some women. (2c)
a. True
b. False
c. Not sure
15. Feelings of fear and anxiety decrease when women leave their abusive partners. (2)
a. True
b. False
c. Not sure
16. What is harm reduction? (Choose one) (3)
a. Programs, policies, and practices that promote abstaining from drugs
b. Programs, policies, and practices that aim to keep those who use drugs safe, including minimizing the risk of death, disease, and injury
c. Advocating for the legalization of particular drugs
d. Reducing harms to society that result from drug use
e. None of the above
17. Risk is: (3e)
a. The probability or likelihood of future or continuing harm or hazard
b. Fear of the unknown
c. Danger that cannot be managed
d. A process of assessment
e. All of the above
18. What a woman tells you is between her and you; therefore, there is no need to coordinate with
other agencies. (6)
a. True
b. False
c. Not sure
19. Compassion fatigue can be easily resolved by taking time off or changing jobs. (7c)
a. True
b. False
c. Not sure
20. An experience can only be considered traumatic if it involves serious physical injuries. (2f)
a. True
b. False
c. Not sure
21. To qualify as intimate partner violence, there must be physical harm. (2)
a. True
b. False
c. Not sure
For each of the following scenarios, please indicate which you believe constitute intimate partner violence. (1/2)
22. Megan has been diagnosed with epilepsy and sometimes needs a wheelchair to go back and forth to the restroom. When Robert gets drunk (about three times a week), he puts her wheelchair in the living room and Megan has to struggle to make it to the bathroom. Robert always tells Megan that if she says anything to anyone, he will divorce her and then no one will help her.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
23. One night over dinner, Joseph and Sophia got into a heated discussion about an article they had both read in the news. Joseph became so frustrated with Sophia’s point of view that he stood up and left the table to go to the bedroom, where he slammed the door.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
24. Casey does not allow Toni to see her family. When Toni attempts to call her family, Casey takes the phone and locks it in the trunk of her car.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
25. Matt and Jennifer argued over what television show to watch. Jennifer was disappointed with the show she saw.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
26. Kim regularly withholds Alex’s hormones for gender transition.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
27. Aisha made spicy chilli for dinner. Caleb said he could not eat it because it was too spicy. He made a frozen dinner instead, and was upset with his meal.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
28. When Danielle receives her paycheck from work, Gene takes her to the bank to cash it. Gene pays their bills online, and keeps the rest in his wallet.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
29. Jamie and Patrick have been dating for 5 years. Last week, Patrick hit Jamie for the first time.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
C2. Post-training Questionnaire
Thank you for agreeing to take part in this evaluation of the Foundations of Violence Against Women (VAW) online training program. Your responses to the following questionnaire will provide valuable feedback as OAITH aims to continuously improve its training programs. The questionnaire should take approximately 20 minutes to complete. All responses are voluntary and will be kept confidential.
Part 1:
1. Please rate how much each of the following aspects of working in the VAW have changed as a result of participating in the Foundations of VAW online training program. A response of 0 indicates no change. +1 to +3 indicates by how much you feel your understanding/knowledge/comfort/ability levels have been enhanced. -1 to -3 indicates by how much you feel your understanding/knowledge/comfort/ ability levels have diminished.
(REACTION)
(LEARNING)
2. Please indicate your level of agreement with the following statements. (1/2)
3. Oppression, or the systematic mistreatment of individuals based on a belief in the innate superiority of one group over another, operates: (1d)
a. At the interpersonal level
b. At the institutional level
c. At the social/cultural level
d. At the interpersonal, institutional, and social/cultural levels
e. At the institutional and social/cultural levels
4. According to intersectional analysis… (1d)
a. Responses to VAW framed in sexism and patriarchy reflect the experiences of all women
b. Women do not occupy one identity at a time and their experiences reflect multiple social stratifications that interact with each other
c. Women are the most disadvantaged social group
d. A and B
e. None of the above
5. Asking yourself questions such as: “What did I do or say?” “How might I do or say this differently?” “What have I learned?” is an important part of… (Pick the best answer) (7)
a. Being a professional
b. Professional portfolio development
c. Self-reflective practice
d. Ethical decision making
e. Self-care
6. Documentation and record-keeping can be harmful to service users. (4d)
a. True
b. False
e. Not sure
7. Written permission to turn over a client’s records is not required when ordered to do so by a court of law. (4d)
a. True
b. False
c. Not sure
8. A feminist and anti-oppression approach to documentation and record-keeping involves promising a client that you will never turn over her records to anyone. (6b)
a. True
b. False
c. Not sure
9. Which of the following options is NOT true: When working with women who have experienced violence, reframing experiences of violence: (3c)
a. Can help women understand that the violence is not their fault
b. Can help to identify mental health conditions
c. Can involve identifying actions that women engage in as coping strategies
d. Can Identify women’s responses to violence as forms of resistance
e. All statements are true
10. Being diagnosed with a mental health condition can be both helpful and problematic for some women. (2c)
a. True
b. False
c. Not sure
11. Feelings of fear and anxiety decrease when women leave their abusive partners. (2)
a. True
b. False
c. Not sure
12. What is harm reduction? (Choose one) (3)
a. Programs, policies, and practices that promote abstaining from drugs
b. Programs, policies, and practices that aim to keep those who use drugs safe, including minimizing the risk of death, disease, and injury
c. Advocating for the legalization of particular drugs
d. Reducing harms to society that result from drug use
e. None of the above
13. Risk is: (3e)
a. The probability or likelihood of future or continuing harm or hazard
b. Fear of the unknown
c. Danger that cannot be managed
d. A process of assessment
e. All of the above
14. What a woman tells you is between her and you; therefore, there is no need to coordinate with other agencies. (6)
a. True
b. False
c. Not sure
15. Compassion fatigue can be easily resolved by taking time off or changing jobs. (7c)
a. True
b. False
c. Not sure
16. An experience can only be considered traumatic if it involves serious physical injuries. (2f)
a. True
b. False
c. Not sure
17. To qualify as intimate partner violence, there must be physical harm. (2)
a. True
b. False
c. Not sure
For each of the following scenarios, please indicate which you believe constitute intimate partner violence.
18. Megan has been diagnosed with epilepsy and sometimes needs a wheelchair to go back and forth to the restroom. When Robert gets drunk (about three times a week), he puts her wheelchair in the living room and Megan has to struggle to make it to the bathroom. Robert always tells Megan that if she says anything to anyone, he will divorce her and then no one will help her.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
19. One night over dinner, Joseph and Sophia got into a heated discussion about an article they had both read in the news. Joseph became so frustrated with Sophia’s point of view that he stood up and left the table to go to the bedroom, where he slammed the door.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
20. Casey does not allow Toni to see her family. When Toni attempts to call her family, Casey takes the phone and locks it in the trunk of her car.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
21. Matt and Jennifer argued over what television show to watch. Jennifer was disappointed with the show she saw.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
22. Kim regularly withholds Alex’s hormones for gender transition.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
23. Aisha made spicy chilli for dinner. Caleb said he could not eat it because it was too spicy. He made a frozen dinner instead, and was upset with his meal.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
24. When Danielle receives her paycheck from work, Gene takes her to the bank to cash it. Gene pays their bills online, and keeps the rest in his wallet.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
25. Jamie and Patrick have been dating for 5 years. Last week, Patrick hit Jamie for the first time.
a. Yes, there is intimate partner violence happening.
b. No, there is not intimate partner violence happening.
c. There is not enough information for me to decide.
Part 2:
26. On average, how many hours per week did you spend on this course?
a. Less than 1
b. 1-2
c. 3-4
d. 4-5
e. 6 hours or longer
27. Please indicate your level of agreement with the following statements.
As a result of completing this training…. (REACTION & OWD PERFORMANCE MEASURE)
(IMPACT)
28. What was particularly helpful about the training?
29. What would you recommend changing about the training?
30. What additional training (if any) would be helpful?
31. Please provide one example of how your practice will change as a result of this training.
APPENDIX D: VARIABLES AND METHODS
Variables
Knowledge and application questions were coded so that 1 indicated a correct response. Scores for each question were then totaled to generate a knowledge measure and an application measure, where a greater score indicates a greater number of questions answered correctly. There were 15 total questions for knowledge, meaning participants could get a maximum score of 15. There were 8 total questions for application, meaning participants could get a maximum score of 8.
Attitudinal questions were re-coded from 0 to 4, with a higher score indicating more favourable attitudes.
Scores for each question were combined into a single measure, with a maximum score of 60 (4 X 15 questions).
For Course 1, ratings of comfort, knowledge, and abilities were coded from 0 (low) to 4 (high), then combined across questions into a single measure, with a maximum score of 52 (4 x 13 questions).
For Course 2, responses were re-coded as 0 = greatly diminished, 1 = no change, and 2 = enhanced.
Responses were combined into a single measure. These responses were analyzed as descriptive only given they were asked in the post-training questionnaire but not the pre-training questionnaire.
Two of the application questions were removed due to issues of ambiguity in the question leading to a high number of incorrect responses.
Methods
Paired t-tests were conducted to determine whether mean scores pre-test differed significantly from those at post-test. Two-sample t-tests were conducted to determine whether the mean scores of the treatment group significantly differed from the mean scores of the control group.
Ordinary least squares regression models were used to determine whether scores at times 1 and 2 varied by participant demographic characteristics. Variation in mean scores by participant characteristics was also assessed using ANOVA.
All quantitative analyses were conducted using STATA 14.
Participants were asked to provide qualitative responses identifying both helpful and problematic aspects of the training, as well as perceived gaps.
Qualitative analyses were conducted using NVivo 11.
APPENDIX E: NON-COMPLETION SURVEY
1. Please choose the option that best describes your current situation.
- Working in a women’s shelter (includes emergency short-stay shelters and longer-term or second stage shelters)
- Volunteering in a shelter or the VAW sector
- Student
- Other (please specify) _____________________________________
2. If you work in a shelter, how long have you been doing this work?
- Less than 2 years
- 2-5 years
- More than 5 years
3. Please tell us why you registered for this course and what you were hoping to achieve by taking it.
(Select all that apply.)
- Improve / learn new job skills.
- Increase knowledge of Violence Against Women and how to provide support services.
- Understand specific issues (eg, Harm Reduction, Risk Assessment, Mental Health, etc.) Please use the “other” box below to tell us what interests led you to register for this course.
- Supplement my studies with practical learning and tools.
- Considering entering the VAW sector and wanted to learn more about issues and/or increase my chances of getting a job.
- Connect with others in the field.
- Other / More info (please specify) _____________________________________________
4. Why did you not complete the course? (Select all that apply.)
- I didn’t have the time required to complete everything.
- After starting, I decided that the course was not relevant to my needs.
- The platform for the course was not user-friendly.
- Content was not easy to follow or understand.
- The content was not new to me.
- Other (please specify) / More information ______________________________________
5. Is there anything else you would like to tell us about your experience of taking this course or why you did not complete it?
