Note: Security ratings and months’ impact are not reported in our pilot evaluations as it lacks robust estimates of impact.
Resilient Families Plus is a 10-week school-based program designed to help students and parents develop knowledge, skills and support networks that promote health and well-being during early Secondary school years. The program comprises five core components of the original Resilient Families program which includes a school curriculum, delivered as part of usual classroom activities, parent education resources and events, and two additional program components that aim to encourage home reading, and improve school engagement and academic outcomes for students in early Secondary school years.
This evaluation was set up as a pilot study to test if the program is feasible in school settings in a 10-week intervention period and whether the program had an influence on academic achievement precursors such as academic self-concept (in Maths and English) and academic resilience.
Thirty-four Year 8 students (number of students who completed both pre- and post-surveys) from two schools and seven parents participated in this pilot with a high proportion of families from low socio-economic backgrounds in Victoria.
The small-scale pilot found that Resilient Families Plus did not have an additional impact on students’ academic self-concept in Maths and English and in academic resilience. Evidence suggests that students who initially reported lower levels of academic self-concept and academic resilience improved slightly after being involved in the program. These findings need to be treated with caution due to the small number of schools involved, inconsistent and incomplete program delivery, the use of other well-being programs that schools were running concurrently (including a Department-sponsored well-being program with similar features), and the lack of a control group to compare gains made over time.
Program fidelity was low as schools truncated the student curriculum and only implemented two or three sessions (estimated to be 15 – 20%) of the whole program and adapted the program (e.g., changing the sequence of activities and completion dates) based on their needs, making it difficult to determine if all program elements were delivered or effective. Two of the parent components of the program were not delivered by the program developer and the two new components of Resilient Families Plus were only completed after students sat the post-intervention survey. Schools reported difficulties delivering the full program as they found it time and resource intensive and were already implementing alternative curricula. Parental participation in the parent activities was very low (8% of participating students or 2% across Year 8 enrolment numbers from the two participating schools) as teachers reported challenges in organising these activities and securing parents’ time and involvement.
For these reasons, the program in its current form is not yet feasible for implementation in schools. The program developer identified barriers such as family stress, work commitments, English as an additional language and schools’ own challenges with parent engagement that may inhibit parents’ participation. To further improve the impact of this component, it would be valuable to ensure that schools and teachers are supported in securing greater parental participation to engage in student learning, especially in responding to issues that these communities may face.
Although the program is affordable, it would not be suitable for a randomised controlled trial (RCT) measuring its impact on student outcomes. There are two main reasons for this. First, the intervention needs to address its adherence to the prescribed model and barriers to faithful implementation, including schools’ interest using the program which makes it difficult to design an effective RCT. Second, the intervention allows schools to adapt its curriculum and activities to their needs and existing curriculum. This flexibility and the way it operates needs to be further assessed prior to a trial.
Subject area: Health and well-being
Recruitment challenges resulted in the pilot study differing from the original evaluation of two intervention groups (Resilient Families and Resilient Families Plus) plus a control group. Rather a smaller scale study of one intervention group (Resilient Families Plus) was conducted.
The Resilient Families Plus logic model and its program components is below. It reflects the anticipated change in students’ self-concept (in English and Maths) and in their academic resilience. Assessment of the program’s implementation occurred through interviews with participating parents, school leaders and the program developer as well as through inventories completed by school leaders and validated with data collected from the program developer.
|Is there evidence to support the theory of change?||No||The Resilient Families Plus program was not fully delivered and therefore the findings are inconclusive as to whether the theory of change can be supported.|
|Was the approach feasible?||No||There was low program fidelity, schools truncated the student curriculum, there was low parent uptake for the educational activities and the program was not delivered within the allocated time.|
|Is the approach ready to be evaluated in a trial?||No||A number of barriers to implementation need to be addressed prior to any future trial.|
Resilient Families Plus is a program delivered by Deakin University to support students in their early years of Secondary school and their parents, to support their well-being and improve students’ academic learning outcomes. The two participating schools were situated below the Australian average on the measure of community socio-educational advantage (ICSEA) and had a disproportionally high percentage of students from the lowest quartile, 64% (School 1) and 57% (School 2) compared to the Australian national average of 25%.
The program cost was calculated from the data provided by the program developers throughout the pilot study. It is rated as very low with a cost of $14,965 per school and $93.53 per student according to the Evidence for Learning Cost Rating approach, based on the approximate cost per student of implementing the intervention in one year.
- Impact of the intervention: Students reporting at baseline low levels of academic self-concept and academic resilience may benefit from Resilience Families Plus more than students who initially reported higher levels on these academic precursor measures. This is because students with lower baseline measures of academic self-concept and academic resilience showed small increases in these measures after being involved in the program. However, these increases were not statistically significant so need to be treated with caution. The small number of schools involved in the project and students who completed both the pre- and post-surveys (2 schools, 34 students, 7 parents), incomplete program delivery at the time students sat their post-intervention survey, other well-being programs that schools were running concurrently and the lack of a control group to compare gains made over time.
- Pilot challenges: There were significant challenges recruiting schools to participate in this evaluation. Neither the Victorian Education Department or research funders provided assistance in recruiting schools and other well-being programs that schools were already running (e.g. one of the two schools was implementing Respectful Relationships1, a Department-sponsored well-being program with similar features), may have impacted schools’ willingness to adopt Resilient Families Plus. A parallel research study undertaken by the program developer2 at the same time meant schools had to disseminate information and gather student consent forms for two separate projects may have delayed return rates, resulting in lower student participation numbers. The program developer’s research study was prioritised over those of the evaluation. For example, students had to complete the program developer’s survey questions before the evaluation’s in the same sitting, which might explain the lower survey completion rates than those of the concurrently run study.
- Implementation issues: Despite schools and parents acknowledging its value, schools faced challenges implementing the full 10-weeks of the student curriculum by the time of the post-test. For example, one school only selected to use 3 of the 10 sessions and then complemented these with other program content. Schools also reported changing the timing and sequencing of the activities. Additionally, schools did not implement the full complement of the parent components and when they did, this was out of the recommended sequence. Schools valued the flexibility the program allowed given they could select how many sessions to include as well as the timing and sequencing of these, however this resulted in poor fidelity to the prescribed program.
- Feasibility: Schools were reluctant to implement all the student curriculum components of the program and instead chose to supplement with alternative curriculum as they deemed the student curriculum as resource and time intensive. While acknowledging the program’s school-based parenting sessions, schools questioned its feasibility given challenges in managing these activities, the low interest and involvement from parents, and the time and involvement required from families and schools. The program developers identified the barriers to implementing the parent components were related to family stress, work commitments, English as a second language and schools’ challenges with engaging parents.
- Readiness for trial: Although the program is affordable, lack of adherence to the prescribed model and barriers to faithful implementation need to be addressed prior to any future trial. Schools’ interest in using the program and being involved in a trial would also need to be determined.
Resilient Families Plus E4L Commentary
The following ‘practitioner-friendly’ reports are free to access and download.
Resilient Families Plus Evaluation Report
Resilient Families Plus Executive Summary
Resilient Families Plus Evaluation Protocol
This evaluation report and supporting materials are licensed under a Creative Commons licence as outlined below. Permission may be granted for derivatives, please contact Evidence for Learning for more information.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Licence.
1. In 2016 Respectful Relationships education was recommended as a core component of the Victorian Curriculum. Respectful Relationships is an initiative to support schools and early childhood education settings promote and model gender respect and equality. https://www.education.vic.gov.…
2. Evidence for Learning and the evaluators agreed to the program developer continuing their business as usual practices, which includes the concurrent research study to inform future improvements. However, the evaluators and Evidence for Learning were not consulted on some of the research activities that were carried out by the program team.