Assessment of Practice in Pre-Registration Undergraduate Nursing Programmes. Phase 2 Evaluation: Survey of students’ and mentors’ experiences of grading student competence in practice.

Authors: Scammell, J., Heaslip, V., Phillips, J. and Cooper, K.

Pages: 1-59

Publisher: Bournemouth University

Place of Publication: Fern Barrow

Abstract:

Purpose and project aim The purpose of the project was to evaluate the impact of the new practice assessment scheme from the perspective of the users and to make suggestions to key stakeholders for quality improvement where necessary. Phase One of the evaluation involved small numbers and only Adult branch students; whilst the findings were informative, it seemed important to test these out on a wider sample.

The aim of Phase Two of the grading practice evaluation was to explore: • mentor and student experiences of using the assessment tool across all nursing branches • the impact of quality assurance processes

Pertinent literature An overview of the historical and current context of practice assessment for nurses in the UK was provided, including the policy literature from the statutory body. Research on practice assessment in nursing was found to be relatively limited particularly concerning the grading of practice. Key studies include Bondy (1983; 1984) and Hillegas & Valentine (1986) from the USA, Glover et al. (1997) from Australia; from the UK, Burns (1992) and Scammell et al. (2007). All studies had limitations but generally the notion of grading practice was well received by users; however there were concerns about quality assurance processes particularly around reliability as well as issues concerning mentor/practice assessor preparation. In light of an increased national emphasis on nurses’ competency in practice, empirical work that evaluates a new practice assessment initiative is timely. Whilst of value locally in terms of reviewing and improving processes, the findings may also interest colleagues elsewhere.

Project design A questionnaire survey of mentors and year two and three nursing students was undertaken to investigate their experiences of using the grading practice assessment tool. All branches of nursing were successfully targeted for inclusion using convenience sampling, but no learning disability students opted to participate. Mentors were accessed via an annual mentor conference; students were accessed via tutor groups.

112 (86%) of the 130 mentors available, completed and returned the questionnaire. The branch representation was adult nursing (62.5%; n=70), Mental Health (21.4%; n=24), Child Health (17%; n=19) and Learning Disability (1.8%; n=2). Questionnaires were distributed to 210 students; 107 were completed (51% response). The sample comprised Adult branch (72%; n=77), Mental Health (19.6%; n=21) and Child Health students (8.4%; n=9). The response rate broadly reflected the proportion of students enrolled within each branch of the targeted cohorts. Most student respondents were undertaking year 3 of their programme (70.1%; n=73); the exception was Child Health where all students were undertaking year 2.

As an evaluation study, formal ethical approval was not sought; however permission was granted to conduct the evaluation by the Associate Dean for Nursing, and the educational leads within placement areas. There was a project steering group with representatives of all parties including students and mentors; their role was to oversee the project process and to liase with colleagues within their organisations. Quantitative data was coded and inputted into the Statistical Package for the Social Sciences (SPSS for Windows version 15.0); where possible chi-square analysis was undertaken to explore whether the results were significant. Qualitative aspects of the questionnaire were transcribed and a content analysis was undertaken.

Given the sample size, it is not possible to generalise from this evaluation. Others may judge transferability to other settings. In reviewing the findings, a number of limitations must be acknowledged. The most significant limitations are that no Learning Disability students opted to participate and the response rate generally from students was not as high as anticipated (51%). This is however a reasonable response rate for a questionnaire survey but if time had permitted, an extension to the convenience sampling may have been beneficial. Other limitations include the fact that findings were generated from reported practice and, as such, may be subject to distortions of memory.

Findings Five key areas emerged from the findings: questions around mentor education indicated that participation in updates was high although the sampling process may have introduced some bias. Indeed poor uptake was identified in Phase One, reflecting other studies and anecdotal evidence for the programme team. The findings around support for practice education were mixed; mentors appeared to want more support than was available. Student experience varied according to branch – Mental health students feeling most supported and Child Health the least. However the use of neutral grades was high in both mentor and student groups indicating some cause for concern. Perception on use of the profile in terms of whether it was fit for purpose indicated positive responses overall but some confusion from both students and mentors as to the status of the written sections in comparison with skills performance; the latter was perceived as the top priority. The fourth area focused on how grading ‘worked’ and yielded some valuable perceptions. Generally respondents liked and wanted to grade or be graded in practice. Whilst the majority of mentors claimed to use the descriptors and found them useful, the number who did not use them was of concern. There were mixed perceptions of the appropriate use of second marking, some students perceiving that they were given middle grades due to logistical problems in accessing second markers, although this was not supported by the mentor data. The vast majority of mentors expressed confidence in grading students. However a significant minority responded neutrally or negatively or did not answer the question when asked whether they felt confident to fail a student. This finding is important when considered alongside the neutral response to feeling supported. Finally accountability within the assessment process appeared strong, although at times profiles were signed by staff who had not worked with the student. It is unclear however whether these mentors were primary mentors or those functioning as second markers.

Overview of discussion The Phase Two evaluation was broadly supportive of the Phase One findings but extended these to other nursing branches and has provided more useful detail on some of the issues underpinning the grading practice scheme. Quality assurance is a prime issue: the evaluation has shown that reliability of the tool could be improved if the descriptors are clear and robust processes are in place to ensure these are consistently applied. Second marking is another feature of reliability and perceptions varied as to whether this was utilised as much as required and is an issue for programme and placement staff to consider, particularly in the light of inter-branch discrepancies. Students and mentors should be encouraged to constructively make any concerns known. Education around grading and support for mentors was another key area; there perhaps can never be enough support but if any staff have concerns around failing students, this needs to be addressed. The reasons for differing perceptions of support across branches, needs to be explored further. Overall the grading of competency in nursing practice yields far more benefits than problems. It is an innovative scheme, not without challenges; both evaluation phases provide valuable data to focus quality improvement effectively.

Conclusion The evaluation captured the views of mentors and students representing all nursing branches, with the exception of Learning Disability students. The issues raised across the branches were broadly similar, notably that the notion of grading practice was welcomed as a means of valuing the practice element of the programme, although implementation was not without challenges. These are reflected in the focus on the quality assurance processes and in fact this is where the branch experience differs most. Mental Health students felt the most supported in placement, were the most likely to receive grades that required to be second marked (indicating full use of the grading range) and were most satisfied that mentor comments matched the grades awarded. In contrast Child Health students were the least satisfied in all these parameters. Adult branch students gave mainly neutral responses which indicate some underlying issues worthy of further investigation. It can be concluded from the mentor data that experience across branches did not differ markedly. The sample accessed updates and most felt confident in grading practice. However almost 18% did not feel confident about failing students and almost another 20% gave a neutral response to this question, indicating a need for further staff development in this area as well as a review of processes designed to support mentors in making these judgements.

Recommendations • Explore ways of engaging learning disability students in sharing their perspective on the practice assessment tool and processes.

Increase transparency of communication channels between placement providers and the university regarding practice assessment.

- Flowcharts or algorithms for mentors and students might be useful - Include these on the back of each practice profile • Review support mechanisms for students within placements. Identify best practice and include minimal standards on the clinical audit documentation to increase parity between placements and branches.

Investigate peer support schemes for students in practice which include preparation and support for those undertaking this role.
Child Health programme team should review support for students with practice partners to identify specific issues for improvement.
Review level descriptors with students, mentors as well as programme teams to ensure clarity of language and processes.
Re-emphasise the use of descriptors in mentor education as well as student preparation for placement • Review criteria and processes for second marking and emphasise within mentor education.
Disseminate HSC audit of practice profiles on a placement provider organisation basis and include a focus on quality of feedback in relation to grade awarded. • Review current practice around preparing and supporting mentors in failing students and develop an action plan for implementation over the next academic year.

https://eprints.bournemouth.ac.uk/21582/

Source: Manual

Assessment of Practice in Pre-Registration Undergraduate Nursing Programmes. Phase 2 Evaluation: Survey of students’ and mentors’ experiences of grading student competence in practice.

Authors: Scammell, J., Heaslip, V., Phillips, J.E. and Cooper, K.

Pages: 1-59 (59)

Publisher: Bournemouth University

Place of Publication: Poole, England

Abstract:

Executive Summary Context Local and national concerns regarding competency in newly qualified practitioners provided the impetus for the undergraduate nursing programme team at Bournemouth University to radically change the practice assessment tool as part of a curriculum rewrite in 2005. Alongside the introduction of the new practice profile, an evaluation strategy was implemented with two stages; Phase One focused on year one of implementation and reported at the end of that year (2005-6). These findings were used to inform mentor education and to amend the tool for new students and those moving into year two. This phase used a qualitative approach to collect data from focus groups with students and mentors (Adult Branch only), and practice/HEI educators concerning their experiences of using the grading practice assessment scheme. Phase Two utilised the qualitative findings to develop a questionnaire survey. Questionnaires were selected as the research tool in order to access greater numbers of students and mentors from across all branch programmes. Education staff was not surveyed in this phase. Purpose and project aim The purpose of the project was to evaluate the impact of the new practice assessment scheme from the perspective of the users and to make suggestions to key stakeholders for quality improvement where necessary. Phase One of the evaluation involved small numbers and only Adult branch students; whilst the findings were informative, it seemed important to test these out on a wider sample. The aim of Phase Two of the grading practice evaluation was to explore: • mentor and student experiences of using the assessment tool across all nursing branches • the impact of quality assurance processes Pertinent literature An overview of the historical and current context of practice assessment for nurses in the UK was provided, including the policy literature from the statutory body. Research on practice assessment in nursing was found to be relatively limited particularly concerning the grading of practice. Key studies include Bondy (1983; 1984) and Hillegas & Valentine (1986) from the USA, Glover et al. (1997) from Australia; from the UK, Burns (1992) and Scammell et al. (2007). All studies had limitations but generally the notion of grading practice was well received by users; however there were concerns about quality assurance processes particularly around reliability as well as issues concerning mentor/practice assessor preparation. In light of an increased national emphasis on nurses’ competency in practice, empirical work that evaluates a new practice assessment initiative is timely. Whilst of value locally in terms of reviewing and improving processes, the findings may also interest colleagues elsewhere. Project design A questionnaire survey of mentors and year two and three nursing students was undertaken to investigate their experiences of using the grading practice assessment tool. All branches of nursing were successfully targeted for inclusion using convenience sampling, but no learning disability students opted to participate. Mentors were accessed via an annual mentor conference; students were accessed via tutor groups. 112 (86%) of the 130 mentors available, completed and returned the questionnaire. The branch representation was adult nursing (62.5%; n=70), Mental Health (21.4%; n=24), Child Health (17%; n=19) and Learning Disability (1.8%; n=2). Questionnaires were distributed to 210 students; 107 were completed (51% response). The sample comprised Adult branch (72%; n=77), Mental Health (19.6%; n=21) and Child Health students (8.4%; n=9). The response rate broadly reflected the proportion of students enrolled within each branch of the targeted cohorts. Most student respondents were undertaking year 3 of their programme (70.1%; n=73); the exception was Child Health where all students were undertaking year 2. As an evaluation study, formal ethical approval was not sought; however permission was granted to conduct the evaluation by the Associate Dean for Nursing, and the educational leads within placement areas. There was a project steering group with representatives of all parties including students and mentors; their role was to oversee the project process and to liase with colleagues within their organisations. Quantitative data was coded and inputted into the Statistical Package for the Social Sciences (SPSS for Windows version 15.0); where possible chi-square analysis was undertaken to explore whether the results were significant. Qualitative aspects of the questionnaire were transcribed and a content analysis was undertaken. Given the sample size, it is not possible to generalise from this evaluation. Others may judge transferability to other settings. In reviewing the findings, a number of limitations must be acknowledged. The most significant limitations are that no Learning Disability students opted to participate and the response rate generally from students was not as high as anticipated (51%). This is however a reasonable response rate for a questionnaire survey but if time had permitted, an extension to the convenience sampling may have been beneficial. Other limitations include the fact that findings were generated from reported practice and, as such, may be subject to distortions of memory. Findings Five key areas emerged from the findings: questions around mentor education indicated that participation in updates was high although the sampling process may have introduced some bias. Indeed poor uptake was identified in Phase One, reflecting other studies and anecdotal evidence for the programme team. The findings around support for practice education were mixed; mentors appeared to want more support than was available. Student experience varied according to branch – Mental health students feeling most supported and Child Health the least. However the use of neutral grades was high in both mentor and student groups indicating some cause for concern. Perception on use of the profile in terms of whether it was fit for purpose indicated positive responses overall but some confusion from both students and mentors as to the status of the written sections in comparison with skills performance; the latter was perceived as the top priority. The fourth area focused on how grading ‘worked’ and yielded some valuable perceptions. Generally respondents liked and wanted to grade or be graded in practice. Whilst the majority of mentors claimed to use the descriptors and found them useful, the number who did not use them was of concern. There were mixed perceptions of the appropriate use of second marking, some students perceiving that they were given middle grades due to logistical problems in accessing second markers, although this was not supported by the mentor data. The vast majority of mentors expressed confidence in grading students. However a significant minority responded neutrally or negatively or did not answer the question when asked whether they felt confident to fail a student. This finding is important when considered alongside the neutral response to feeling supported. Finally accountability within the assessment process appeared strong, although at times profiles were signed by staff who had not worked with the student. It is unclear however whether these mentors were primary mentors or those functioning as second markers. Overview of discussion The Phase Two evaluation was broadly supportive of the Phase One findings but extended these to other nursing branches and has provided more useful detail on some of the issues underpinning the grading practice scheme. Quality assurance is a prime issue: the evaluation has shown that reliability of the tool could be improved if the descriptors are clear and robust processes are in place to ensure these are consistently applied. Second marking is another feature of reliability and perceptions varied as to whether this was utilised as much as required and is an issue for programme and placement staff to consider, particularly in the light of inter-branch discrepancies. Students and mentors should be encouraged to constructively make any concerns known. Education around grading and support for mentors was another key area; there perhaps can never be enough support but if any staff have concerns around failing students, this needs to be addressed. The reasons for differing perceptions of support across branches, needs to be explored further. Overall the grading of competency in nursing practice yields far more benefits than problems. It is an innovative scheme, not without challenges; both evaluation phases provide valuable data to focus quality improvement effectively. Conclusion The evaluation captured the views of mentors and students representing all nursing branches, with the exception of Learning Disability students. The issues raised across the branches were broadly similar, notably that the notion of grading practice was welcomed as a means of valuing the practice element of the programme, although implementation was not without challenges. These are reflected in the focus on the quality assurance processes and in fact this is where the branch experience differs most. Mental Health students felt the most supported in placement, were the most likely to receive grades that required to be second marked (indicating full use of the grading range) and were most satisfied that mentor comments matched the grades awarded. In contrast Child Health students were the least satisfied in all these parameters. Adult branch students gave mainly neutral responses which indicate some underlying issues worthy of further investigation. It can be concluded from the mentor data that experience across branches did not differ markedly. The sample accessed updates and most felt confident in grading practice. However almost 18% did not feel confident about failing students and almost another 20% gave a neutral response to this question, indicating a need for further staff development in this area as well as a review of processes designed to support mentors in making these judgements. Recommendations • Explore ways of engaging learning disability students in sharing their perspective on the practice assessment tool and processes. • Increase transparency of communication channels between placement providers and the university regarding practice assessment. - Flowcharts or algorithms for mentors and students might be useful - Include these on the back of each practice profile • Review support mechanisms for students within placements. Identify best practice and include minimal standards on the clinical audit documentation to increase parity between placements and branches. • Investigate peer support schemes for students in practice which include preparation and support for those undertaking this role. • Child Health programme team should review support for students with practice partners to identify specific issues for improvement. • Review level descriptors with students, mentors as well as programme teams to ensure clarity of language and processes. • Re-emphasise the use of descriptors in mentor education as well as student preparation for placement • Review criteria and processes for second marking and emphasise within mentor education. • Disseminate HSC audit of practice profiles on a placement provider organisation basis and include a focus on quality of feedback in relation to grade awarded. • Review current practice around preparing and supporting mentors in failing students and develop an action plan for implementation over the next academic year.

https://eprints.bournemouth.ac.uk/21582/

Source: BURO EPrints