How to create assessment tools….post validation rant

I know, you’ve got your TAEASS502, so you’re qualified….

But seriously, assessment tool validation makes me cranky. Sometimes I think we just make life easy for ASQA auditors. Every tool I’ve validated this month I’ve been able to find a critical non-compliance within 1 minute. That is not cool!

So you can quit reading this now, and go directly to page 57-58 of the User’s guide to the Standards for Registered Training Organisations 2015… or you can hang around for a cranky, but plain English explanation

1. Use comprehensive checklists to ensure all training package requirements are addressed and assessment is conducted in accordance with the principles of assessment and the rules of evidence.

We all know that you use a checklist when conducting an observation assessment, but that’s not the ONLY time you should use a checklist.

If the task requires the student to write a report, essay, portfolio, journal or any kind of document, you need to have a checklist to ensure that all of the required evidence is contained AND assessed.

The same applies if the student is submitting any physical evidence, for example a cake they baked. You’d have a checklist to assess the different components of the cake. Its presentation, its texture, its taste…whatever the criteria of the unit requires

If you’re writing assessment tools that don’t have specific benchmark answers (ie written tasks, case studies etc), then you need a checklist!

It’s that simple

2. Provide clear guidance on ‘what the assessment task is’ to both the assessor and the student—so that both assessors and students know what is to be assessed; the context and conditions of assessment; how and when assessment is to occur; and the environment for the assessment.

Make sure your instructions are crystal clear on how the assessment will happen. Even if its just a written knowledge test, the student and assessor need to know exactly what the requirements are, when and where the task will occur,  how it will happen, whether its open book or not, will it be in the classroom, online, in the workplace or homework?

If it’s a role play or workplace observation, then you need to make sure the assessor knows exactly what needs to be in place to set up the assessment:

You should write implementation instructions explaining to the assessor what the need to do to prepare for the assessment (gather documents, organise with the workplace, set up of the simulation area, how equipment or facilities should be arranged, who will play what role, what the actors in the role play should do and say etc)

Obviously, the student needs to know what will happen too. They need to understand what their role in the role play is, what criteria you will be assessing them against, what they are expected to ‘do’.

3. Ensure tasks for observation of a student’s practical skills are well described and include observable behaviours. Do not, for example, simply cut and paste information from the unit of competency such as the performance criteria.

Firstly, lets define ‘observable behaviours. In plain English, if you can’t see it, it’s not observable.

You cannot observe someone “understanding the procedure” (unless you have mind reading powers). You cannot observe someone ‘interpreting’ either.

And see that underlined bit? That DOES NOT mean rewording the PCs will cut it!

Lets take a look at a skill most of us have….making a sandwich.  And many of you will know, there is a unit of competency for this skill

Take a look at the unit here https://training.gov.au/Training/Details/sithccc003

Now if we want to write an observation task for this, we need to write a checklist with observable skills.

Lets look at element 2 of this unit

Make sandwiches.

2.1.Use safe food-handling practices to hygienically prepare sandwiches based on requirements. 2.2.Cut ingredients and completed sandwiches to ensure uniformity of presentation. 2.3.Combine ingredients appropriately based on flavour combinations, customer preference and standard recipes. 2.4.Use toasting and heating equipment safely, as required. 2.5.Make sandwiches in a logical and sequential manner while minimising waste.

To make this a simple and observable checklist, we simply write a checklist of the actual observable behaviours we need to see from the student.

• Washes hands with clean water and soap
• Sanitises bread board or counter
• Lays bread out
• Butters bread in a consistent way
• Cuts cheese slices evenly
• Cuts ham slices evenly
• Places cheese on bread
• Places ham on bread
• Places bread slice on top of ham
• Makes sure cords and sandwich maker are place safely on the counter
• Picks up sandwich and places into sandwich maker
• Closes sandwich maker
• Opens sandwich maker to check progress
• Removes sandwich when it is sufficiently cooked

No matter who the assessor is, this checklist will ensure that every single assessment decision will be the same.

Its reliable, because its observable, and there is nothing here to debate.

If you’re rearranging wording of PCs and PEs to write your checklists…you’re doing it wrong!

4. Have processes in place for assessing the practical application of knowledge and skills.

• Conduct observations of the student to allow them to demonstrate the practical application of their knowledge and skills.

Every assessment tool should include an observation of some kind. The end.

• Avoid overuse of document-driven knowledge assessment.

To assess ‘bake a cake’ we don’t need the student to write an essay on good cake baking. They don’t need to do case studies about cake baking. They don’t need to write a cake recipe either.

There will be some knowledge requirements, but the truth is, even those don’t need to be written. You could verbally assess.

There is nothing wrong with written assessment, but do not overdo it. There is no need to make students write copious documents and assessment tasks for every unit.

This is about competence. Is the person able to bake a cake or not? Yeah they need to know what the utensils are, and maybe different types of cake recipes, but they don’t need to write an essay (or a recipe)!

Here’s another cut and paste (page 53 of the user guide). ” Ensure skills are assessed by observing the student actually carrying out the relevant task in an appropriate environment. It is not appropriate to assess a student’s ability to carry out a task by simply requiring them to describe how they would do the task.“

DO NOT WRITE QUESTIONS LIKE “HOW WOULD YOU….” and imagine that this is application of skill. IT IS NOT!

• Ensure observation instruments include an assessment of a student’s knowledge—for example, by asking the student why they are doing something and ensuring they are not just copying what has been shown to them.

As I mentioned, verbal questioning can be used to assess knowledge. But even if you do a written knowledge test, YOU MUST ensure your observation task includes some verbal questioning.

Why are you mixing the batter that way? Why is it important to heat the oven first? What will happen if you don’t mix the ingredients well?

Those questions confirm that the student isn’t just baking a cake by rote – it confirms they ‘get’ the process. That they understand why it is that they take the steps they do.

MAKE SURE YOUR OBSERVATIONS INLCUDE VERBAL ASSESSMENT

5. Clearly define benchmark criteria for each task to be performed, to allow assessors to be able to make a judgement about a candidate’s competency. Again, these should not be cut and pasted from the unit’s performance criteria.

There is a reason ASQA felt the need to state the cut and paste thing twice. Its because it happens so often. STOP IT!

Benchmark criteria is often confused with ‘example answers’. For may questions, there is a right or wrong answer that can be put into a marking guide (for example questions about legislation).

But many responses won’t have set answers. For example, most units require students to understand and apply organisational procedures. These will vary in every industry and organisation. If the student needs explain a procedure, your ‘example answer’ may not be correct in the context of the student’s workplace.

That’s why ASQA use the word ‘criteria’.

A criteria is the standard by which something may be judged or decided. So what is the criteria for a satisfactory response?

The criteria may be that the student’s response aligns with the organisational procedure (that the assessor has reviewed to confirm alignment).

And by the way, if the assessor has to review to confirm alignment, you better make sure you’ve put that in the instructions for the assessment.

Use criteria as your benchmarks. Yes you can provide example answers, but establish a criteria so that the assessor is crystal clear on what standards the students response (or demonstration of skills) must meet.

And I’m not finished…..

Observations….So you’ve written ‘demonstrated effective communication’ in your checklist. What the hell is effective communication? My interpretation and yours may be very different – so set a criteria!

For example: Student used paraphrasing to confirm understanding, student spoke clearly in plain English, student was easy to understand, student used eye contact and body language to demonstrate to the audience they were listening.

That’s criteria, and its observable!

6. Make sure assessments enable a student to consistently demonstrate competence. Assessment should:

• be practised in multiple situations

That’s pretty simple. The student must be able to demonstrate their skills in different contexts, environments and situations. Observing the skill once wont cut it

• be practised over time

I still don’t understand how some units still get delivered (in entirety) in 1-2 days, but I can tell you, there is an extremely high chance that the evidence collected isn’t sufficient. It would be surprising to me if the assessment evidence was able to show consistent performance over time (buy hey, congrats to the assessment writer and assessor if it does – you’re way smarter than me). And I’d love to see the (compliant) justification for Amount of Training in those TAS – that’s a kind of magic beyond my understanding.

• incorporate the provision of feedback.

Make sure that your assessment tool provides for feedback. Don’t just have a feedback bit at the end (well do, but do more). If there is a role play or observation, include a debrief into the instructions. Students need feedback, give it to them as often as you can.

7. Completed assessment records demonstrate what the candidate did, what criteria were covered and on what basis the assessor determined the candidate as competent.

This is fairly easy to do with written tasks, but this is VET, and we should be doing practical assessments often.

If the assessment will be conducted in the workplace, then you must ensure the assessor has an area to document what actually happened.

• Where did the assessment take place?
  • What kind of workplace?
  • Who was there?
  • What equipment was used?
  • What actually happened? What did the student DO?

I create a section in each work based observation for the assessor to describe the context and environment. All of the stuff above.

ASQA want to see exactly what took place in the assessment evidence. If that forklift student was assessed, make sure the instrument records the type of forklift, the environment (in a warehouse, outside, on a construction site?), what did they move? Make sure all of this is able to be recorded in the tools you create.

8. Check that assessment criteria in assessment tools address foundation skills—for example, oral communication and ‘interaction with others’ skills.

So often forgotten….. Make sure your assessment is addressing foundation skills. This is not optional!

And lasty, this one is from me.

9. Just because you mapped it, doesn’t make it so

Don’t map written questions as meeting assessment conditions that require a real or simulated workplace (with facilities and equipment). Come on. The student sat there with pen and paper (or maybe keyboard) and answered the questions. They didn’t use PPE or workplace facilities to do that.

And just because the word in the question is the same as a word in a PC/PE doesn’t make it valid. You’re not fooling anyone.