How to write dumb knowledge questions

I’m writing this post in the hope of preventing RTOs from having to pay me (or others) to fix dumb assessment tools. I’m focusing this post on knowledge questions. I’m going to provide an example of a poorly framed and invalid knowledge evidence question and explain why it is so dumb.

Let’s start with the KE requirements from the unit SIRXCEG003 Build customer relationships and loyalty.

Lets unpack what the KE actually requires….

Lets focus on the area I’ve highlighted. What evidence do we need to be assured of the learner’s knowledge of these items?

What I read is that the student needs to demonstrate their knowledge of a number of requirements that specifically relate to his/her organisation and/or industry. For example:

• Evidence that they know/understand their own organisation’s service standards
• In some cases there may be an industry service standard that needs to be met. For example, in a food retail environment, there are food safety standards that employees must adhere to when providing service to customers (e.g. having clean hands, not selling unsafe food items etc). If that’s the case, the student needs to understand those standards.
• The types of needs and expectations that customers of the business (or industry) have. For example, the expectations of customers shopping for a handbag at Louis Vuitton are likely to be quite different to customers who are shopping at Discount Dave’s. The student needs to show us they recognise the needs and expectations of customers in thier industry sector and their retail store.
• the basics of their own organisations customer database and how to use it

There are more evidence requirements obviously, but the list above begins to draw a picture. The most important thing to understand here is that the student needs to demonstrate knowledge of all of these things in the context of their own organisation, and where appropriate, retail industry sector.

In plain English, the knowledge they must have is about their own job. If they don’t have a job, the assessment will need to simulate a business so the student can demonstrate this knowledge based on that simulated business.

To demonstrate this knowledge, they’ll need to read and interpret/explain their organisations service standards and customer service policies/procedures. They’ll need to demonstrate their understanding of the products and services offered by their organisation and how these benefit the specific customer cohort. They’ll need to demonstrate that they know what the customer database system does, and the basics of how its used.

And we, as assessors, need EVIDENCE of this knowledge so we can make a decision about the learner’s competence.

What the KE doesn’t require

Here are a few things that students DO NOT need to be assessed on:

• How to get this knowledge
• What a service standard is
• What a customer database is
• Needs and expectations of all customers (we only need to know about customer service needs and expectations of the customers of the specific retail business or retail sector.)

All of the things above are underpinning knowledge, so the learner does need to know this stuff, but they do not need to be assessed on these items. Formative learning might include some quizzes that check this knowledge as part of a learning program, but none of this knowledge is appropriate for a summative assessment.

Let’s look at a poorly framed question

Here is an example of a poorly framed question that will NOT draw out valid evidence.

To build customer relationships and work effectively on behalf of your organisation you will need to understand:

• industry and organisational professional service standards
• the attitudes and attributes expected when engaging with customers
• the range of different customer service needs and expectations that are applicable to the organisation for which you work
• the benefits to customers and to the organisation of building good relationships and supporting loyal customers
• the essential features and use of customer databases (if they are applicable to our organisation).

How will you acquire this understanding?

Soooo….what is this question going to draw out? If it was me being asked the question, I’d explain things I’ll do (in the future) to learn this stuff. For example, I’ll read the company service standards, I’ll talk to colleagues to find out about the types of things customers are likely to need or expect. I’ll use the help function on the customer database and complete any work-based IT training programs that are offered to me.

What part of that answer would provide my assessor with evidence that I have the knowledge required by the training package?

NONE OF MY ANSWER!

This is why the question is dumb

This question is just plain dumb. Here are a few reasons why:

1. It will never draw out the knowledge evidence requirements, so it will not produce valid evidence.
2. Its actually asking how I will acquire the knowledge, instead of having me demonstrate that I already hold the knowledge – which is what the training package actually requires!
3. Even if the question was valid, the chances of a Certificate III level trainee answering this question in a way that covered off on every single dot point sufficiently, is probably fairly remote
4. There is no guidance as to how in-depth an answer a student should provide. Would one sentence be enough? Can I answer in dot points?

I could keep going, but I won’t, because I now need to get back to fixing a pile of assessment tools (from the same source) and creating a whole lot of additional assessment questions and tasks so that the assessor can make an assessment decision based on valid evidence.

Poor tools, poor result

RTOs can choose to buy (or build) rubbish assessment, but it will always catch up with them in the end. Be assured, the cost of these issues being picked up at audit, and/or having to pay someone to fix the issues, far outweighs the savings you thought you made by buying (or building) cheap assessment tools.

I get it’s hard to find good tools, and I get it’s a big investment, I’ve owned and run RTOs myself, and the reason I learned to write tools is because I couldn’t afford to keep wasting money on tools that didn’t fit my needs or just were not written well.

Not every RTO owner or manager can write their own tools, but make sure you have someone on staff who can recognise good tools against bad ones before you invest. Don’t buy tools based on price, and don’t have the false expectation that any assessor can write a compliant, usable tool.

Coleen