Assessing Competence – Best Practice

This video has been created to share with you a selection of the best practices for building a portfolio of evidence to confirm an individuals competence. It should prove particularly useful for companies working on the certification regime when confirming individuals fitness and propriety ,

Let’s start with the four stage assessment process. This can be used with most forms of assessment and fits nicely with the requirements of the regime certification regime.

Many people believe that the first stage of the process is observation. Observation is one of the four methods of assessment which we will touch on later in this video. The O stands for objectives . Before we start with any assessment we have to know the reasons why we are undertaking it. When faced with this question under the certification regime it is highly likely that the most common answer relates to confirming that an individual can perform the certification function to the required standards. In other words they are competent. You can go further than this by slightly widening your objectives of the assessment.

Anyone who holds a certificate issued by their company should be proud to hold it. It is likely that individuals will consider the process that they had to undertake to be granted a certificate and compare this to individuals in a similar position with other companies. Imagine how the certificate Holder will feel if they know that their certificate as well as representing the fact that they can perform the certification function in question that they are also regarded as an ambassador of your firm’s culture a champion of your firm’s values an advocate of quality and an exemplar of customer service.

This Will create a real buzz for the individual in question and act as a great source of motivation. For a company such a process could be used as an excellent marketing and recruitment tool as well as providing a real boost to enhance consumer confidence. Whatever the objectives of the assessment happen to be it is important that you have defined the success measures. These will help ensure that you can monitor whether or not you have achieved the objectives in question.

Once the objectives of the assessment have been agreed the next stage is to gather evidence. The letter R stands for record. There are four main methods of assessment. Can you guess what they are? These are test, observe ask questions and data.

When it comes to testing people the use of external appropriate qualifications and CPD systems are very useful. However these qualifications will not always cover off knowledge and understanding of in house policies processes and procedures. If you are designing your own tests to be used in house it is important to ensure the questions you design are unambiguous and objective. Best practice would recommend that you pilot any short question test with another person or small group of people before you release it to a wider audience. This is to ensure the interpretation of your questions is consistent across the population.

Another way of ensuring consistency is the use of multiple choice questions. When creating your own tests it is important for the individual to understand the consequences of not meeting the required standard. For example the test pass mark might be 80%. What happens if the person fails to achieve this level? You should also ensure that the content upon which the test is being administered in relevant and current to the individuals role they perform. When you are designing tests to be used remotely you should ensure that you have processes in place to authenticate the fact that the name on the test paper corresponds to the person physically taking the test.

The second method of assessment his observation . Here is a very simple exercise at this stage to demonstrate some of the challenges with conducting observations . If you are wearing a wrist watch -without looking place move your wrist so that the watch is out of sight. Can you describe the watch? . What writing is on the face? Does it have a day date function, does it have numerals or symbols? What other things are shown on the face? Most people struggle to describe their watch accurately. Why is this? Think about the reason you look at your watch . On most occasions you are checking the time.

You quickly glance at the watch lift the information you seek and move on. When it comes to describing the wider features of the watch you struggle. As an observer you must be mindful of this characteristic and not drift into a state where you cannot concentrate on the observation. You should make sure that you are capturing appropriate evidence throughout the observation.

The next point to cover under observation is to ensure that as an assessor you capture objective rather than subjective evidence. Take a look at these examples that have been included in observation aids Completed by supervisors when working with advisors. Do you think each statement is either a fact or an opinion. Let’s look at the answers he led the client in his questioning is an opinion . In order to make it factual we would need an example of one of the questions asked by the adviser. He told the client to cancel the existing policy is a fact. He was very creative in his sales approach is an opinion .

We need an example of what the advisor actually did or said to create this opinion. In front of the client he analysed the information very thoroughly . Again this is an opinion what did the advisor say or do that led to this conclusion he answered the money objection with the appropriate sales aid. This is actually an opinion as it does not clarify what the appropriate sales aid is. If on the other hand the standards that the supervisor used to determine competence clearly indicated the reference number of the appropriate sales aid then this would count as a fact the adviser pointed out that April was a good month for pension planning .

This is actually a fact . The advisor is offering his opinion that April was good month but it was a fact that he said it.

People often ask how much evidence should be captured during an observation. It can prove difficult to capture everything that the advisor says or does and how the customer reacts. The key parts to capture are all the points in the conversation where the advisor has done things well or has said or done something that may mean they have failed to reach the competent standard. You need to concentrate on not only the words that were said but the tone in which they were delivered and the reaction of the customer to hearing what was said. These are all relevant points in the communication process.

The third method of assessment is A which stands for asking questions. This is different to the questions that may be asked in a written test. You can ask questions at anytime but two of the most common occasions are during an interview or at the end of an observation to ascertain whether the person could have dealt with a slightly different scenario had it happened during the event . During interviews the most common type of question is a criteria based question which asks the candidate to answer with specific examples of what they said or did, in other words provide evidence.

At the end of an observation had the customer not raised any objections during the meeting it is perfectly reasonable for the supervisor to ask a question of the advisor to explain how they would have reacted had the customer raised a particular objection at a given stage of the process. Their answer is admissible as evidence towards the advisors overall competence. This is a recognised and acceptable practice used by vocational qualification assessors.

The final method of assessment is data. Most commonly this is making use of key performance indicators to monitor an individual’s performance . It goes without question to say that these indicators must be regarded as such . You must be prepared to investigate further any adverse indicators and not take all of them at face value despite the fact that they are not presenting any concerns.

There are several different categories of key performance indicators . Many firms seem to restrict the use of indicators to measures of quality and quantity. Firms are encouraged to widen the range of indicators used to embrace such things as leading, lagging, directional and outcomes. These are just a few of the categories you can use to provide a wider and deeper look at activities being undertaken. There are more types of indicators available.

Having identified the four main methods of assessment it is time to look at the types of evidence that can be gathered during the recording phase. There are three main types of evidence.

The first with these is known as primary direct evidence. This category of evidence carries the largest evidential weight which means you need less of it to make a judgement over the persons competence. The reason being is that this evidence is actual work performed by the individual themselves. This could be paperwork relating to a case that they have written or a report or an observation of an activity that they have performed. With this category of evidence more supervisory resource is needed to collect it but as I have said you need less of it.

The second category of evidence is called supporting direct. This is completed by the individual being assessed what is not actual work. It is made up of aspects items such as tests , case studies, role plays and other and any other activities the individual completes which are not day to day real work. There is much debate about the effectiveness of using role plays to determine an individuals competence especially when the assessor is also performing the role of the customer. Supervisors can be easy customers, raising no objections and to a certain extent answering questions from an awareness of the process and the advice and products being discussed.

Many firms have chosen to use role plays as it proves to be more convenient than arranging observations with advisors meeting customers off site. One way of encouraging firms to use live cases whenever possible is to ensure that the role play itself is well structured, quite technical and difficult to deal with and not therefore regarded as a bit of a walk in the park. Maybe the live call is better for the adviser and regarded as potentially easier. This requires time to prepare appropriately and sometimes the need to find a third person to act as the customer.

The final category of evidence is known as supporting direct . This is evidence about the individual. The most common supporting direct evidence used are key performance indicators which include aspects such as upheld customer complaints. This category would also embrace items such as customer feedback or customer compliments. It will also include reports about performance on courses or items such as 360 degree feedback.

Having looked closely at evidence within the recording phase let’s return to the four stage assessment process and move on to the letter C which stands for classify. At this stage in the process you will have gathered an amount of evidence which you need to match against the standards that relate to the individual’s competence. One of the challenges in following this process is to resist the temptation to classify evidence as you go. In situations where supervisors are observing individuals the observation aids that are used often steer them into this practice which is not reflective of best practice.

The best way is to use a blank sheet of paper and then use the evidence against the standards that you are using.

The standards that you use within the classify part of the process should cover all aspects of the assessment you are trying to measure. You need to insure that as far as possible your standards are both objective and measurable.

The final of the four stages in the assessment process is E which stands for evaluate. Quite simply based on the evidence against the standards is the individual competent or not. You will need sufficient evidence to make an informed judgment. On most occasions performing a task once is not sufficient evidence to confirm an individuals competence. See if there is an opportunity to gather more evidence in a slightly different scenario rather than repeating exactly the same thing. You may encourage activity management which I will touch on later.

Evidence itself has a shelf life. It is unlikely that any evidence gathered in respect of assessing the competence of an individual will be valid after 12 months. Depending on the frequency of the task performed by the person being assessed governs how recent the evidence needs to be to form a view about their competence . For example if the person being assessed performs a task every week it is reasonable to expect that the evidence being used to confirm their competence is recent. If on the other hand the evidence is of a task the individual only performs once a year that is not unreasonable to allow the evidence to be used providing it is less than 12 months old.

It is best practice to use real work to build your portfolio of evidence using case studies, tests and role plays to fill gaps.

It is now time to turn our attention to the three main rules of assessment. These are the rule of transparency, the rule of reliability and the rule of validity

The FCA published the stock take report in August 2019 . It highlighted several findings from their examination of the SM& CR regime in a sample of banks and building societies. By following the rules of assessment you will be able to address the issues raised by the FCA in their report .

The rule of transparency states you should make sure that you have standards that define competence that are clear measurable complete understood and accepted by the individuals you are assessing. To be complete the standards must not only cover the technical skills but also the expected behaviours you would wish to evidence. You would be expected to demonstrate how the competence standards linked to your own corporate values and culture as well as the FCA says own code of conduct

the rule of reliability applies when you are in a situation whereby you have several supervisors assessing the competence of others. You will need to ensure that you have mechanisms in place so that each assessor is broadly operating in the same manner. In other words if you were to swap the supervisor you would expect them to be able to draw broadly similar conclusions of another persons competence similar to that by the previous supervisor. Many companies in dealing with this rule will apply either some form of validation exercise by assessing a video or a sample case to help ensure consistency of the application of competent standards.

The third rule applies to ensure the method of assessment is appropriate to the nature of the competent standard you are assessing. You would be expected to evidence the skills of the individual as well as the technical knowledge. In most situations an assessment of skill is assessed through a demonstration rather than a paper based exercise such as a case study. You will need to think about best ways to gather the evidence against competent standards that you are measuring and ensure that it is appropriate the task in hand.

Before we move on to best practice on how to build a portfolio of evidence it is worth visiting some of the common assessment traps assessor’s can fall into when performing their role. To demonstrate the first of these find a piece of paper write down your immediate response to the following questions

• What is your favourite colour
• Write down the name of a piece of furniture
• Select a number between one and five
• What is your favourite type of flower? (not self raising but a plant)
• Name a type of animal you would find in the zoo

Now let’s look at the answers you recorded. Did you have any of the following answers. Red is your colour chair as your piece of furniture #3 as your number rose as your type of flower and lion as your animal

There is nothing to worry about if I have guessed any of your answers. My answers are based on the favourites of some 10,000 people in an exercise that was conducted . You share your favourites with many other people. What is the point of this? What we have just performed is a predictive behaviour test where I am trying to guess the result before I undertake the assessment. It is based on other experiences and previous performances . It is important as an assessor you enter any assessment with an open mind and a blank sheet of paper . You should not be trying to predict the outcome of what happens .

This behaviour is known as Halo and horns. Imagine the situation where you are going out with one of your favorite advisors to conduct an observation . You get on really well with the advisor you look forward to spending time in their company and they are a good performer. You are likely to forgive any minor indiscretions when it comes to meeting compliance standards or competent standards that are used within your business. You know in your mind that this was just a minor indiscretion and will not happen in future. You are therefore tempted to pass the individual because you are wearing the Halo.

Conversely you might be spending the day with another advisor who is struggling in your mind to perform to the required standards and regularly makes mistakes. You are entering this particular observation with the mindset that this advisor will make a mistake and will fail. They are almost condemned to failure as you will mark them down at any opportunity.. You are wearing the horns and the advisor will fail. As I just said it is important to enter any assessment with an open mind not the preset agenda

in the late 1920s early 1930s professor Elton Mayo was involved in conducting a series of experiments with a group of women working on a Telegraph relay assembly line in the Hawthorne works of the western electric company in Chicago . He was trying to work out the changes that occur by changing a number of the working conditions in which the women were working . One of the key findings that came out of this study is referred to as the Hawthorne effect. When someone is being assessed and that they know they are being assessed their performance changes .

As an assessor it is important that you realise the impact you have on being present. Where you position yourself in the room, how you take your notes all have an effect on the person being assessed. To give you an example of how this works think about a scenario where I have arranged for a driving examiner to accompany you when you next drive your car . The examiner will be measuring your performance against the current standards needed to pass the driving test . My question is would you change the way in which you drive with the examiner in the car .

Most people faced with this scenario admit that they would need to change the way in which they drive – trying to remember the position of the hands, mirror signal manoeuvre etc

There is a further explanation for this behaviour coming from a model known as the learning step ladder. You start off as being unconsciously incompetent you don’t know what you don’t know . Somebody then tells you and you become consciously incompetent and require training. Having received the training you can do the task but you have to think about the way in which you do it to meet the standards required In other words you become consciously competent. Over time and performing the task for many times you can almost do it without thinking in fact you become unconsciously competent until the assessor walks in the room.

At this stage you immediately revert back to thinking about the standards that you need to meet and become consciously competent rather than unconsciously competent. As an assessor working with people on a regular basis you must be mindful of the impact you have on the people you’re working with.

The final area to look at is the effect of paradigms.

Here is a picture that you may have seen before but at this stage I would like you to look at the picture and decide what you see. Some people can only see a young lady others could I receive an old lady. Other people can see both. Stephen Covey in his book the Seven habits of highly effective people described paradigms as being formed by our life experiences growing up that create a view of the world which we see through our own lens of experience. What we see and the way we interpret what we see influences the way we respond. You can’t see paradigms but they are there.

So for our example if you can only see the old lady your response may have been different to the one had you only seen the young lady.

Stephen Covey in his book provides a couple of good examples of paradigms in action and how a paradigm shift might occur. This is something that happens when you realise your view of the world is somewhat different to the reality of what is happening . Let’s take a look at one of his examples. Imagine you are the captain of a battleship Out on manoeuvres. It is late at night it is very dark. You are on the bridge when a shout goes out light bearing off the starboard bow . As captain your immediate question is the light stationary or moving . The response comes back stationary captain.

As captain you order to signal that ship that we are on collision course and to change course 20 degrees. The signal goes out a response comes back. you change course 20 degrees. This ruffles the feathers of the captain who demands the 2nd signal goes out stating that I am the Captain of this vessel you change course 20 degrees. The reply comes back I am an able Seaman second class you change course 20 degrees . At this stage for captain barks an instruction for a signal to be sent tell him that this is the SS Missouri of battleship the largest vessel on the ocean we are on collision course you change course 20 degrees .

The signal goes out a reply is received we are a lighthouse . The captain realises that the view of the world as he sees it through his own lens of experience is wrong . At this stage a paradigm shift occurs and he has to take evasive action. From an assessment point of view it is important for an assessor to remember that what they see and hear is their version of what has happened . It’s always worth exploring how the other person who is being assessed saw and interpreted the situation that was going 0n. You never know they might have seen something or interpreted something differently to yourself.

You do not have so right on being correct.

The final part of this video looks at best practice when it comes to building a portfolio of evidence . The idea is that we have to build a portfolio using a mixture of primary direct supporting direct and indirect evidence to satisfy ourselves that the individual assessed is competent . The portfolio of evidence is represented by the blue box. In building our portfolio of evidence we must follow the rules of evidence. Let’s take a quick look at those rules – they will sound familiar.

The first rule is the rule of sufficiency we must make sure that we have enough evidence to make a judgment as to whether a person is competent. In many instances once is not enough to satisfy ourselves that competence has been achieved. The rule of objectivity requires us to state facts not opinions . We are looking for things that the person said or did together with the outcome rather than an opinion of how they approached a particular task . The rule of clarity Requires us to ensure that the person being assessed is aware of the fact that an assessment will be taking place, the passmarks involved and the consequences of failing to meet the required pass mark.

The role of currency tells us that evidence has a shelf life and any content we test should be relevant to the employees current role. The rule of validity requires us to ensure that the evidence we are gathering is appropriate to the standard we are trying to measure, Remember to evidence of skill is likely to require a demonstration rather than completion of the case study, Finally the rule of authenticity requires us to ensure that we have an audit trail between the person performing the task and the evidence that we are using. For example if the person is taking a test how do we know it was them and not a colleague.

Having looked at the rules of evidence it is now time to look at the work involved in building a portfolio of evidence successfully . It is possible that an assessor can complete a range of activities attempting to fill the portfolio with the required evidence only to fall foul of expectations because they are demonstrating the characteristics of an activity manager rather than a competence manager .

Let me share with you an example of how an activity manager might approach building a portfolio of evidence. This is in respect of a financial adviser but the points apply equally to any assessor working on a portfolio. In many T&C schemes there is a requirement for a supervisor to conduct a regular observation of an advisor. If you look at the observations that have been conducted over the previous 2 years you would expect to see a variety of scenarios assessed . For example there would be observations for pensions, investments, savings, mortgages and protection assuming the advisor is authorised in all these areas.

If an area has been observed more than once you would expect a slightly different aspect of the advice area to have been assessed. For example within the investment area the first observation was based on investing a portfolio for income but the second investment case was to mitigate inheritance tax liabilities. Gathering evidence is all about the breadth and depth of the advisers role.

Unfortunately when it comes to the activity manager they give no thought as to the reasons behind performing the observation . They regard it as an activity that must be completed . When you analyse the pattern of observation activity that they have undertaken you find that the observations have been conducted in the same advice area. In some cases the problem is compounded by the fact that the observations conducted within the one investment area are all of a similar nature . So instead of adding new evidence to the portfolio you are simply replacing evidence that you already have.

Further traits of activity managers all the way in which they select cases for checking. Activity managers tend to pick the more straight forward cases that require less time to check and avoid the ones that look more complicated . If they operate out of call centres they will tend to look at the length of the call to be assessed and decide from there whether or not to actually perform one. They tend to choose the shorter more simple calls .

Activity managers will also concentrate on performing the minimum amount of activity required by a tnc scheme and rarely exceed these requirements. They will complete the necessary forms but tend to record only the minimal amount of acceptable evidence. There is also a tendency to create what can be called cut and paste development plans where the same development theme is applied across a range of advisors.

On a more positive note we have competence managers who think carefully about how they build the portfolio of evidence and select opportunities that contribute to the accumulation of evidence. Before undertaking an observation they would try to establish whether or not there was an opportunity to gather new evidence of an advisors performance rather than repeating the same observation. Competence Managers will also check beyond the minimum requirements and ensure that their evidence or note taking is full and informative. The competence manager will vary their checking routines so that advisors cannot predict which aspects of their work get checked.

Development plans are tailored to individual needs. Finally the competence manager will be prepared to go that extra mile to get things right – if another observation is required they will do so,

That just about concludes our journey through assessing competence . It is worth having a brief recap to see what we have covered. We have looked at the four stage assessment process objectives, recording, classifying and evaluating within that we looked at the four methods of assessment the testing, observation, asking questions and data. We’ve also uncovered the rules of assessment that relate to transparency reliability and validity as well as the six rules of evidence . We’ve had a look at the assessment traps including the Hawthorne effect and paradigm shift and how you approach building a portfolio of evidence to avoid being an activity manager.

I hope you found this useful if you do have any questions please do not hesitate to get in touch