Selection or draw?

After the abolition of the lottery in 2017, there has been talk for some time now about the possibility of allowing lottery again as a selection method for programs with a ceiling on student numbers. Various arguments for reintroducing lottery are presented in the media, for example that it would be fairer. Is that always the case? What can we say about the pros and cons of selection and lottery?

The scientific insights about the advantages of selection compared to lottery are not unambiguous. It is important to realize that the ways in which selection is carried out are very diverse, while drawing lots is carried out in the same way everywhere. That seems to make the choice of lottery easier. Selection can indeed be a complicated process, but it also has a number of advantages. PhD research by Sanne Schreurs, for example, shows that the selection in Medicine at Maastricht University is predictive and cost-saving and does not disadvantage minorities[1]. Below we describe what NOA means by a good selection.

A good selection

1. The best student?

The underlying idea of selection is that you select the 'best' student for the limited places that are available. However, what is meant by 'best' can be a broad concept. Is it the student who can study best? Is it the student who does well and suffers little delay? Or is it the student who ultimately becomes the best professional? If it is clear exactly what the 'best' is, then it is important to realize that this is most likely not one profile. Certainly not if different specialisations are also included within one study. A surgeon ultimately needs, at least partially, different competencies than a psychiatrist.

For example, in the Medicine selection of Maastricht University, the desired final level of doctors is looked at and selected on this basis. So it is not the characteristics of a good student that are looked at, but the characteristics of a good doctor. They select on seven competencies, including professional behavior, cooperation and communication, in addition to the final grades of secondary school. Good doctors distinguish themselves in these areas; not only on good professional knowledge.

Advice NOA: Design a clear profile of the students you want to select. Knowledge about which characteristics/characteristics distinguish a good student/professional from a less successful student/professional is important in this respect. This is reflected in the choice of the components of the selection and the weighting used to make a selection.

2. Reliable test instruments

The starting point after that must be the use of reliable and valid instruments and assessment protocols. This leads to an objective and standardized assessment of various characteristics of a student. Every student is therefore assessed in the same way on predetermined criteria. This is one of the biggest advantages of using a selection. This also provides opportunities for students whose high school grades are an underestimation of their abilities. In addition to cognitive aspects, softer aspects such as personality and motivation can also be taken into account.

Advice NOA: Use reliable, valid and scientifically based instruments and methods. For this reason, using motivation letters and unstructured interviews as a selection tool are not recommended. The information that students provide in this way can differ due to factors that have nothing to do with, for example, motivation for the study, and there is a high risk of subjectivity in the assessment[2].

3. Prevent student preparation

It is also advisable to use selection instruments for which little preparation of the student is possible. As an example, it is often mentioned that mainly one type of student, who prepares very well for the selection, is selected. This also prevents a better network or more financial resources from increasing the chances of selection. For example, through extra tutoring, training for selection tests, extra free time to work on a good CV or help with writing motivation letters.

Advice NOA: Include a combination of more general components (e.g. abilities, personality) and more study/subject-specific components (subject test, assignment, situational judgement test) in the selection. For example, use 'Worksamples' (a simulation of part of the future work) so that the student also gets an idea of the study and the future work. The reliable, valid and scientifically substantiated instruments and methods mentioned under point 2 have the advantage that social desirability and test preparation can be partially overcome or prevented. These test instruments can be supplemented with tests that measure more study/subject-specific components and skills. Take a critical look at knowledge tests if they are used. Does good preparation affect the results, and is this desirable? Is every participant able to prepare equally well without extra help?

4. Attention to diversity

Selection can lead to the student population for a programme becoming less diverse than it would be without selection. That is something that is not desirable and does not fit in with the mission of education. The biggest criticism of selection is that it can be detrimental to certain groups. In the event of an unweighted draw, everyone has an equal chance of being admitted and this problem would not arise.

It is therefore important to take this into account when selecting. This starts even before the selection process with the recruitment for the training and the information provided by the training. There is a chance of self-selection because candidates think they don't have a chance anyway and therefore do not apply for the study. In the selection itself, diversity can be taken into account by the choice of components/instruments that are included in the selection. Choose those components/instruments that have been investigated to see whether they make an undesirable distinction between different groups (bias). In tests and questionnaires from NOA, this is already taken into account during development and instruments are adapted if necessary. In addition, the weighting of the results on the components of the selection is important. How are the results of the various components weighted in order to arrive at a ranking?

Advice NOA: Pay attention to diversity before and during the selection. Take a critical look at the registrations and the instruments that are used. Are these free of bias? Diversity can also be taken into account in the weighting, for example by means of quotas where places are reserved for the best students of a specific group.

If drawing lots is allowed again, a combination of selection and drawing lots could also be chosen. In that case, the selection could be used to offer a place to the best candidates and to immediately reject the least suitable candidates. A lottery could then be used to determine which candidates will be offered the limited number of remaining places.

After the selection

After the selection, the results of the candidates offer a wealth of information that is often left unused.

Firstly, the results of the selection offer starting points for (study career) guidance. At the start of the study, the student and the study programme both have an idea of what is going well and what may need extra attention. The results on the various parts of the selection can be an excellent starting point for mentoring conversations at the beginning of the study and during progress interviews.

Secondly, the results are a starting point for an evaluation of the selection itself. Are the students selected to be selected? Questions about applications versus those selected in terms of capabilities, characteristics and diversity, for example, can be answered using the data from the selection. For example, the costs of the selection at Maastricht University that we mentioned above are recouped within two years because students drop out less often.

Later in the first and subsequent academic year, the candidates' results on the selection instruments can still be of value in relation to the study results. For example, how do the students do on the aspects on which the selection is composed? Is the profile that was drawn up at the beginning of the selection (see point 1) still correct?

With this article, we have tried to give an idea of the aspects involved in setting up a responsible selection procedure. If you have any questions and/or would like to discuss the possibilities further, please feel free to contact Corine Sonke, 020-5040800 or send an email to info@noa-vu.nl.

[1] https://www.observantonline.nl/Home/Artikelen/articleType/ArticleView/articleId/17989/Manier-van-selecteren-in-Maastricht-werkt-wel

- https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-018-1316-x

[2] An MMI, a standardized interview and assessment protocol is an example of how standardization is possible if interviews are chosen.