Part 6: Application

Chapter 18: Competencies, the great pretender
Chapter 19: National tests and university grades
 

Synopsis

I first indicate how the notion of competency standards is overtly central to the whole competency movement, the introduction of which is shown to be overtly politically motivated. Thus the crucial links between political power and educational standards that are argued for in Chapters 3 and 4 become transparent.

I then go on to examine the validity, or more accurately, the invalidity of competency standards in the light of the thirteen sources of error specified in the previous chapter. The applicability of this analysis to a particular case is thus demonstrated.
 

Context: The re-birth of competencies in Australia

This same movement saw management practices move from control and exploitation of workers, modified at times by paternalistic concern for them, to a set of more manipulative practices described under the rubric of human resource management. This required the objectification of workers as a set of competencies, a necessary precursor to their ultimate replacement by more efficient computerised and robotic systems.

So the imposition of competencies as the basis of Australian technical and professional training during the late 1980s was in no way a decision informed by considered professional opinion; it was, from the start, an overtly political manoeuvre designed to solidify economic ideology in work practices, to demonstrate how skill and efficiency would reap their rewards in the "fair and just" game of the new internationally-competitive capitalist world order:

And of the essential arbitrariness of those standards or the necessary error in their measurement there is no word in this seventy one page report. Those two pillars of educational measurement, reliability and validity, do get a mention in the last page of the report. We are informed that assessment under the National Framework for the Recognition of Training (appropriately capitalised as a recognition of omnipotence) "provides for consistency as well as quality," and that one of the five principles of this approach is that "Assessment practices used shall be valid, ie. the techniques used must actually assess what they claim to assess." Furthermore, they must be reliable, in that "assessment approaches shall be able to be relied upon" (p71). And the Lord said, "Let it be done." And behold, it was done.

As Mc Donald (1994) nicely puts it: "The piece of commonsense that says that merely categorising something does not necessarily mean you can measure it easily, seems to have been lost" (p2).

In the first six pages of the NTB report, dealing with overview and context, the word "flexibility" appears eight times and "consistent" or "stable" six times. It is within this fundamental tension that the whole framework contradicts itself into nonsense. Let's unpack the argument in some detail:

But there is to be flexibility. "Flexibility is required to enable specific industry and enterprise characteristics and necessary performance outcomes to be accommodated" (p8). The report goes on to indicate what is meant by flexibility; how flexibility is itself to be stabilised:

So what have we got here? National Competency standards "provide the specification of the knowledge and skill and the application of that knowledge and skill to the standard of performance required in employment" (p9). And "assessment is the process of judging competency of an individual against prescribed standards of performance" (p11). So assessment is clearly in the Specific frame of reference, related to specific workplaces, related to particular jobs, related to performances that can be specifically described, and their levels clearly delineated and categorised (Norris, 1991).

On the other hand, the competencies are to be general, are to be non-specific, and furthermore have the truly remarkable quality of reflecting "not only industry's current but future needs"(p8), indicating perhaps a growth industry in astrology and clairvoyance training.

That the contradictions are so explicit gives hint to their genesis; they are the product of a succession of committees: Special Ministerial Conference on Training (1989), the Finn Report (1991), the Carmichael Report (1992), the Mayer Committee's Report (1992), and finally the input of the committees of the National Training Board.

At what level of discourse is all this? Are we involved in discourse at the rational-empirical level? Is this rhetoric really about measurement of competence? Or is this discourse at the mythical level? Is this about the construction of a national icon called competency standards around which a whole structure of power relations may be developed, and a whole new generation of workers constructed?

Porter (1992), speaking more specifically of the Carmichael report, sees it as

Sources of error in competency assessment

    Temporal errors

Firstly, there are the temporal errors in the criteria themselves. As Melton (1994) comments, "Inevitably the standards set reflect the perceptions of a particular group responding to perceived needs at a particular point in time. These will change as perceptions and needs change with time"( p288). Perhaps errors of this type are best categorised under construction errors, which become solidified in time because of the enormous structural and bureaucratic complexities involved in the production of the criteria of competency, which, despite what Melton says, do not usually specify a standard, a measurable level of adequacy.

Of more immediate concern are the variations in a particular person's performance over time. How are these to be interpreted? If competency is to be attached to the assessed person, then one adequate performance means the person has it, so is competent, so long as we assume that the thorny problem of adequacy has been solved. But if no adequate performance occurs, this could be a function of context, and does not necessarily imply incompetence. On the other hand, if competency is attached to events, then every performance ought to be adequate if the person is competent. Regardless of context? Confusion abounds!

In practice, temporal errors are confounded because no two events in which a human can engage can be identical, because time is change. Just as no two evaluations of a human event can be identical; they may involve identical categorisations, but the interpretative meaning of those categorisations change with time and with persons.

Potential confusions around the temporality of the measurement of competency standards abound; in current practice they are solved by pretending that they do not exist. They are a major source of error and confusion related to the meaning of any such measure.

    Contextual errors

If performance depends on context, then assessment is about events in context, and competence is someone's judgment that a particular contextual behaviour is adequate. This is surely what "work" is all about, whether it is in school or on a personal project or in a paid job.

But this is so messy, because then a label can't be pinned on a person, because it belongs equally to a context. So how do we get back to a context-free categorisation? Easy:

Unfortunately, the empirical data contradicts the assumption, and makes the idea of such "attributes" very suspect, or at the least quite unmeasurable. Stanley (1993) sums up the current position:

    Construction errors

The original idea of competencies, in the Specific frame of reference, was to detail and teach all the little tasks that seemed to constitute the performance, and then test that they were all learnt to the required level of adequacy. The notion of competency standards as currently interpreted has moved a long way from this reductionist view. As Bowden (1993) explains it:

Melton (1991) elucidates the dilemma cogently:

    Labelling errors

There are, as in all assessment systems, two types of labelling errors: There is the label of the particular competency; and there is the label of the categorisation of that competency.

As we move away from the Specific frame that can describe very specific eventful behaviours, we experience greater confusion in the meaning of the name that will become, in discourse, the referent for some practical competency that is ultimately defined either by some practical events in the world of work, or as some attribute or trait of a particular person. Regardless, whether we are talking of very generalised competencies such as "understands basic scientific principles," or very specific ones such as "adds two 2-digit numbers," what competency might mean in these domains is inevitably contested, and is different when viewed from different value positions or contexts, so the name will mean different things to different people. And this is not solved by the curriculum or test writer redefining such terms for their own purposes. As explained elsewhere, such a tactic may increase the reliability of the test, but it also increases its invalidity, because the user of the data generated from such tests is necessarily constrained to interpret the data in terms of the labels provided; labels to which they will attach their own meanings for their own purposes, and not magically absorb those constructed by remote curriculum and test constructors.

Similarly for the meaning of the label "adequacy" which is a necessary component of any discourse about competencies. Even if the problem of the meaning of the label that describes "what is being measured" could be solved, and we had a "scale" that was valid, we are still left with the problem of what is adequate along that scale; with the problem of the standard. This is also permeated with arbitrary and idiosyncratic definitions and interpretations, as well as enormous contextual variations; in short, another immense area of uncertainty, confusion, and hence error in personal categorisation and its interpretation.

    Attachment errors

When competencies are described in terms of some particular assessor's evaluation of "adequate" work performance in a specific workplace, attachment errors are at a minimum - so long as competence is clearly tied to that particular work at that particular place by that particular person. Any reduction of the specification description, any attempt to attach the label to the person assessed, represents an attachment error, and, at least in the philosophical frame of this study, makes any competency claim ontologically invalid.

When such competence is reduced to a number of specific performances under specified conditions at specified levels of adequacy, attachment errors are at a minimum when all of this information is retained in the assessment description. Attempts to combine this information into one statement about competency, of which the specific behaviours are elements, is a logical type error which makes any competency claim logically invalid. Attempts to give a meaning to such a summation of elements involves both a comparability error, and an epistemological error in that the summation can have no meaning. Any such summation, by losing the contextual data related to the individual elements, results in an attachment error because the data now becomes attached to the person being assessed.

When, on the other hand, competence pretends to be some fixed attribute or skill or trait of the person examined, an attribute that is somehow "measured" by the person's interaction with a test, then the attachment error occurs when this measure is attached to other contexts, to other workplaces. It will then become apparent as contextual error or prediction error.

    Frames of reference errors

Already the instability of the concepts of "competency," "competencies," and "competent" have been demonstrated. Norris (1991) comments that

Messick (1984) confuses the issue further when he claims that competence is what a person knows and can do under ideal circumstances, whereas performance is what is actually done under existing circumstances. So competence is potential, is ability imminent. It follows that one successful performance demonstrates competence, because the conditions cannot be more than ideal, so one must assume they were less for any successful performance. On the other hand an unsuccessful performance can never demonstrate incompetence, because the conditions may not have been ideal.

So in theory there is confusion as to whether we are dealing with traits or demonstrated skills, concepts that require the General frame of reference, or particular defined behaviours, which require the Specific frame. In practice the confusion proliferates, for invariably the description of the standards that define the cut-off is either non-existent or vague, as indeed is the measuring instrument or instruments which will provide data to which the standard must be compared. So the practical assessment of what is adequate must be made in the Responsive frame - an intuitive response from the assessor. Such "subjective" admissions are, of course, utterly inadmissible, for the success of the whole charade is dependent on the appearance of objective accuracy and precision. Luckily, this is possible if the assessment mode shifts to the Judge's frame. So this is what happens, and certainty is reestablished, albeit in a different frame than that theoretically intended.

To summarise, analysis of competency assessment in terms of frames of reference indicates semantic chaos, discourse riddled with self contradictions. Out of it all there still emerges, from all involved, belief that the system works. And in as much as people are categorised, it does indeed work. To further believe however that some accurate measure of minute error has emerged from such conceptual confusion and personal lack of awareness is to substitute blind faith for rational thought.

Invalidity from this source is thus profound, and stems from the epistemological irrationality that must occur when frames of reference with contradictory assumptions are amalgamated without distinction into a single discourse.

Where does all this leave the individual student? Apparently presented with a list of clearly defined outcomes, things to know and do at predetermined levels of competence, closer inspection leaves the student with a list of ambiguous topic headings and ill-defined "skills," on the basis of which he or she will be tested, and then categorised by comparison with opaque standards visible only to the professional eye of the teacher. Was it ever otherwise?

    Instrumental errors

Referring to standardised and/or criterion referenced tests, Berlak (1992) notes that "The credibility of these tests depends upon the claim that they are scientific instruments" (p181). Just so the credibility of competency assessment as a whole. The notion that these assessment systems are based on the measurement of clearly defined standards is what provides the educational, moral and public relations glue that transforms a set of fragile value and assumption struts into a powerful cognitive structure.

Yet it is surely a false claim. There are rarely such standards available, even at the practical level. At the level of physical factory products, standards that are related to some criteria of quality can sometimes be set up and measured, but these are a far cry from the "attributes" that predate competence in personal performance.

As described in the section on frame of reference errors, the whole discourse is emersed in epistemological confusion. What is important to note here, however, is that by pretending to belong to the Specific frame, the professional necessity to provide estimates of standard errors of measurement, necessary in the General frame, is side-stepped; not that educational practice ever paid much attention to that professional necessity.
 

The instrument, as apposed to any test, thus is firmly inside the mind of the assessor, an intuitive judgment hidden beneath the overt scientism of the competency label with its overtones of specific behaviours and definable standards.

    Categorisation errors

Competencies must be described and then categorised. To categorise a competency we must first measure it and then compare the measure to a standard. As a result of this comparison we may then categorise the performance as adequate or inadequate, or the person as having, or not having, the competence.

So can we measure accurately these competencies that are described? Norris (1991) comments :

Levin (1978) described the use of minimal outcomes in schools in the United States. It applies equally to the use of competencies in Australia in the 1990s:

Melton (1994) accurately describes the sort of processes that are actually involved in competency assessment:

    Prediction errors

Because competencies in Australia have been specifically politically invoked to improve work practices and hence profitability in industry, prediction errors occur when the produced competencies do not specifically do all of those things; it is possible, remotely so in my view, that the educational events wrapped around competency standards might indeed in some cases have some validity in regard to the first of these claims, related to work practices, though some early research does not support this (Gillis, 1995). Of course, even if there is some correlation between the competence measure and some later predicted outcome, this in itself does not indicate causal link between the two categorisations that is mediated through the competency attribute.

In fact, as I have argued in the section on Consequential errors, it is unlikely that any empirical data will be collected in this regard because it is the assumption on which the whole scheme is premised, and thus not amenable to investigation.

    Logical type errors

In all of its cyclic incarnations, competencies as specific behaviours have invariably encountered the criticism that they are reductionist, that they fragment knowledge, that they are in essence, trivial. Perhaps it is sufficient here to give two references:

If the former, then the logical type error occurs in the summation, in the confusion of members of a class (the specific behaviours) with the whole class (the competency). In the latter case the logical type error occurs in the confusion of a description of a class (the generic competency) with members of that class (specific context-related work performances).

Either way confusion is confounded and error escalated through the attempt to define and describe competencies in any place other than their area of actual performance.

    Value errors

Competence implies some purposeful act; a person is competent when she does something adequately in some context. So the first question to be asked in a competency judgment is: What ought the person do in order to be deemed adequate? This is not a factual question, but a value premise. And it is where every list of competencies must begin. Pearson (1984) argues that "until the value premise is made the competency claim cannot get off the ground" (p34). Thus all competency descriptions are based on value premises, which are usually unstated.

One implication of this is, as Norris (1991) points out, that "In the effort to describe competence in precise, transparent and observable terms, to predict the specific outcome of effective action, what is in fact happening is the pre-determining of good practice" (p334).

To the extent that competency requirements dictate school programmes, they also determine that "The measure of success that is applied for the schools is not the degree to which they foster intrinsically meaningful activities, but the degree to which they satisfy competence-related outcomes (Levin, 1978, p311). Levin (1978) goes on to assert that "Certification standards are signals to the schools of what is considered important by society, and their message will not be lost in individual teacher decisions or organizational ones" (p314).

Jackson (1993) perceives that the underlying intent of competency based teaching and assessment is to provide more governmental control on teaching institutions, and any effect on individual learning is secondary to this:

Yet as we have seen, this whole operation begins from a particular view, generally not stated explicitly, of best practice: a particular positioning; a particular attachment to certain sorts of power and affect relations; a particular consciousness about work and its effects; and a begging of the question of who benefits from these particulars.

Where does the individual student position himself in this value matrix? He is supposedly acquiring the competencies that will allow flexibility in various job performances. Yet his experience may deny the usefulness and relevance of what is being presented. Even so, the competencies must be achieved. So rather than flexibility, such a student will learn not flexibility, but conformity; not a producer of new work practices, but a consumer of old ones.

Invalidity in terms of value then derives not only from the bias that derives from unstated value assumptions, but from the very specificity of stated intentions, and their contradiction by associated social effects; that is, by those very contradictions that are at the heart of symbolic violence.

    Consequential errors

Elsewhere in this dissertation I have argued the centrality of assessment procedures to the construction of the individual in society. Commenting on the scene in the United States, Berlak (1992) comments:

Students in this context are cogs in a gigantic machine. They are disempowered in terms of the substance and the value assumptions that predate what is to be learnt. There is no notion here of learning that grows out of specific purposes, learning styles or values of students, or of curricula negotiated to meet such purposes. Nor indeed is there any sense of relatively autonomous teaching agencies offering, among them, a proliferation of solutions to the relatively intractable problems of job training. As presented, competency assessment is the solution. The problems, whatever they may be, have been pre-empted. The job of training is to implement the solution. The function of evaluation is to indicate that the solution has been implemented. The closed black and white fantasy circle is complete.

Invalidity in terms of consequences stems most profoundly from the loss of the initial problem, which has been firmly removed outside the closed circle of competency discourse. For the National Training Board (1992), "the overriding aim is to increase the competitiveness and productivity of Australian industry"(p4), an aim now subsumed under the solution, which is assumed to be the National competency assessment system; so within the discourse of competency assessment not only can this question about productivity not be answered, it cannot even be asked, because its answer is itself.

For the individual student the potential errors in categorisation are immense. This particularly applies to students already enmeshed in work practises. Their learning cannot be based on local analyses of work environment deficiencies, or on creative transformations of work practices, because it is dedicated to their absorption of pre-ordained competencies which are supposed to magically provide such solutions. And if (when) the magic doesn't work, there is no place to go, for success has too long been dependent on the acceptance of absurdity.
 

Summary