Tools And Techniques Of Measurement And Evaluation

VALIDITY:

It is the degree to which a test measures what is suppose to measure. (L. R. Gay)

• Test validity refers to the degree to which the test actually measures what it claims to measure.
• Test validity is also the extent to which inferences, conclusions, and decisions made on the basis of test scores are appropriate and meaningful.
• Validity is the strength of our conclusions, inferences or propositions.
• Validity refers to the accuracy of an assessment -- whether or not it measures what it is supposed to measure.
• If a test is valid, it is almost always reliable.

Measurement of validity:

There are three ways in which validity can be measured.

Type of Validity	Definition	Example/Non-Example
Content	The extent to which the content of the test matches the instructional objectives.	A semester or quarter exam that only includes content covered during the last six weeks is not a valid measure of the course's overall objectives -- it has very low content validity.
Criterion	The extent to which scores on the test are in agreement with (concurrent validity) or predict (predictive validity) an external criterion.	If the end-of-year math tests in 4th grade correlate highly with the statewide math tests, they would have high concurrent validity.
Construct	The extent to which an assessment corresponds to other variables, as predicted by some rationale or theory.	If you can correctly hypothesize that ESOL students will perform differently on a reading test than English-speaking students (because of theory), the assessment may have construct validity.

In order to have confidence that a test is valid (and therefore the inferences we make based on the test scores are valid), all three kinds of validity evidence should be considered. So, does all this talk about validity and reliability mean you need to conduct statistical analyses on your classroom quizzes? No, it doesn’t. (Although you may, on occasion, want to ask one of your peers to verify the content validity of your major assessments.) However, you should be aware of the basic tenets of validity and reliability as you construct your classroom assessments, and you should be able to help parents interpret scores for the standardized exams.

Types of Validity:

There are four types of validity commonly examined in social research.

1. Conclusion validity asks is there a relationship between the program and the observed outcome? Or, in our example, is there a connection between the attendance policy and the increased participation we saw?
2. Internal Validity asks if there is a relationship between the program and the outcome we saw, is it a causal relationship? For example, did the attendance policy cause class participation to increase?
3. Construct validity is the hardest to understand in my opinion. It asks if there is there a relationship between how I operationalized my concepts in this study to the actual causal relationship I'm trying to study/? Or in our example, did our treatment (attendance policy) reflect the construct of attendance, and did our measured outcome - increased class participation - reflect the construct of participation? Overall, we are trying to generalize our conceptualized treatment and outcomes to broader constructs of the same concepts.
4. External validity refers to our ability to generalize the results of our study to other settings. In our example, could we generalize our results to other classrooms?

Characteristics of Validity:

There are following characteristics of validity commonly examined in social research.

1. Content Validity: How well the sample of test items represents the content the test is designed to measure.
2. Predictive validity: How well predictions made by a test are confirmed by later behavior of subjects.
3. Concurrent validity: Similar to predictive validity, but behavior is measured at same time as test.
4. Construct validity: How well a particular test can be shown to measure a particular construct (a theoretical construction about the nature of human behavior, such as intelligence, anxiety, or creativity).
5. Face validity: How closely the test appears to measure what it's supposed to measure.

RELIABILITY:

Reliability is the degree to which a test is consistently measures whatever it measure.

(L. R. Gay)

Test reliability refers to the degree to which a test is consistent and stable in measuring what it is intended to measure. Most simply put, a test is reliable if it is consistent within itself and across time. Reliability is the consistency of your measurement, or the degree to which an instrument measures the same way each time it is used under the same condition with the same subjects. In short, it is the repeatability of your measurement. It is the level of internal consistency or stability of the test over time, or the ability of the test to obtain the same score from the same student at different administrations (given the same conditions). Reliability is usually expressed as some sort of correlation coefficient. Values may range from .00 (low reliability) to 1.00 (perfect reliability). Reliability refers to the extent to which assessments are consistent. Just as we enjoy having reliable cars (cars that start every time we need them), we strive to have reliable, consistent instruments to measure student achievement. Another way to think of reliability is to imagine a kitchen scale. If you weigh five pounds of potatoes in the morning, and the scale is reliable, the same scale should register five pounds for the potatoes an hour later (unless, of course, you peeled and cooked them). Likewise, instruments such as classroom tests and national standardized exams should be reliable – it should not make any difference whether a student takes the assessment in the morning or afternoon; one day or the next. Another measure of reliability is the internal consistency of the items. For example, if you create a quiz to measure students’ ability to solve quadratic equations, you should be able to assume that if a student gets an item correct, he or she will also get other, similar items correct.  The following table outlines three common reliability measures.

Type of Reliability	How to Measure
Stability or Test-Retest	Give the same assessment twice, separated by days, weeks, or months. Reliability is stated as the correlation between scores at Time 1 and Time 2.
Alternate Form	Create two forms of the same test (vary the items slightly).  Reliability is stated as correlation between scores of Test 1 and Test 2.
Internal  Consistency	Compare one half of the test to the other half.  Or, use methods such as Kuder-Richardson Formula 20 (KR20) or Cronbach's Alpha.

Estimation of Reliability:

There are two ways by which reliability is usually estimated

1. Test/Retest: Test/retest is the more conservative method to estimate reliability. The idea is that you should get the same score on test 1 as you do on test 2.

The three main components to this method are as follows:

o    Implement your measurement instrument at two separate times for each subject

o    Compute the correlation between the two separate measurements;

o    Assume there is no change in the underlying condition (or trait you are trying to measure) between test 1 and test 2.

2. Internal Consistency: Internal consistency estimates reliability by grouping questions in a questionnaire that measure the same concept. For example, you could write two sets of three questions that measure the same concept (say class participation) and after collecting the responses, run a correlation between those two groups of three questions to determine if your instrument is reliably measuring that concept.

The primary difference between test/retest and internal consistency estimates of reliability is that test/retest involves two administrations of the measurement instrument, whereas the internal consistency method involves only one administration of that instrument.

The Relationship of Reliability and Validity

In order for assessments to be sound, they must be free of bias and distortion. Reliability and validity are two concepts that are important for defining and measuring bias and distortion. Test validity is requisite to test reliability. If a test is not valid, then reliability is moot. In other words, if a test is not valid there is no point in discussing reliability because test validity is required before reliability can be considered in any meaningful way. Likewise, if a test is not reliable it is also not valid.

OBJECTIVITY:

Objectivity is the extent to which the instrument is free from personal error (personal bias) that is subjectivity on the part of the error. (C.V Good)

The objectivity of test refers to the degree to which equally competent scores obtained the same results. (Norman E. Gronlund)

ACCURACY:

A term used to describe the size of the relative error.                                      (C.V Good)

ADECUACY:

A characteristic evidenced by its sufficient length to sample widely the behaviour it is designed to measure.                                                                                         (C.V Good)

Methods of Data Recording

The assessment techniques in this category may be used with any of the ongoing student activities as well as with the quizzes and tests. The appropriateness of the technique for the purpose intended should act as a guide.

• Anecdotal Records

Anecdotal records refer to written descriptions of student progress that a teacher keeps on a day- to-day basis.

A teacher may decide to keep anecdotal records on students' ability to manipulate materials at assessment stations, to work in a group, to work in a test-taking situation, or to complete a project or a written report. There are situations where a teacher will keep anecdotal comments on the development of specific skills related to instructional objectives, on the behavior of a student, or on the attitude expressed or demonstrated by a student. Anecdotal records are as flexible as a teacher wishes to make them.

• Observation Checklists

Observation checklists are lists of criteria a teacher determines are important to observe in students at a particular time. Beside each of the criteria, a notation is made as to whether that particular criterion was observed.

Checklists can be used to record the presence or the absence of knowledge, particular skills, learning processes, or attitudes. They may be used to record such information in relation to written assignments, presentations, classroom performance, test-taking behaviors, individual or group work, fulfillment of the requirements of a contract, self- and peer-assessment of work, or completion of an assessment station. How a teacher wishes to use an observation checklist depends upon the type of student progress information required.

• Rating Scales

Rating scales have the same usage as observation checklists. The essential difference lies in what is indicated. Observation checklists record the presence or absence of a particular knowledge item, skill, or process. Rating scales record the degree to which they are found or the quality of the performance.

Anecdotal Records Description

An anecdotal record is a written description of the observations made on students. These records are usually collected in a specific book or folder.

Evaluation Context

• Formative

The very act of recording observations may serve to alert you to some aspect of a student's learning or attitude that may need immediate attention; for example, an outburst caused by frustration.

• Summative

Since the anecdotal record concentrates on describing incidents of student performance over a period of time, the sequence of anecdotes can serve as a record of the student's development towards long term goals such as lifelong learning, healthy self-concept, cooperative learning, skill development, work/study habits, knowledge attainment, and interest/attitude.

• Diagnostic

Through the regular spotlighting of a student's performance, areas needing special attention may emerge. Examples include communication skills and personal development. Your anecdotal records may start to show that Billy is consistently having trouble in expressing coherent thoughts. As a consequence, you may decide to investigate the causes of this behavior more thoroughly.

Using Technique to Best Advantage

Entries must be made with appropriate frequency. They should eventually encompass all the students, although some students may warrant more entries than others. Anecdotal records offer you a way of recording aspects of your students' learning that might not be identified by other techniques.

Guidelines for use

• What to write

First, you write a description of the incident in an objective way by describing what actually happened. Then make further notes on your analysis of the situation, any comments you want to make, and any questions you pose to yourself that may guide further observations.

• When to use

For many teachers, the time when students are engaged in writing offers an opportunity to demonstrate that teachers are writing, too. You can use a portion of your writing time for recording your anecdotes. Teachers who do not have these opportunities may use times when students are engaged in independent work. In program areas such as physical education and home economics, there are parts of the period when students change clothes or tidy up equipment. You might be able to use these times for recording entries. Whichever scheme is chosen, it should offer regular opportunities for entering observations.

• How to record

Various formats have been developed. A notebook with each entry dated offers a powerful chronological record; although it is sometimes difficult locate a particular student. Alphabetized notebooks, looking like large address books, are available and they permit easy reference by student name. Alternatively, a loose leaf format may be used so that the entries may be entered chronologically, and at the end of the year may be reformatted by student name. One further idea: modern technology has provided us with conveniences for recording and storing student progress data that range from electronic student data files available on various software programs to removable self-stick notes that can be used to record the anecdote and then be affixed to the student record.

Example:          No example is required for the open-ended, unstructured anecdotal record. The examples that follow are formats for anecdotal records designed to give you ideas as to how to set up this type of data recording method. Keep in mind these are only examples.

Using the Information for Student Evaluation

While the entries themselves are usually not shown to the student or the parents/guardians, they can form a valuable basis for communication. They allow you to flesh out your year-end reports on the more holistic dimensions of student growth.

Observation Checklists Description

The observation checklist is a listing of specific concepts, skills, processes, or attitudes, the presence or absence of which you wish to record. If the observation checklist is used relatively frequently and over time, a longitudinal profile of a student is assembled and ultimately evaluated.

Evaluation Context

The observation checklist is most appropriately used in situations where you wish to assess your students' abilities, attitudes, or performance in process areas. For example, it can assess communication skills, cooperative learning skills, extent of participation, interest in the topic, and psychomotor skills.

Using Technique to Best Advantage

Used on a single occasion, the observation checklist can provide formative evaluation information for the situation in which it is used For example, to learn how effective students are when working in groups, a checklist to observe them in a single group session can be used. This will provide information to guide future instruction.

Observation checklists are most useful when collected over time and used summatively or diagnostically. Once you decide to use observation checklists in your evaluation plan, you must use them systematically. They are misleading when used sporadically.

Guidelines for Use

Usually the observation checklist is used during class time. Therefore, it must be simple. The most efficient way to collect data is to record learning progress on four or five students at the same time. If you choose to observe four students per lesson and you have 28 students, you will cover the class once every seven lessons. At the end of the term or unit, you will have several observations on every student. If your class is working in groups, do one group every day. If not, use your seating plan to identify groups of students sitting in the same area. If you choose students alphabetically, you may find that your eyes have to cover too much of the room in order to encompass the selected students.

• Before the unit or course begins, develop an estimate of what would constitute appropriate learning outcomes for your students. If you intend to use the information for making criterion-referenced judgments, decide on what your criteria will be. You may wish to develop minimum criteria (e.g., "six of the eight behaviors must be observed over the course of the unit"), or you may wish to develop different criteria levels for what would constitute excellent, satisfactory, or unsatisfactory work. Decisions on criteria should be made before the observation sequence begins.
• Before every class, enter the names of the students, the date, and the activity. During class, pay special attention to the selected group so that you build an impression of their level of competence or execution of the skills, processes, or attitudes you wish to record.

Recording options: You may simply mark an entry on the item's first appearance and leave it at that, or you may record an item's every appearance (e.g., ). If you develop some measure of degree to describe the item (e.g., !, ?, or X), you have transformed your observation checklist into a rating scale. This is a characteristic of rating scales and checklists that gives you more flexibility. Make sure you record the date and the class on every observation checklist you use.

• After class, annotate the checklist sheet with any appropriate thoughts. For example, "Fire drill interrupted the group activity - recorded instances are therefore lower than I anticipated." Ä File the checklist sheet with the others so that the class set is available for evaluation at the end of the course or unit. Large envelopes are useful here.

Example: The example checklists are designed to give you ideas as to how to set up this type of data recording technique. Keep in mind these are only examples.

Using the Information for Student Evaluation

Arrange the sheets into piles according to the student groups. Read them all over once or twice to develop a feeling for the overall class picture. For criterion-referenced judgments, refer to the criterion levels you made initially. For norm-referenced judgments, estimate where each student lies relative to the others in the class and make your judgment. If you have looked for very general or broad items, be careful not to over interpret your data - for example, "On these aspects of the course Kim seems to be performing a little bit more consistently than most of the students." This may be about the level of sophistication that is possible, depending on how you constructed the instrument. For self-referenced judgments, all the checklists on one particular student can be studied, providing a measure of progress over the span of the unit or course. This is one of the most powerful uses of the checklist.

• Where you can, start with an existing checklist and modify it according to your needs.
• Choose items that relate to the intended learning outcomes of the unit. If you wish to use checklists in several courses and they have many overlapping items, develop a master list and eliminate those items that are inappropriate for the specific unit or course.
• Choose items that you can observe or reasonably infer. If an item is too vague (e.g., interest in the subject), you may not be consistent throughout the term in your estimation and recording of it.
• Keep the list of items manageable. Twelve is about the maximum.
• Keep the language of the items simple and jargon-free. In that way you can use the checklists at parent-teacher or student-teacher interviews.

Variants

Develop checklists that detail one particular series of components. For example, a checklist on the correct operation of a microscope may be useful in minimum competency situations where something just has to be done correctly.

As previously mentioned, the observation checklist shares many characteristics with the rating scale. This is an advantage that can be a time-saver for you.

Rating Scales Description

Rating scales are measuring instruments that allow representation of the extent to which specific concepts, skills, processes, or attitudes exist in students and their work.

Evaluation Context

Rating scales enable the teacher to record student performance on a wide range of skills and attitudes. They are particularly useful in situations where the student performance can be described along a continuum, such as participation in a debate or skill in preparing a microscope slide.

Guidelines for Use

As the rating scale is usually used during class time, it must be simple to use.

• Developing the rating scale

Once you decide upon the activity you wish to rate, break it up into its constituent parts. Make the parts as specific as possible so as to increase the scale's reliability. For example, instead of globally rating "performance in debates," decide on what performance criteria you wish to observe in the student. Perhaps "states argument," "demonstrates background preparation," "responds to opposition arguments relevantly" might together give a less inferential picture of the student's performance than the rating on the global behavior alone.

The next task is to develop the scale points. You might use the old stand-by: "very good/good/ average/poor/very poor," or you can develop more descriptive scale points. For the criterion mentioned above, "states argument," you could choose to use points based upon how forceful the student was: "very forceful/forceful/average/ diffident/very diffident."

• Before the unit or course begins

If you intend to use the information for making criterion-referenced judgments, decide on what your criteria will be. You may wish to develop minimum criteria such as, "six of the eight behaviors must be rated at the satisfactory level or higher over the course of the unit." Or you may wish to develop different criteria levels for what would constitute excellent, satisfactory, or unsatisfactory work.

• Before every class

Enter the names of the students, the date, and the activity. This will usually be governed by the activity being rated. If Peter and Petra are facing off in today's debate, then theirs are the names entered.

• Recording

As you form an impression of student behavior on each criterion, mark the point on the continuum.

• After class

Examine the individual criteria and decide on an overall rating for each student on the total behavior being rated. File the rating sheet with the others so that the class set is available as a record. Large envelopes are useful here.

Example: In the first example provided, the full sheet on 'Performance in Debates' is developed. The other examples that follow are designed to give you ideas as to how to set up this type of data recording method. Keep in mind these are only examples.

Two Variants

Rating scales have many variants and any book on measurement will offer examples. Two variants are described here.

• Self-evaluation

Rating scales are very useful in allowing students to perform self-evaluation on their own work. Present the student with a rating scale that covers the aspects of the unit or project which you wish him or her to self-evaluate. Examples may be the amount of effort expended in research, the amount of effort expended on initial organization, the extent to which the student reflected on the initial organization, the amount of reorganization, or the effort spent on writing. The student's ratings on the five-point scale can form a useful starting-point for teacher- student dialogue. 

• Number line

The number line is a variant that is particularly useful with pre-reading students. On a long piece of paper, draw a horizontal line and mark off five to ten intervals. On the extreme left- hand mark, draw a sad face, at the mid-point draw a neutral face, and at the right-hand mark, draw a happy face. Mount the number line on the wall at a suitable height. The student then places the left palm on the sad face and, in response to a question (such as "How much did you like that story?"), positions the right palm accordingly. If the story was not a success, then both hands overlap on the unhappy face. By training the students to pass by the number line fairly quickly, you can obtain rapid feedback on the question you pose. With experience, more sophisticated questions can be asked. Here are examples from a unit on estimation. "When you guessed the number of peas in the pea pod that I showed you, how sure were you of your answer?" "Now, when you guessed the number of Smarties in the bottle, how sure were you?"

Dialouge Editing

Dialogue editing is one of the least understood aspects of motion picture sound post production. Most people have some grasp of the roles of sound effects editors or backgrounds editors or music editors. To most moviegoers, Foley is a charming—yet perhaps a bit silly—process, and it’s not hard to understand it once you’ve seen it. Mixing, too, is a pretty straightforward concept, even if massively complex in actual fact. Few non-pros, however, understand dialogue editing. “Does that mean you take out the dirty words?” is a common response when I tell someone what I do for a living.
A dialogue editor is responsible for every sound that was recorded during the shoot. She takes the more or less finished film from the picture editor, makes sense of the edited sounds, organizes them, finds out what works and what doesn’t. The dialogue editor wades through the outtakes to find better articulations, quieter passages, sexier breaths, and less vulgar lip smacks. He replaces washy wide-shot sound with clean close-up takes, establishes depth in otherwise flat scenes, and edits tracks for maximum punch and clarity. Dialogue editors work to remove the film making from the film. Dolly squeaks, camera noise, crew rustling, and light buzzes must go; otherwise, the magic of the movies is compromised. These editors help present the actors in their best light, quieting dentures, eliminating belly noises, and sobering slurred syllables. And when the production sound can’t be saved, the dialogue editor is involved in the ADR process, that is, the rerecording of voices in the studio to replace problem field recordings or to beef up performances. Dialogue editing is all of these things and more. Dialogue is what makes most films work. The dialogue editor makes the dialogue work. There are many kinds of films. Some are driven largely by juxtaposition of images, most notably silent classics such as Potemkin and Alexander Nevsky. Others use a language of camera motion or composition to speak to the viewer, obvious examples being Citizen Kane and Besieged. Still others use color or shading to make a point. Among the many examples of color in storytelling are films as diverse as Juliet of the Spirits, Women on the Verge of a Nervous Breakdown, and The Corpse Bride. Many films rely heavily on music to express themselves, while others manipulate through sound effects. Film-makers concoct a personal stew of visual and aural tools to reveal or enhance a story. But the overwhelming majority of narrative films rely on the spoken word to tell the story, develop characters, and touch hearts. The modern motion picture has more in common with theater than with early montage cinema, in that only a minority of current (especially commercial) films trust image over dialogue. However cinematic or structurally sophisticated, most of the films you work on will tell their story through dialogue. You can eliminate all (or most) of the sound effects in a scene, the backgrounds can largely be muted, or the music can be dumped. The scene will still work. But for the most part you can’t eliminate the dialogue track, even for a moment. The dialogue holds key story information as well as the room tone, the “air” that defines the location of the scene. We’ve grown so accustomed to the language of film that blind acceptance of the implausible is the norm. After a century of tutoring in film language, we accept the convention of cutting from one speaking character to another and we interpret this unnatural collision as a “conversation.” We have no trouble accepting the abnormal phone split, in which we hear both sides of a telephone conversation. And we don’t shake our heads in disbelief when listening to a conversation in a distant long shot—sounds we rationally know would be out of bounds to us. We want to believe that what we see and hear is very real. Only something that patently breaks the rules of the language of film will shatter our confidence and make us question the reality we’ve been accepting. Most people know that films are rarely shot in order but rather in a sequence determined by location and set availability, actors’ schedules, exotic equipment rental coordination, and (hopefully) overall efficiency. Even within scenes, the shot sequences may not be in script order. Don’t be surprised when you find that two people who are apparently talking to each other on screen were not together when the close-ups were shot. This shooting discontinuity makes the dialogue editor’s job even more crucial. We must do everything in our power to convince the viewer that these characters actually were talking with each other; we must remove any impediments to the audience’s total belief in the reality of the scene. Many photographic mistakes dash our confidence in a film. Crossing the axis, exposed flying wires, ridiculous costumes or makeup, or absurd anachronisms can take us out of the magic. Many audio mistakes jar us out of the fantasy as well. Poorly matched ADR, grossly inappropriate backgrounds or sound effects (birds out of season or location, incorrect guns or motors, etc.), insensitive use of perspective, or a sloppy mix will drag us out of the film, even if for a moment, and it’s a battle to reestablish trust after such a betrayal. Dialogue editors play a vital role in the subterfuge of movie making. It’s up to us to turn the many shots and production sounds into a convincing, coherent scene that even the most skeptical film goer will watch without being aware of the film making that went into it. Unlike sound effects editors, background editors, or music editors, our magic most certainly goes unseen (or unheard)—if it’s performed well. Only our failures attract attention.


So, just what is the dialogue editor’s part in this process?
• To organize and manage the material. When you first screen a film,
the sound may seem to work, but just beneath the surface it’s
unorganized and unfocused. One of your biggest jobs is to make sense
of it; otherwise, you can’t help the dialogue discover itself.
• To smooth the transitions between shots so conversations appear to be
happening in the same space, at the same time.
• To fix articulation problems, overlaps, and other language issues. This
usually involves searching through alternate takes to find similar
deliveries that can be used for patches.
• To address unwanted, unseemly actor noises such as lip smacks,
denture clatter, and hungry belly rumbles. At times any one of these
may be appropriate, but each must be challenged, removed, replaced,
or at least thought about.
• To pay attention to changes and adds that will enhance the story or
motivate characters’ actions. This often involves a combination of
observation and talking with the director, editor, and writer.
• To remove unwanted external noises, whether crew or dolly noise,
unnecessary footsteps, or birds chirping during day-for-night scenes.
The list of on-set noises is endless, and each one of them reminds the
viewer that we’re making a movie.
• To replace sections of dialogue corrupted by distortion, wind, clothing
rustle, boom handling, and the like.
• To determine what can’t be saved through editing and must be
rerecorded through post synchronization. This is done in conjunction
with the ADR supervisor.

• To serve as the arbiter of sync issues in the film. Usually the other
departments follow your lead in deciding the film’s sync.
• To prepare the dialogue tracks for the dialogue premix (predub). Your
tracks not only must sound good but must be presented to the mix in
a logical and efficient manner so that the mixer can spend precious
time telling a story with the dialogue tracks rather than merely sorting
out disasters.
• To assist in the dialogue premix. You more than anyone know the
tracks, and you planned a particular interpretation of the scene while
cutting them, so you need to be involved at this stage. You also have
to be available to make fixes or changes to the dialogue tracks during
the premix.
• To separate production effects from the dialogue track in preparation
for the M&E mix.
Although it’s occasionally derided as boring or purely technical by the ill-
informed, dialogue editing is the glue that holds the production sound
together. No other facet of sound editing requires such a wide array of
skills.