There are two issues here: the brief and unrepresentative nature of the 
writing sample and the remoteness of the multiple-choice questions from 
what we're really interested in.

Those are both valid complaints, but there's one significant problem: 
once you accept the need for some sort of standardized assessment 
independent of the evaluation by the students' teachers, you are 
essentially forced into both choices by considerations of fairness and 
of the reality of assessments.

Any writing done for a high-stakes evaluation must be done under 
controlled circumstances to prevent cheating. That means no carefully 
revised portfolios. So we're stuck with a timed essay. For the same 
reason, you can't announce a topic ahead of time, as some students will 
then memorize prepared essay which they will then reproduce. That also 
means the topic has to be broad, because we can't presuppose particular 
content knowledge for a general writing test.

There's also been significant research done which shows that while 
shorter times allows result in poorer essays, the time constraint 
affects students more or less equally. That is, when you extend or 
reduce the time, the rank order of students stays pretty much consistent.

The upshot is that if you're going to evaluate an essay, a short, timed 
piece on an unannounced topic is the only fair way to do it.

As for the multiple-choice portion, the problem with the essay is that 
it's the part of the test that shows the most variability. To really 
know how good a writer the student is, we ought to have them write a 
number of essays over a longer period of time. That, obviously, is 
impractical outside the classroom, and the whole point of a standardized 
test is to get information that is independent of classroom assessment. 
Multiple-choice questions are significantly more reliable, but the 
question is are they measuring anything that we care about.

The real question is not do multiple-choice questions such as those that 
appear on the SAT directly measure a student's writing ability. Clearly 
they don't, and I doubt even the test makers would try to defend that. 
The relevant question to ask is are they a valid proxy for good writing? 
In other words, is there a correlation between the ability to get a high 
score on those questions and actual good writing. The validity studies 
that ETS has done claims that there is, or at least that they correlate 
fairly well with grades in freshman composition classes.

Could you design a better test than the one that appears on the SAT? 
Almost certainly. I'd be willing to bet, however, that any test that 
could show better evidence of validity than what the SAT does would wind 
up making the same basic decisions about format.

On 1/15/2011 1:43 PM, Craig Hancock wrote:
>      If I missed John's point, I hope he will point that out.
>      Certainly, the best way to test writing is to look at actual writing.
> There has been lengthy discussion about whether a timed writing sample
> on a topic a student has never seen before is a good test of that.
> (The NCTE task force was highly critical.) Those of us who teach
> grammar as a subject have various ways to test whether students are
> learning the material. I am not a big fan of multiple choice tests.
> Knowledge about language can be tested directly. If you believe it has
> no value, then it makes no sense to test it.
>
> Craig>
>
>
>   Craig, you miss John's point; students who do well on a SAT grammar
>> question aren't necessarily better writers, but your question is as
>> guilty--if not more so?  His point reveals your hypocrisy in criticizing
>> SAT questions.  Be honest, it is hard--perhaps impossible--to write a
>> multiple choice grammar question that reveals writing eloquence.  At
>> least, most SAT questions can expose errors in skill.
>>
>>
>> On Jan 15, 2011, at 2:15 PM, Craig Hancock wrote:
>>
>>> John,
>>>     You seem to be making very thoughtful decisions about how to help
>>> your
>>> students. If there's an important hoop, you have to take it seriously,
>>> but you don't need to be subservient to it.
>>>    When I look at SAT sample questions, I find about one in four
>>> perplexing. I think they want students to "intuit" a better sentence,
>>> but they also want those intuitions to match the intuitions of the test
>>> makers, which, I have to admit, are not always the same as mine. I see
>>> the same dynamic at work with writing teachers who rewrite students'
>>> sentences in ways that sometimes seem arbitrary or idiosyncratic. In
>>> effect, they want the students to share their intuitions, to write like
>>> them, without reflection on how difficult (impossible?) that might be.
>>> I think this is one symptom of an overall loss of knowledge about
>>> language, and you're right: the students affected the most will be
>>> those whose background is different from the testmakers. Testing
>>> explicit knowledge would level the playing field. All students would be
>>> on equal footing.
>>>    As much as possible, I want students to own their own writing. They
>>> should be free to break conventions if they can do so knowing what
>>> those conventions are. They should make choices that they feel best
>>> convey their own evolving intentions. Knowledge of conventions and
>>> knowledge of rhetorical options are more important than behavior.
>>>
>>> Craig
>>>
>>>>
>>>
>>> Craig,
>>>>
>>>> I see another side of this issue every day – students who do very well
>>>> on
>>>> SAT questions aren't necessarily better writers for all their awareness
>>>> (
>>>> I hesitate to use knowledge) of SAT error patterns. And then the
>>>> insinuation is made, yet again, that grammar instruction doesn't
>>>> improve
>>>> student writing. Your point about conceptualization is well taken; so,
>>>> while the SAT test doesn't necessarily approach that aspect, it's more
>>>> upon me as a teacher of that test-taking population to approach
>>>> essential
>>>> skill sets with a larger picture in mind – the rhetorical function of
>>>> grammar in a particular phrase, paragraph, etc. We should always hope
>>>> for
>>>> transfer of knowledge within and across disciplines, and as we all know
>>>> the SAT isn't constructed to demonstrate that type of thinking, I find
>>>> it
>>>> a matter of classroom practice.
>>>>
>>>> The SAT and related test-prep methodology and practices manage to keep
>>>> students tracked and stratified, which to me is of even greater concern
>>>> (and a matter for a different thread). Those who do well have paid to
>>>> learn how to do well, or at least better than the average test taker.
>>>> It's
>>>> that student who also can afford to pay to actually attend the school
>>>> he
>>>> or she was "smart enough" to get into. Let's be courageous enough to
>>>> admit
>>>> that "school" is a glorified class system of haves and have nots and
>>>> that
>>>> Education has done a fine job in keeping those distinctions in proper
>>>> working order.
>>>>
>>>> But back to the change in curriculum addressed by the article, we can
>>>> hope
>>>> it moves beyond correctness and into the dynamics of language. I was
>>>> pleasantly surprised to watch my young nephew, in the 4th grade, learn
>>>> about predicates and adverb phrases explicitly (while I have 11th grade
>>>> students who arrive unable explain nouns, verbs, prepositions,
>>>> fragments,
>>>> etc.) ... but again, it's a step on a greater staircase. If we want
>>>> others
>>>> (let's start with NCTE, yes?) to believe that this particular, specific
>>>> type of knowledge is valuable and can in fact improve student writing -
>>>> which I fully believe it can - then the rest is up to good teaching.
>>>> Purposeful, explicit, critical, rigorous, and ongoing instruction at
>>>> that.
>>>>
>>>> Thanks...
>>>>
>>>> John
>>>>
>>>>
>>>>> Karl,
>>>>> It's interesting that they still equate grammar with
>>>>> "conventions" and
>>>>> with error, though they open up with more sophisticated terminology.
>>>>> The SAT test doesn't measure explicit knowledge about language; it
>>>>> simply asks you to find (or intuit) the best choice among options.
>>>>> There isn't, for example, a need to identify a structure as a
>>>>> prepositional phrase or modal auxiliary. No need to handle the "in
>>>>> early morning dawn" type of question we have been discussing other
>>>>> than to choose it as an alternative. It's interesting that they also
>>>>> separate proofreading and grammar from "more conceptual" skills,
>>>>> clearly not even aware that other views of grammar are possible. (One
>>>>> core concept of cognitive grammar--grammar is conceptualization.)They
>>>>> still don't seem to be making the judgment that knowledge about
>>>>> language is valuable in itself.
>>>>> This is pretty much true of the National Governor's Standards
>>>>> as well.
>>>>> They are a bit better, but still old school in their construal of
>>>>> grammar. Whether you are for it or against it, it still seems to be
>>>>> focused on correctness.
>>>>>>
>>>>> Craig
>>>>>
>>>>> I'm surprised that no one has brought this up. It appears Texas
>>>>> schools>  are going to get a lot more explicit grammatical instruction.
>>>>>>
>>>>>>
>>>>> http://www.dallasnews.com/sharedcontent/dws/news/localnews/stories/DN-grammar_15met.ART.State.Edition1.14a5f2e.html
>>>>>>
>>>>>> When Texas was arguing about new curriculum standards, I heard
>>>>> a lot
>>>>>> about the fight over the science standards, but nothing at all about
>>>>>> English standards.
>>>>>>
>>>>>> Are there any Texas educators on the list who would care to comment
>>>>>> about what difference these changes are making in the trenches?
>>>>>>
>>>>>> To join or leave this LISTSERV list, please visit the list's
>>>>> web interface
>>>>>> at:
>>>>>> http://listserv.muohio.edu/archives/ateg.html
>>>>>> and select "Join or leave the list"
>>>>>>
>>>>>> Visit ATEG's web site at http://ateg.org/
>>>>>>
>>>>>
>>>>> To join or leave this LISTSERV list, please visit the list's web
>>>>> interface at:
>>>>> http://listserv.muohio.edu/archives/ateg.html
>>>>> and select "Join or leave the list"
>>>>>
>>>>> Visit ATEG's web site at http://ateg.org/
>>>>>
>>>>
>>>> John Chorazy
>>>> English III Academy, Honors, and Academic
>>>> Pequannock Township High School
>>>>
>>>> Nulla dies sine linea.
>>>>
>>>> To join or leave this LISTSERV list, please visit the list's web
>>>> interface
>>>> at:
>>>>      http://listserv.muohio.edu/archives/ateg.html
>>>> and select "Join or leave the list"
>>>>
>>>> Visit ATEG's web site at http://ateg.org/
>>>>
>>>
>>> To join or leave this LISTSERV list, please visit the list's web
>>> interface at:
>>>      http://listserv.muohio.edu/archives/ateg.html
>>> and select "Join or leave the list"
>>>
>>> Visit ATEG's web site at http://ateg.org/
>>
>> To join or leave this LISTSERV list, please visit the list's web interface
>> at:
>>       http://listserv.muohio.edu/archives/ateg.html
>> and select "Join or leave the list"
>>
>> Visit ATEG's web site at http://ateg.org/
>>
>
> To join or leave this LISTSERV list, please visit the list's web interface at:
>       http://listserv.muohio.edu/archives/ateg.html
> and select "Join or leave the list"
>
> Visit ATEG's web site at http://ateg.org/
>

To join or leave this LISTSERV list, please visit the list's web interface at:
     http://listserv.muohio.edu/archives/ateg.html
and select "Join or leave the list"

Visit ATEG's web site at http://ateg.org/