J of A: How the CPA Exam is Scored

The Journal of Accountancy has published an article in their May 2011 issue titled “How the CPA Exam is Scored”.

Some items of note that CPA Exam candidates should be aware of:

The article states that BEC essays are currently graded by humans.

“Because a new section of essays was introduced into the Business Environment and Concepts (BEC) section this year, initially, those essays will still have to be scored by humans, Mattar said. The Examinations Team will need to build computer models after it receives enough sample responses. If a test taker’s total score is close to the passing score, the candidate’s written responses will be automatically regarded [sic] by human graders.”

Another71.com asks: Where is the AICPA getting this army of CPAs to hand-grade tens of thousands of essays? 8,478 BEC exams were taken in the January/February 2011 testing window. BEC has three essays and let's assume that one of them gets tossed out as a pre-test essay. This leaves nearly 17,000 essays to grade by CPAs during tax season? I'm sure there is a reasonable answer…that just seems like an incredible amount of essays to grade by hand.

If anyone out there in AICPA-land wants to edjukate me ~ shoot me an e-mail at jeff@another71.com and I'll post the explanation. Again – I'm sure I'm missing the obvious.

Update: The article says BEC essays are human graded, but the PDF on the AICPA Website says they are computer graded?

Most responses are scored by a computer grading program, which is calibrated using human scorers. In some cases, responses are scored by a network of human graders (all CPAs). If your score is close to the passing score, your written responses will be automatically re-graded by human graders. When there is more than one grader for a response, the average of the scores is used as the final grade.

The simulations you receive have nothing to do with your performance on the MCQs

“Task-based simulation (TBS) questions are pre-assigned and are not chosen based on performance on the multiple-choice testlets.”

I still see people fretting about their terrible sims they saw on their exam and whether it had anything to do with their MCQs. This just reinforces this fact.

Source: Journal of Accountancy

Medium Testlets have difficult questions and Difficult Testlets can have “easier” questions.

“How do you decide which questions are difficult and which are medium?

The difficulty levels of the test questions (and other statistics that are used to describe each test question) are determined through statistical analysis of candidate responses. At the question level, difficulty is not quantified as a category (for example, moderate or difficult), but as a numeric value along a scale. Testlets are classified as either medium or difficult based on the average difficulty of the questions within that testlet.

Does that mean difficult testlets can have easier questions and medium testlets can have difficult questions?

Yes. All testlets have questions ranging in difficulty. Questions in difficult testlets just have a higher average level of difficulty than those in medium testlets.”

“Discrimination” – a back-door method of curving the exam?

“Three statistics describe the questions: difficulty—whether the question is generally easier or more difficult for candidates; discrimination—how well the question differentiates between more able and less able candidates; and guessing—the chances of candidates answering the question correctly just by guessing.”

If you do well on the BEC MCQs, you can bomb the essays and still pass.

“However, it would be very difficult to do so as you would have to perform exceedingly well. It is advisable to be prepared for both the multiple-choice and the written communication questions.”

If you do well on the AUD, FAR, and REG MCQs, you cannot bomb the sims and still pass.

“The portion contribution from task-based simulations in those sections is too large. You would need to get some of the task-based simulations questions correct to pass.”

Source: Journal of Accountancy

Read the entire article here.