What exactly does e.g. "95% comprehension" mean?

Question

I recently browsed the paper Schmitt, Jiang, and Grabe, The Percentage of Words Known in a Text and Reading Comprehension, The Modern Language Journal (2011).

Its main aim was to find the relationship between x% vocabulary coverage (i.e., the proportion of words a student knows) and x% comprehension (measured using a comprehension test). They found:

The total increase in comprehension went from about 50% comprehension at 90% vocabulary coverage to about 75% comprehension at 100% vocabulary coverage.
_{Schmitt, Jiang, and Grabe}

So clearly vocabulary coverage and comprehension are distinct concepts.

But this raises the question as to what e.g. "95% comprehension" means. Prof. Stephen Krashen mentions it in this interview:

... the studies range from 90 to 98 percent 95 percent; depends on the text, depends on you...

It's not clear to me what mental picture Prof. Krashen has here, especially in light of how 100% vocabulary coverage only gives about 75% comprehension, judging from Schmitt, Jiang, and Grabe (2011).

Question: What exactly does e.g. "95% comprehension" mean?

Answer 1

Reading comprehension involves asking questions of readers after they read a text. Typically, these would be vocabulary questions and questions about meaning. These are then graded using whatever scale the test designer wants to use.

The scale would assign percentages or numbers to the questions. So, depending the number of correct answers from test taker answers correctly, a percentage of comprehension can be devised.

Here's a round-up of the type of typical test one might see:

What's the format of a reading comprehension test? Reading comprehension tests typically consist of a passage of text followed by a series of questions related to the content of the passage. You should not use any outside knowledge of the subject matter, only base your answers on what is written in the passage. The purpose of the test is to assess your ability to understand and analyse written information.

The length of the passage can vary, especially depending on the type of question being asked. The content of the passage can be on any topic, from science to history to literature.

The questions that follow the passage can come in different formats, but some common types include:

Multiple choice questions: These questions provide a list of options, and you must select the correct answer from the choices given.
True or false questions: These questions ask you to determine whether a statement about the passage is true or false.
Short answer questions: These questions require you to provide a brief answer in your own words, they will typically be one-word answers.

Matching questions: These questions ask you to match a term or concept from the passage with its definition or description. Open-ended questions: These questions require you to provide a detailed response in your own words, often asking you to analyse or interpret the passage.

reading comprehension tests

Every test will have its own scoring rules. So, for example, if you have 20 questions, and the test taker gets 18 right, one would say that's a 95% comprehension rate because 18 out of 20 is 95% of it.

What is discussed above is for older kids, not little ones (5-9).

For the text cited by the OP, one would have to look at the questions asked of the readers and the way the tests are scored.

Answer 2

Reading comprehension means how much of a text one understands by reading. This is, of course, not helpful. Each and every research project has to operationalize the concept of reading comprehension; that is, make it measurable or otherwise useful. The results vary significantly based on the operationalization. A given field of empirical research might have developed standard ways of operationalizing any given concept, or might not have done it.

Schmitt et al. discuss their operationalization on pages 31-32 of the article. They pose two types of questions, MC (maybe multiple choice?) and GO (graphic organizer) completion tasks, also called «information transfer tasks». They have further references on MC tasks and their validity.

It seems that in the article in question reading comprehension then means the ratio of these tasks that are answered correctly. So 70 % comprehension means that 70 % correctness.

There is no basis to assume that Krashen means the same thing with reading comprehension. I imagine he is not thinking of a particular operationalization in a particular study, but rather the more general meaning of how much of a text one understands, and 95 % might simply mean that the text is understood very well, with maybe some incidental details missed.

Knowing all the words in a text does not imply full comprehension, because the text can be cognitively demanding to understand or in an unknown domain or simply not make sense. You might want to try this by reading legal texts, some university-level mathematics texts and instructions for repairing or adjusting a bicycle. Even if you understand the words, understanding the contents requires extra cognitive effort on top of that. Reading is more than understanding words.