does the ratio of the mean to the sample size indicate anything interesting?

[fleet-footed Urban Youth]

say two classes of students take the same test. on this test, they have to answer their choice of 5 questions out of a pool of 10 questions.

for one class, if the average points earned for a certain question is 10, and the number of students who actually answered that question is 5, then the ratio here is 2.

for the other class, if the average points earned for the same question is 15, and the number of students who actually answered that question is 3, then the ratio here is 5.

the difference in this metric between the first and second class is -3, and the ratio between the first and second class is 40%.

so, my question: who gives a shit? does this mean anything? could it indicate anything about student performance, preparation, judgement, confidence, etc. on a class-wide scale? it seems like it should but i have no idea whether it does and if so, what?

[Mike PSS]

Are the questions equally "difficult"?
Are they all within the students field of study?

[fleet-footed Urban Youth]

1. they're all supposed to be equally difficult but at the end, it might be discovered that students do significantly worse on some than on others. in that case, grade adjustments can be made. what i think might be interesting is to look at the ratio i'm talking about for these "difficult" questions. if one class does much better on one of these than the other but fewer overall answer it, does that indicate better judgement about what questions to attempt? better preparation?

2. no, actually few of the students would be majors in the test's field because it's a survey course.

[schneibler elf]

holy shit, I just completed my calculations on this, you'll never believe what I came up with

8=====D

[fleet-footed Urban Youth]

[Mike PSS]

someone's excited about the results

[Mike PSS]

1. they're all supposed to be equally difficult but at the end, it might be discovered that students do significantly worse on some than on others. in that case, grade adjustments can be made. what i think might be interesting is to look at the ratio i'm talking about for these "difficult" questions. if one class does much better on one of these than the other but fewer overall answer it, does that indicate better judgement about what questions to attempt? better preparation?

2. no, actually few of the students would be majors in the test's field because it's a survey course.

Take this as opinion.

The ratio you indicate has no real value since you already have the average score per pupil. Value would be better placed in the standard deviation of answers since in the first class you could have the top three students score close to a 15 average while the bottom two scored low to give a low average (and thus a low ratio).

To reinforce this, your answer to question 2 indicates that there is non-learning bias in the selection of questions to answer; a student will think that they know the subject. Therefore the score average is a better indicator of the humility of the 2nd class versus the 1st class; the 2nd class better recognizes their skills and shortfalls (at least on that question). If this trend held for all 5 answers then this would be reinforced. I'm sure Pingu or someone else has a term for this type of selection bias.

[el jefe]

without really plugging into the problem, it strikes me that the number of students who answered the question should be divided by the total number of students in the respective class. otherwise, whatever you're attempting to measure is being thrown off by simple class size, which is presumably not of interest.

eta: then, the ratio you're considering is more like (mean score)/(how often people chose to answer it)

[fleet-footed Urban Youth]

Take this as opinion.

The ratio you indicate has no real value since you already have the average score per pupil. Value would be better placed in the standard deviation of answers since in the first class you could have the top three students score close to a 15 average while the bottom two scored low to give a low average (and thus a low ratio).

To reinforce this, your answer to question 2 indicates that there is non-learning bias in the selection of questions to answer; a student will think that they know the subject. Therefore the score average is a better indicator of the humility of the 2nd class versus the 1st class; the 2nd class better recognizes their skills and shortfalls (at least on that question). If this trend held for all 5 answers then this would be reinforced. I'm sure Pingu or someone else has a term for this type of selection bias.

honestly that's kinda the conclusion i had been coming to. sometimes there seems to be a relationship (like in the example i mentioned) and sometimes you see the exact same ratio between two questions where for one the averages are about the same and for the other they're really different because of how many actually answered them. oh well. i didn't really need it, was just kinda procrastinating with spreadsheets.

[fleet-footed Urban Youth]

without really plugging into the problem, it strikes me that the number of students who answered the question should be divided by the total number of students in the respective class. otherwise, whatever you're attempting to measure is being thrown off by simple class size, which is presumably not of interest.

eta: then, the ration you're considering is more like (mean score)/(likelihood an arbitrary student will answer it)

i was thinking about something like that too but when i tried it, i couldn't figure out what the fuck the numbers were actually telling me.

[buttershug]

without really plugging into the problem, it strikes me that the number of students who answered the question should be divided by the total number of students in the respective class. otherwise, whatever you're attempting to measure is being thrown off by simple class size, which is presumably not of interest.

eta: then, the ratio you're considering is more like (mean score)/(how often people chose to answer it)

or?
(mean score)/(100-per cent of students that got it right)
or
(mean score)(per cent of students that got it right)

And if you get the ratios for all the questions and then get the mean of them and the standard deviation for each ratio, you can get an idea if any particular question gives too many or too few points.

er sumtin along those lines.

[fleet-footed Urban Youth]

can't count the number of students who got it right because the answers aren't binary. there are degrees of correctness.

eta: or, there would be a number of different ways to do this: "who got no credit," "who got at least passing credit," "who got an average degree of credit," "who got a perfect score."

[el jefe]

i was thinking about something like that too but when i tried it, i couldn't figure out what the fuck the numbers were actually telling me.

i think it sort of measures how well the problem ' s actual difficulty matches with its apparent difficulty. a really high mean score, and comparatively few people attempting it suggests a given problem is easier than it looks. and, conversely, a low mean score and high frequency of attempts suggests a problem is harder than it looks.

[fleet-footed Urban Youth]

it's just that the end numbers could range from 0.8 to 73 or some shit. maybe i had to put them in a ratio as well....?? gah! math!!!!

[el jefe]

that might be fine. it wouldn't give you an "absolute" scale, but it would still tell you how each of the problems compare to each other. this one is the most deceptively difficult, that one is the least. etc.

[gandalf]

Have u tried taking teh l0g of teh ratio l0gging alwayz l00kz way m0ar sofistik8d

[Pingu]

The Frequency & Discrimination Analysis is accessed through 'Reports' and is used to look at the number of students that have selected each option in a question and how well it discriminates between the upper and lower 27% of students. These values help to identify how well the question is working.

The item difficulty of each question, ranges from 0.0 to 1.0, and is calculated as the average score for the question divided by the maximum achievable score. An item difficulty of 0.0 indicates a very hard question (no-one got it right), while a value of 1.0 shows a very easy question (no-one got it wrong). Questions that have item difficulties of more than 0.75 or less than 0.25 are less effective in differentiating users. An ideal value is 0.5.

If the number of students getting the answer correct is under 50% (t < 50%) then double-check that the correct answer has been properly set when editing the question. If this is not the case then the correct answer can be set see: Marking: Correcting Mistakes.

Discrimination ranges from -1.0 to +1.0, and is a Pearson product-moment correlation between the item and the total test score. A correlation close to +1.0 means that the question is measuring the same thing as the test. A low correlation shows that getting the question right is not related to a good test score. A negative correlation means that getting the question right associates with a low-test score. Users who answer correctly questions with a discrimination of less than 0.25 up to -1 do not necessarily do well in the test as a whole.

The report (see above) shows the number of students in each group (u=upper group, l=lower group) that correctly answered each option in each question and the corresponding discrimination value (d). Questions that discriminate well have a value of above 0.2 although a low value may be acceptable for core knowledge.

hth

[fleet-footed Urban Youth]

cool

[Pingu]

That's for Right/Wrong questions of course, but you could probably adapt it to questions with continuous grades.

[Autonemesis]

https://en.wikipedia.org/wiki/Quarti..._of_dispersion
https://en.wikipedia.org/wiki/Coefficient_of_variation

[PerseusOmega9]

say two classes of students take the same test. on this test, they have to answer their choice of 5 questions out of a pool of 10 questions.

for one class, if the average points earned for a certain question is 10, and the number of students who actually answered that question is 5, then the ratio here is 2.

for the other class, if the average points earned for the same question is 15, and the number of students who actually answered that question is 3, then the ratio here is 5.

the difference in this metric between the first and second class is -3, and the ratio between the first and second class is 40%.

so, my question: who gives a shit? does this mean anything? could it indicate anything about student performance, preparation, judgement, confidence, etc. on a class-wide scale? it seems like it should but i have no idea whether it does and if so, what?

http://talkrational.org/showthread.php?t=67279