Social Science questionare how reliable are they?

[Wordy]

Social Science often use questionares. How reliable are they.
I have a link to an example of social psychology that use questionare
but I wonder how reliable such are.

Take the feeling of disgust. On a scale from 1 to 5 rate the following.

Eating cooked flesh from Pigs.
Eating a cooked small worm.
Eating raw apple.
Eating cooked puppy Dog.

Here in Sweden eating the cooked puppy Dog would be very disgusting.
Dogs are almost treated as humans. They are family members and it is
absolute taboo to eat meat from dogs.
but in other cultures that is seen as a delicatesse or totally ok to eat.

If your religion forbid you to eat pork then maybe you feel disgust.

What I am skeptical of is that one ask a person to judge on a scale how something feels. Is that really possible.

Doesn't experience tell us that we are extremely poor at knowing what goes on within us.

Wouldn't the result be more reliable if one actually gave them something that looked like pork or worm or puppy meat and asked them to just eat it and see what they do. If their body language show they don't want to that could be more reliable then to ask them in a questionare without having to confront the actual eating.

Same when they do study on religious feelings. To just ask what one feels is as my experiences tells me very unreliable.

Ok I could be very different than most people, a bit nerd in approach and kind of socially clumsy and not typical at all. So my experiences could be unique?

But my body kept my religious feelings hidden to my conscious recall for some 20 years so if a social scientist had asked me every year how I felt about religious matters then the questionare would have been misleading for some 20 years without me or them knowing about it.

but if they had made video recording instead for asking me. If they have played a Mahalia Jackson Gospel then I would have looked enthusiastic and happy and they would hear me say things like. "Why on earth doesn't we atheists sings like this?" but if they had asked me I would have written that I hated anything religious and had no feelings other than hate for it. "Delete all trace of religion from history!" I would have told them.

After these 20 years of hate had taken its toll on my body it gave up and let me feel the religious feelings me had had since me ten years old but which had kept hidden to me because of the cognitive dissonance.

I care very much about science, I want science research to be reliable and the more I look into the usage of questionare I get skeptical on how reliable it is.

Is it not a known fact that people tend to over estimate their own abilities to know their own shortcomings. They think highly of themselves. we seems to be built that way by evolution.

link soon

http://faculty.virginia.edu/haidtlab/mf.html

Moral Foundations Theory Homepage

A collaboration among Jon Haidt, Jesse Graham, Brian Nosek, Craig Joseph, Pete Ditto, Ravi Iyer, Sena Koleva,
and anyone else who wants to critique the theory or propose different ways to divide up the moral domain.

They use questionare. They challenge critics to come up with better versions. 1000USD if you are able to come up with better theory?

[nygreenguy]

I believe most of these people also use statistics. While not perfect, it can correct for those strange outliers, or it can tell you that your perceptions are wrong on what the public thinks!

I would also like to think that they take the culture into account. So they would ask different questions in different areas.

I simply havent seen enough evidence to make me not think they dont know what they are doing. That would take quite a hefty bit of evidence.

[Wordy]

Yes I know they try to compensate for such but the result seems to indicate they trust self reporting too much.

That is how I get it. But it could be cause me has had experiences that tells me that it one can't rely on what people say or answer in questionares.

none of my cohabitats know them to be religious and both of them ended up as believers.

For twenty years me had no idea that I felt religious and when I woke up to that feeling I knew me had had it the whole time just that it was kept hidden to my conscious access.

So close to me I have three persons that would have given wrong answer to the social science questions cause we had no way of knowing what was really going on.

Their questions fail to get to what is real and only touch on what is accessable through conscious recall or superficial access.

But that is not the truth, that is a kind of social lie to keep face socially.

As a strong atheist there was no way for me to save face knowing me to feel religious.
the cognitive dissonance was too strong for my body to allow me to be aware of those feelings. So it was hidden to me.

Now not everybody have these feelings so they maybe do have access to the lack of these feelings but both my cohabitats didn't know either. that is alarming. They where the most trusted persons me have been involved with.

So the social science don't capture reality, it only see what the body allow to be revealed and that is a social lie to keep face.

[VenDexter]

If a questionnaire is designed well and revised over time, it can be made to measure the level to which someone is being "dishonest" to save face. Actually, honest vs. dishonest probably isn't the best way to describe it it. Consistent vs. inconsistent would be a better measure.

A good example of this is the Minnesota Multiphasic Personality Inventory. It repeats questions is several different ways using different approaches to help determine if the person is answering consistently.

By answering consistently, it is determined that the person has a congruent sense of self and for the most part is being honest.

[Wordy]

The Personality Test
The staple of modern Scientology recruitment is the Personality Test, ...... The three, plus the Minnesota Multiphasic Personality Inventory used in the ...
www.clambake.org/archive/oca/oca.html

Maybe there is two versions of it. The original and the one Sciento use. I don't trust these tests. I 've done many of them and it is not easy to answer them honestly at all.

Could you give us an example from it and your answer to that question and why that would be the correct answer?

[hecaterin]

Well, there is a lot of research in good social science methodology. I used to teach that shit once, before I got totally sick of the whole academic thing and turned myself into a programmer.

There are a lot of things you can do to improve the reliability of your instruments. You have to be very cautious with phrasing, and do pre-tests of questionnaires before unleashing them for real. You need to balance options, and in a long questionnaire, ask effectively the same question several times, making sure the phrasing is equally positive and negative. There are statistics you can run to check correlations between the questions to check for inconsistencies - if one question stands out badly it may be a f'd up question that you missed in the pre-test; if one person's answers vary wildly, they may be mischievous.

You also have to be very cautious with sample selection. And be aware that you are testing what people *say* that they do, not what they actually do, nor what they think they do, or think that they ought to do.

I'm often horrified at how bad it can get, and still get published, though. It's trivially easy to manipulate questions to get what you want to hear.

[Wordy]

hecaterin,
I am obviously biased in this thing. When I read you almost all supprot what I wrote.
But I guess you don't see it that way?

To all of you guys.

I found this ?pop? psychology test. Not social science formally but about how social we are and a kind of example on an "instrument" a kind of questionare as I spell them.

The EQ test
http://www.eqsq.com/eqtest.php

Now as many of you know. It all depends on how well they have done the questions so they are balanced and not trigger the person answering too much in either direction.

But is that easy at all. I try to quote a few of them.

I can easily tell if someone else wants to enter a conversation.

Suppose me was good at empathy and had good social skills then I would answer strongly agree to that claim. But how are they gonna have any reliable estimation on how people answers.

Is it not expected of us to be able to do such things. So will not those answering overestimate their ability to do such things. The only way to find out is to have hidden cameras and put them in situation where someone try to get a say in a conversation.

I really enjoy caring for other people.

I know a lady in my hometown that gladly would answer strongly agree to that. she loves to do such things. But she has almost no empathic skills what so ever.
She only thinks she has. She is a brute who force her opinion on others wanting to "mothering" them to comply with her wishes on what is best for them to do in their lives.

I find it hard to know what to do in a social situation.

That claim rather point out those who are bad at empaty and bad at social skills.
But it fails on those who are sociopaths and manipulators or others like them cause they are good at knowing what to do but have only the intellectual knowledge of what it is like to have empathy and really don't care what happens to their victims. Or even love to inflict sorrow on others. The more the merrier. Does the question really work as a measurment on EQ as it is set up to do?

I guess I get illegal if I quote too much.

what I try to say is that not all persons are good at reading such questions and answer in a way predictable by those posing the questions.

Yes I know they do it again and again and change until they get reliable result.

But how do they know what is a reliable result?

Maybe me overdo it but as far as I know these sort of research where you put questions in front of persons and expect them to answer and then "measure" is not to be trusted as far as I get it.

A maybe odd example is the question that Garrett pose in this forum.

"Beam me up Scotty" Would you accept to be shot with a gun on earth for to go to Mars via a kind of teleportation invention? Not word for word but you get it.

Materialist are supposed to answer yes and if you answer no then you don't get what being a materialist is kind of challenge.

Typically me ended up saying no and thus being a Materialist/Physicalist not knowing what that is. Mission accomplished I guess.

http://talkrational.org/showthread.php?t=1064&highlight=scotty

[Fathermithras]

As noted above, there are numerous statistical and research methods that are used when you get into the quality surveys that are used. They can detect false answers reliably among other things. Not sure about all surveys, but in psychology designing, norming, analyzing, and giving these tests are honed to a very reliable finished product through years of prep.