RESEARCH ABOUT RESEARCH METHODS
click here to return to methods page

Journals

Action Research International, an on line journal   http://www.scu.edu.au/schools/gcm/ar/ari/arihome.html 
 

Electronic Journal of Business Research Methods    http://www.ejbrm.com/    "provides perspectives on topics relevant to research in the field of business and management."


Harvard Family Research Project's Evaluation Exchange  http://www.hfrp.org/evaluation/the-evaluation-exchange  on line periodical. "addresses current issues facing program evaluators of all levels, with articles written by the most prominent evaluators in the field. "


The International Journal of Qualitative Methods   http://www.ualberta.ca/~ijqm/   "The journal is multilingual and multidisciplinary, with a focus on qualitative research methods."

Journal of Official Statistics   http://www.jos.nu/   many articles about research methods.


Journal of MultiDisciplinary Evaluation   http://survey.ate.wmich.edu/jmde/index.php/jmde_1/index   articles describing evalaution projects, and evaluation issues, methods, etc.


Journal of Research Practice   http://jrp.icaap.org/index.php/jrp   "The Journal aims to highlight the dynamics of research practice, as it unfolds in the life of a researcher, in the growth and decline of a field, and in relation to a changing social and institutional environment."

Marketing Bulletin    http://marketing-bulletin.massey.ac.nz/welcome.asp    On like journal about marketing, but many articles are about survey methods or other research methods. All are available for free.  Some articles are, "Theatre Audience on Stage: Three Experiments Analysing the Effects of Survey Design Features on Survey Response in Audience Research", and "An Experiment with Skip Instructions." 
** Didn't work on 6/5/08. I'll try again **


Methodological Innovations Online   http://erdt.plymouth.ac.uk/mionline/public_html/   This "is an international peer reviewed social research journal. It publishes high quality papers in research methods and methodology from all social science disciplines."


National Centre for Social Research   http://www.natcen.ac.uk/natcen/pages/nm_newsletters.htm   has this Survey Methods Newsletter. Some articles include 'Efficient Sampling with Rare Subgroups' and 'Impact of the Geographical Size of Clusters in the Precision of Survey Estimates'.


Practical Assessment, Research and Evaluation   http://pareonline.net/   some articles about general evaluation and research methodology. For example, "Notes on the use of data transformations" in volume 8.  Also see the Mertler paper below in the section about web surveys.


Qualitative Research Journal   http://www.latrobe.edu.au/aqr/index.php?option=content&task=view&id=17&Itemid=35   or go to the Association site   http://www.latrobe.edu.au/aqr/   and click on journals.  The current journal is password protected, but previous issues are available.

Qualitative Social Research      http://www.qualitative-research.net/index.php/fqs/index    "A peer-reviewed, free and interdisciplinary online journal for qualitative research."  a journal of research about qualitative methods.  Some of the forums include: "Subjectivity and Reflexivity in Qualitative Research II", and "Text, Archive, Re-Analysis", and others.
 

Qualitative Sociology Review   http://www.qualitativesociologyreview.org/ENG/index_eng.php   Research using qualitative methods, and some about qualitative methods. For example: "Oh yes, I remember it well!" Reflections on Using the Life-Grid in Qualitative Interviews with Couples .


The qualitative report    http://www.nova.edu/ssss/QR/index.html    edited on line journal about qualitative research.  A number of articles about qualitative methods.  Also see a section, Practicing Qualitative Research    http://www.nova.edu/ssss/QR/practice.html


Social Research Update    http://sru.soc.surrey.ac.uk/    lots of articles about many different techniques. published quarterly by the Department of Sociology, University of Surrey, England. "Each issue, we will cover one topic in sufficient depth to indicate the main directions of recent developments and provide a bibliography for further reading. If there is room, we shall also list courses, addresses and other useful information."


Surveillance and Society   http://www.surveillance-and-society.org/journal.htm   has a couple of interesting issues. Issue 2/3, 'Doing Surveillance Studies': Methodology Issue, and issue 4, 'People Watching People': Personal Issue.


Survey Methodology   http://www.statcan.gc.ca/ads-annonces/12-001-x/index-eng.htm    From StatCan. "To draw on the expertise of internationally recognized professionals, look no further than Survey Methodology — a publication containing the very latest information on the development and application of survey techniques. Published twice yearly, Survey Methodology focuses on the development and evaluation of techniques used in data collection and interpretation."


Survey Practice   http://surveypractice.org/   A new on line journal from American Association of Public Opinion Researchers. This journal is "designed to provide current information on issues in survey research and public opinion. The articles in Survey Practice emphasize useful and practical information designed to enhance survey quality by providing a forum to share advances in practical survey methods, current information on conditions affecting survey research, and interesting features about surveys and people who work in survey research."


Survey Research Methods   http://www.surveymethods.org/   actually forwards to   http://w4.ub.uni-konstanz.de/srm/   on line peer reviewed journal.  This journal is listed here   http://www.surveymethodology.eu/journal/   and "is the official journal of The European Survey Research Association"   http://www.surveymethodology.eu/home/  


For developing countries, check out the Health InterNetwork Access to Research Initiative   http://www.who.int/hinari/en/   with free or low cost access to journals, for non profit institutions in developing countries. Includes these journals: American Journal of Evaluation, Evaluation and Program Planning, International Journal of Social Research Methodology, and others which may have related information.


Individual papers, proceedings, conferences, etc.

From the US govt

Choi BCK, Pak AWP. A catalog of biases in questionnaires. Prev Chronic Dis [serial online] 2005 Jan [date cited]. Available from: URL:
http://www.cdc.gov/pcd/issues/2005/jan/04_0050.htm   Good overview of errors in questionnaires.

Papers from the Census Bureau www.census.gov/srd/www/byyear.html   Includes :Working Papers in Survey Methodology (SM), which includes research about questionnaire design, identifying the causes of nonsampling errors and evaluations of alternative designs. Another section is The Research Report Series (RR) which covers research in statistical methodology and estimation. Examples of papers include: Improving survey quality through pretesting. by T Demaio, J Rothgeb and J Hess; Questionnaire Design Effects on Interview Outcomes by Jeffrey C. Moore and Laureen H. Moyer; and Better Formatting for Lower Response Burden by Andrew Zukerberg and Meredith Lee.

 

Census papers from Survey of Program Dynamics   http://www.census.gov/spd/pubs.html  Two papers in particular are "SPD98-1: Teens Talk: Are Adolescents Willing and Able to Answer Survey Questions?" and "SPD99-2 Comparing Reponse Rates for SPD, PSID and NLSY"
 

CDC conference proceedings  http://www.cdc.gov/nchs/products/pubs/pubd/other/miscpub/miscpub.htm  lists two publications specifically.    One is the Health Survey Research Methods Conference Proceedings. April 1996, described briefly at http://www.cdc.gov/nchs/products/pubs/pubd/other/miscpub/6hsrm.htm   and downloadable at the miscpub.htm site listed above. A more recent conference proceedings is also listed at the  miscpub.htm  , from 2004,  Health Survey Research Methods: Eighth Conference.   The other is A New Agenda for Interdisciplinary Survey Research Methods: Proceedings of the CASM II Seminar, which reviews current needs and proposes future   directions for interdisciplinary survey methods research.  Described and downloadable from http://www.cdc.gov/nchs/products/pubs/pubd/other/miscpub/casm.htm      Be warned, both are pdf files and large, so over a modem will take a long time to download.

 

FCSM     http://www.fcsm.gov/    The fcsm is the Federal Committee on Statistical Methodology, and has papers available from some conferences.   2007 papers are here   http://www.fcsm.gov/events/papers2007.html     2005 papers are here   http://www.fcsm.gov/events/papers05.html   Has papers about random digit dialing, but also web surveys. 2003 conference papers are available here   http://www.fcsm.gov/events/papers2003.html   2001 papers are available at    http://www.fcsm.gov/events/papers2001.html   listing papers such as Experiences with Incentives in a Public Attitudes Survey Using RDD and Are Late/Difficult Cases in Demographic Survey Interviews Worth the Effort?: A Review of Several Federal Surveys.  The FCSM site also has a report:  Measuring and Reporting Sources of Error in Surveys as pdf.  

The FCSM site also links to other federal agencies with research papers, for example the BLS site:
BLS research papers   http://www.bls.gov/search/osmr.htm   then select statistical survey papers and then search.  Some papers include "Response Analysis Surveys for Assessing Response Errors in Establishment Surveys", "How the Memorability of Events Affects Frequency Judgments," and many others.  Most are from conferences, eg, from SRMS, AAPOR, etc.

BLS also has this   http://www.bls.gov/ore/home.htm   Office of Survey Methods Research (OSMR) which has a number of research papers about survey methods.

 
Studies of Welfare Populations: Data Collection and Research Issues   http://aspe.hhs.gov/hsp/welf-res-data-issues02/   has sections on response rates, non response, missing data, measurement error.
  

Other Conferences

Providing Incentives to Survey Respondents: Final Report   http://www.copafs.org/    report in text, from copafs, Council of Professional Associations on Federal Statistics at their home page,  from 1993.


Proceedings from Bulletin of the International Statistical Institute   http://www.stat.fi/isi99/proceedings.html
    Click on all for a list of all papers.  Some of the papers are about surveys, such as "A Rolling Sample Survey for Yearly and Decennial Uses", and "Alternative Designs for the U.S. Consumer Expenditure Survey", and "The Application and Development of Survey Sampling in China", and lots of others.
    Also see the Baltic-Nordic Conference on Survey Sampling, 2002   http://www.matstat.umu.se/personal/Gunnar/banocoss/   which has some papers available.


Proceedings of the Survey Research Methods Section   http://www.amstat.org/sections/SRMS/proceedings/    Proceedings are available on line 2 years after the  meeting. Has many research papers on various topics.  They contain every methods paper presented at ASA or AAPOR from 1978 through 2 years after the meeting.

SRMS also has links to past conferences    http://www.amstat.org/sections/SRMS/conferences.html    some of which have papers available. For example:
The 21st International Methodology Symposium, "Innovative methods for surveying difficult-to-reach populations"   http://www.statcan.gc.ca/conferences/symposium2004/index-eng.htm   has papers.


MOLS2006. Methodology of Longitudinal Surveys   http://www.iser.essex.ac.uk/events/conferences/mols-2006   has abstracts now but will soon have papers.


Statistics Canada International Symposium Series - Proceedings  http://www.statcan.gc.ca/bsolc/olc-cel/olc-cel?catno=11-522-X&CHROPG=1&lang=eng  has various papers, such as "Center Sampling: A strategy for surveying difficult to reach populations", and "Data collection, sometimes all it takes is a little persuasion ... in the form of information".


Others, organizations, individuals.


Frederick Conrad   http://sitemaker.umich.edu/fred.conrad/home   has papers on  Conversational Interviewing, web surveys, cognitive interviewing. Main focus seems to be on not using standardized interviewing formats, as they can lead to misunderstanding, for example.

Evaluation South Asia   http://www.unicef.org/rosa/monitoring_evaluation.html   "Evaluation South Asia has been recently published. This publication ... looks at some important challenges to the direction in which evaluation in the developing world is heading, and provides some answers to those challenges.  It aims to ... promote an understanding among public institutions and civil societies that evaluation feeds into knowledge management.  It further seeks to show evaluation as a strong tool for improving effectiveness, efficiency and sustainability of programmes, and to demonstrate innovative evaluation methods."

Gary King's papers     http://gking.harvard.edu/preprints.shtml     Papers include "Enhancing the Validity and Cross-cultural Comparability of Survey Research"  and "Misunderstandings among Experimentalists and Observationalists about Causal Inference" 

Institute for Social and Economic Research   http://www.iser.essex.ac.uk/publications/working-papers   working papers include The Impact of a Mixed Mode Data Collection Design on Non-Response Bias on a Business Survey (2005-16),  A Review of Methodological Research Pertinent to Longitudinal Survey Design and Data Collection (2005-29), many others. A search for Peter Lynn will find many of them.


Jon Krosnick   http://communication.stanford.edu/faculty/krosnick.html    papers include Response Rates in Surveys by the News Media and Government Contractor Survey Research Firms, The Survey Response Process in Telephone and Face-to-Face Surveys: Differences in Respondent Satisficing and Social Desirability Response Bias, and many many more.


RTI has research papers    http://www.rti.org/    see one of their conference pages   http://www.rti.org/page.cfm?objectid=848A548F-5661-4F34-9380E267533C3AED   for several survey papers


Statistical Significance Testing Fall 1998 Issue of Research in the schools   http://www.personal.psu.edu/users/d/m/dmr/sigtest/sigtest.htm   the link to the journal is in pdf format, linking to other pdf chapters.  For the journal of research in the schools, but many chapters are general, and can apply to any researcher.


Werner W. Wittmann 's international web page   http://www.psychologie.uni-mannheim.de/psycho2/psycho2.en.php?language=en   click on publications, then papers.  Two papers include "The reliability of change scores: Many misinterpretations of Lord and Cronbach by many others. Revisiting some basics for longitudinal research.", and " How to Fool Yourself with Experiments in Testing Theories in Program Evaluation and Psychological Research."

Washington Statistical Society Methods Seminars, on respondent incentives  http://www.scs.gmu.edu/~wss/methods/index.html   a number of papers on various incentive topics.


National Centre for Social Research, working papers from Department of Work and Pensions    http://www.dwp.gov.uk/asd/asd5/wp-index.asp          includes several research papers, such as these from 2002:  Encouraging and Maintaining Participation in the Families and Children Survey: Understanding why people take part. This research sought to explore the reasons why people (continue to) participate in FACS or to withdraw their co-operation from the survey. Longitudinal qualitative research approaches in evaluation studies.  This paper focuses on the specific role that longitudinal qualitative research can play in addressing the range of research questions that arise in developing and assessing policy. And this paper from 2001: Research Methods for Policy Evaluation.


Propensity Score Estimation with Boosted Regression for Evaluating Causal Effects in Observational Studies   http://www.rand.org/pubs/reprints/RP1164/    about how to deal with  causal effect modeling with naturalistic rather than experimental data.


Websm   http://www.websm.org/   has bunches of papers on all sorts of topics.


Joint Program in Survey Methodology   http://www.jpsm.umd.edu/jpsm/   has some on line presentations (click on on line media). I can't quite get them to work but they look very interesting.


Research about qualitative methods
 

Association for Qualitative research   http://www.latrobe.edu.au/aqr/   The web site has papers from their 1999 conference.  More recent conferences aren't available on line.  One example from the 1999 conf is:  The Process of Synthesising Qualitative Research: A case study http://www.latrobe.edu.au/aqr/offer/papers/HSuri.htm    "The purpose of the main project is to comprehensively synthesise research, both quantitative and qualitative studies"    The main site also links to their journal and you can read the journals except the most recent.

 

Research Methods Division, Academy of Management, In-Depth Discussions and Helpful Tutorials   http://division.aomonline.org/rm/   Annual publications, includes articles about various methods, such as Ethnostatistics, Research Methods and Organizational Behavior,  Not just a numbers thing: Tactics for improving reliability and validity in Qualitative Research. and Some Thoughts on Publishing Qualitative Research.  These are available up through 2002.
 


Specifically about web survey methods

Conducting Research Surveys via E-mail and the Web  http://www.rand.org/pubs/monograph_reports/MR1480/    "The authors discuss the advantages and disadvantages of using e-mail and the Web to conduct research surveys, and also offer practical suggestions for designingand implementing Internet surveys most effectively"

Journal of Computer Mediated Communication   http://jcmc.indiana.edu/   has some articles of interest, for example from April 2005,  Researching Internet-Based Populations: Advantages and Disadvantages of Online Survey Research, Online Questionnaire Authoring Software Packages, and Web Survey Services  at   http://jcmc.indiana.edu/vol10/issue3/wright.html   and a few other articles on the site.


Don Dillman's papers page   http://survey.sesrc.wsu.edu/dillman/papers.htm   as of June 15, included Melevin, Paul T., Don A. Dillman, Rodney Baxter, and C. Ellen Lamiman. 1998. Personal Delivery of Mail Questionnaires for Household Surveys: A Test of Four Retrieval Methods. Revised version forthcoming in Journal of Applied Sociology.  And others.
 

ANZMAC  Australian and New Zealand Marketing Academy  http://www.anzmac.org/       They have conference proceedings, to four conferences, up to 2002.  Click on "proceedings archives" then you can the conferences.Check the papers section.  See for example Medlin, Christopher, Roy, Subroto, and Ham Chai, Theong ( 1999): "World Wide Web Versus Mail Surveys: A Comparison And Report".   There is no description of the papers, so it's kind of hit and miss.
 
 

Virtual Surveys site,   http://www.virtualsurveys.com/   See their site map and then papers page, which include Pop-up Surveys. What works, what doesn't work and what will work in the future.    and a few others.
 

Mertler, Craig A. (2003). Patterns of response and nonresponse from teachers to traditional and web surveys.    http://pareonline.net/getvn.asp?v=8&n=22    found different response rates but similar responses. 


Designing an Intranet User Survey   http://www.intranetjournal.com/articles/200412/ij_12_13_04a.html   brief intro. Paul Chin  12/13/2004


Public Opinion Quarterly.  http://poq.oxfordjournals.org/archive/   Papers from Spring 2004 are free access. One paper, for example is "A Comparison of Web and Mail Survey Response Rates"   http://poq.oxfordjournals.org/cgi/content/full/68/1/94   Michael D. Kaplowitz, Timothy D. Hadlock and Ralph Levine.  Public Opinion Quarterly 2004 68(1):94-101; doi:10.1093/poq/nfh006
  also articles from 2007, vol 71, number 5   http://poq.oxfordjournals.org/content/vol71/issue5/index.dtl   are free. This issue is about cell phones.


Are representative internet surveys possible?  http://www.statcan.gc.ca/bsolc/olc-cel/olc-cel?catno=11-522-X20010016291&lang=eng   2001 study by Tom Smith.


Comparing the Results of Probability and Non-Probability Sample Surveys   http://communication.stanford.edu/faculty/krosnick.html   by Jon Krosnick, about comparing internet surveys with other methods.


Cell phones

Public Opinion Quarterly, articles from 2007, vol 71, number 5   http://poq.oxfordjournals.org/content/vol71/issue5/index.dtl   are free. This issue is about cell phones.

DC-AAPOR Workshop on Cell Phone Numbers and Telephone Surveying in the U.S.   http://www.dc-aapor.org/cpworkshop.php   sept 4, 2008



SITES ABOUT MISSING DATA, non response, multiple imputation, etc
 

Prevention and Treatment of Item Nonresponse    http://www.jos.nu/Articles/abstract.asp?article=192153    Edith D. de Leeuw, Joop Hox, and Mark Huisman. "we present a typology of missing data patterns and their origins. Based on this typology and the potential sources of item nonresponse, we outline how missing data can be prevented as much as possible. Finally, we discuss how knowledge of the data collection process can improve the statistical treatment of the remaining missing data."


Univ of Texas Statistical FAQs, missing data   http://www.utexas.edu/its-archive/rc/answers/general/gen25.html  overview of missing data, how to handle, some software links.


Joop Hox's home page  http://www.geocities.com/joophox/   Especially see the papers and programs sections, which includes presentations on a class he gave on missing data.  Dr. Hox is a faculty of social science methodology at  Faculty of Social Sciences of Utrecht University. Some of his papers are listed here   http://www.geocities.com/joophox/publist/pubenjh.htm   

 

The multiple imputation FAQ page    http://www.stat.psu.edu/~jls/mifaq.html    questions and answers about multiple imputation.  Also see his home page    http://www.stat.psu.edu/~jls/    for some presentations on multiple imputation.
 

Methodology for collecting data about survey non-respondents, an ISER working paper, listed at   http://www.iser.essex.ac.uk/pubs/workpaps/index.php   Click on working paper box, then its 2002, paper 2002-05.  This paper reports the development of a method to collect limited survey data from non-respondents to personal interview surveys.  This site also has some other interesting working papers on various methods issues. For example, "Respondent Behaviour in Panel Studies - A Case Study for Income-Nonresponse by Means of the British Household Panel Study" is a study of motives for non response about income. This is a March 2003 working paper.
 

Interagency Household Survey Nonresponse Group (IHSNG)   http://www.fcsm.gov/committees/ihsng/ihsng.htm    bunch of papers about non response, for example, "Measuring Survey Nonresponse by Race and Ethnicity".
 

Studies of Welfare Populations: Data Collection and Research Issues    http://aspe.hhs.gov/hsp/welf-res-data-issues02/index.htm    several chapters about response rates, non response, missing.


Statistical Clearinghouse   http://www.nss.gov.au/nss/home.NSF/pages/About+SCH?OpenDocument   click on "seminars". One is "Drawing Conclusions from Surveys with a Low Response Rate" but there are other seminars of interest. Also see research papers here  http://www.nss.gov.au/nss/home.NSF/ResearchPapers?OpenView  with papers such as Low Response Rates and Their Effects on Survey Results.


NSM14: Measuring and improving data quality
http://www.statistics.gov.uk/StatBase/Product.asp?vlnk=3138
This monograph is the second report of the GSS(M) Task Force on non-sampling error. It describes the various sources of non-sampling errors, discusses ways to measure the bias and increase in variance, which may result and provides a guide to methods, which can be used to reduce the impact on data quality.
listed on this page
http://www.statistics.gov.uk/methods_quality/publications.asp
along with others



Research about survey scales, e.g., use of neutral, don't know, and so on.


THE SCIENCE OF ASKING QUESTIONS,  Nora Cate Schaeffer and ­Stanley Presser
http://arjournals.annualreviews.org/doi/abs/10.1146/annurev.soc.29.110702.110112
Annual Review of Sociology, Vol. 29: 65-88 (Volume publication date August 2003)
(doi:10.1146/annurev.soc.29.110702.110112)
people in non university settings (like me) can only see abstract, but a personal use copy is available here
http://www.ssc.wisc.edu/cde/faculty/schaeffe.htm   provided by the author. For personal use only.


The "Don't Know", "Undecided", and "Neutral" Response Options
from statpac  http://www.statpac.com/surveys/undecided-category.htm


Middle Alternatives, Acquiescence, and the Quality of Questionnaire Data
Colm O’Muircheartaigh, Jon A. Krosnick, and Armin Helic. The Harris School, Working Paper Series: 01.3   http://harrisschool.uchicago.edu/about/publications/working-papers/abstract.asp?paper_no=01.03+++  (if that doesn't work,
http://harrisschool.uchicago.edu/About/publications/working%2Dpapers/pdf/wp_01_3.pdf)


Krosnick, J.A., et al. 2002.  'The Impact of "No Opinion' Response Options on Data Quality-Non Attitude Reduction or an Invitation to Satisfice?"  Public Opinion Quarterly  66: 371-403.
http://are.berkeley.edu/~hanemann/


Jolene D. Smyth, Don A. Dillman, Leah Melani Christian, and Michael J. Stern. 2005.
Comparing Check-All and Forced-Choice Question Formats in Web Surveys: The Role of Satisficing, Depth of Processing, and Acquiescence in Explaining Differences. Social and Economic Science Research Center Technical Report 05-029.
Washington State University: Pullman. 30pp.
http://survey.sesrc.wsu.edu/dillman/papers.htm


And if you got to here, check out this very interesting SPOOF


Journal of Obnoxious Statistics    http://www.xs4all.nl/~edithl/jobs.htm    The Journal of Obnoxious Statistics (JOBS) is created specially for Lars Lyberg by his friends and colleagues to commemorate the 20th anniversary of the Journal of Official Statistics.

 

last verified 12/25/08
Last updated 12/25/08
click here to return to methods page


 

Australasian Evaluation Society    http://www.aes.asn.au/    has some papers from their conferences available. Not always 'research' but interesting. They also have the Evaluation Journal of Australasia   http://www.aes.asn.au/publications/journal.html   with many research papers.
  Didn't work on 12/25/08.  I'll check again.