diff --git a/.idea/.gitignore b/.idea/.gitignore new file mode 100644 index 0000000..b58b603 --- /dev/null +++ b/.idea/.gitignore @@ -0,0 +1,5 @@ +# Default ignored files +/shelf/ +/workspace.xml +# Editor-based HTTP Client requests +/httpRequests/ diff --git a/.idea/EmpiricalStandards.iml b/.idea/EmpiricalStandards.iml new file mode 100644 index 0000000..0c8867d --- /dev/null +++ b/.idea/EmpiricalStandards.iml @@ -0,0 +1,12 @@ + + + + + + + + + + + + \ No newline at end of file diff --git a/.idea/modules.xml b/.idea/modules.xml new file mode 100644 index 0000000..3bf1eda --- /dev/null +++ b/.idea/modules.xml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/.idea/vcs.xml b/.idea/vcs.xml new file mode 100644 index 0000000..94a25f7 --- /dev/null +++ b/.idea/vcs.xml @@ -0,0 +1,6 @@ + + + + + + \ No newline at end of file diff --git a/Standards/ActionResearch.md b/Standards/ActionResearch.md index 9749690..5809254 100644 --- a/Standards/ActionResearch.md +++ b/Standards/ActionResearch.md @@ -1,4 +1,5 @@ # Action Research + Empirical research that investigates how an intervention, like the introduction of a method or tool, affects a real-life context @@ -28,6 +29,7 @@ standards. ## Specific Attributes ### Essential Attributes + - describes the context or site of the intervention(s) @@ -52,8 +54,11 @@ standards. - reports lessons learned by the organization - researchers reflect on their own possible biases - + + ### Desirable Attributes + + - uses direct quotations extensively - uses member checking to assess resonance @@ -61,11 +66,14 @@ standards. - findings plausibly transferable to other contexts - triangulation across quantitative and qualitative data - + + ### Extraordinary Attributes + - research team with triangulation across researchers (to mitigate researcher bias) - + + ## General Quality Criteria Example criteria include reflexivity, credibility, resonance, usefulness diff --git a/Standards/CaseStudy.md b/Standards/CaseStudy.md index 161a3ba..8df0453 100644 --- a/Standards/CaseStudy.md +++ b/Standards/CaseStudy.md @@ -1,4 +1,5 @@ # Case Study and Ethnography + "An empirical inquiry that investigates a contemporary phenomenon (the "case") in depth and within its real-world context, especially when the @@ -35,6 +36,8 @@ captures a large quantitative dataset with limited context, consider the ## Specific Attributes ### Essential Attributes + + - explains why the case study approach is appropriate for the research question - justifies the selection of the case or site that was studied @@ -46,8 +49,11 @@ captures a large quantitative dataset with limited context, consider the - presents a clear and well-argued "chain of evidence" from observations to findings - clearly answers the research question(s) - + + - reports the type of case study (see *Types of Case Studies*, below) - describes external events and other factors that may have affected the case or site @@ -57,8 +63,11 @@ captures a large quantitative dataset with limited context, consider the - cross-checks interviewee statements (e.g. against direct observation or archival records) - uses quotations to *illustrate* findings (note: quotations should not be *the only* representation of a finding; each finding should be described independently of supporting quotations) - + + ### Extraordinary Attributes + + - multiple, deep, fully-developed cases with cross-case triangulation - uses multiple judges and reports inter-rater reliability (cf. Gwet & Gwet 2002) @@ -66,7 +75,8 @@ captures a large quantitative dataset with limited context, consider the - uses direct observation and clearly integrates direct observations into results - created a case study protocol beforehand and makes it publicly accessible - + + ## General Quality Criteria Case studies should be evaluated using qualitative validity criteria diff --git a/Standards/EngineeringResearch.md b/Standards/EngineeringResearch.md index fef95ce..6b6d08d 100644 --- a/Standards/EngineeringResearch.md +++ b/Standards/EngineeringResearch.md @@ -1,4 +1,5 @@ # Engineering Methods + ## Engineering Research (AKA Design Science) @@ -20,6 +21,8 @@ standard is not appropriate for: ## Specific Attributes ### Essential Attributes + + - describes the proposed artifact in adequate detail^1^ - justifies the need for, usefulness of, or relevance of the proposed artifact^2^ @@ -38,8 +41,11 @@ standard is not appropriate for: - assumptions (if any) are explicit; do not contradict each other or the contribution's goals; plausibly hold for the evaluation subjects - uses notation consistently (if any notation is used) - + + ### Desirable Attributes + + - reviews the theoretical basis of the artifact - provides correctness arguments of the key analytical and theoretical contributions (e.g. theorems, complexity analyses, mathematical proofs) @@ -51,12 +57,15 @@ standard is not appropriate for: - provides a replication package including datasets and analytical scripts and EITHER a comprehensive description of the artifact OR source code if artifact is virtual - justifies any items missing from replication package based on practical or ethical grounds. - + + ### Extraordinary Attributes + - contributes to our collective understanding of design practices or principles - presents ground-breaking innovations with obvious real-world benefits - + + ## General Quality Criteria - Comprehensiveness of proposed artifact description diff --git a/Standards/Experiments.md b/Standards/Experiments.md index ec8b3c5..01a62a6 100644 --- a/Standards/Experiments.md +++ b/Standards/Experiments.md @@ -1,4 +1,5 @@ # Experiments (with Human Participants) + A study in which an intervention is deliberately introduced to observe its effects on some aspects of reality under controlled conditions @@ -24,7 +25,9 @@ Science Standard** or the **Engineering Research Standard**. ## Specific Attributes -### Essential Attributes +### Essential Attributes + + - describes how characteristics of phenomenon under investigation relate to experimental constructs - states formal hypotheses - justifies use of one-sided hypotheses (if any) based on face validity or previous work @@ -42,8 +45,11 @@ Science Standard** or the **Engineering Research Standard**. - EITHER: shares raw, de-identified data OR: explains why sharing raw data is impractical or unethical - discusses construct, conclusion internal and external validity - discusses alternative interpretations of results + + +### Desirable Attributes + -### Desirable Attributes - justifies hypotheses and Bayesian priors (if applicable) based on previous studies and theory - discusses alternative experimental designs and why they were not used (e.g. validity trade-offs) - includes visualizations of data distributions @@ -55,12 +61,15 @@ Science Standard** or the **Engineering Research Standard**. - analyzes construct validity of dependent variable - uses and reports manipulation checks - pre-registration of hypotheses and design where venue allows + + +### Extraordinary Attributes + -### Extraordinary Attributes - reports multiple experiments or replications in different cultures or regions - uses multiple methods of data collection; data triangulation - longitudinal data collection with appropriate time-series analysis - + ## General Quality Criteria diff --git a/Standards/GeneralStandard.md b/Standards/GeneralStandard.md index cd1968c..23a782e 100644 --- a/Standards/GeneralStandard.md +++ b/Standards/GeneralStandard.md @@ -1,4 +1,5 @@ # The General Standard + ## Application @@ -44,6 +45,8 @@ following. ## Specific Attributes ### Essential Attributes + + - states a purpose, problem, objective, or research question - methodology is appropriate (not necessarily optimal) for stated purpose or questions - describes in detail what, where, when and how data were collected @@ -57,8 +60,11 @@ following. - language is not misleading; any grammatical problems do not substantially hinder understanding - visualizations/graphs are not misleading (see the Information Visualization Supplement) - complies with all applicable empirical standards + + +### Desirable Attributes + -### Desirable Attributes - summarizes and synthesizes a reasonable selection of related work - clearly describes relationship between contribution(s) and related work - states epistemological stance (e.g. post-positivism, interpretivism, critical realism) @@ -71,12 +77,15 @@ following. - visualizations (e.g. graphs, diagrams, tables) advance the paper’s arguments or contribution - clarifies the roles and responsibilities of the researchers (i.e. who did what?) - provides an auto-reflection or assessment of the authors’ own work (e.g. lessons learned) - + + ### Extraordinary Attributes + + - applies two or more data collection or analysis strategies to the same research question (see Multimethodology Supplement) - approaches the same research question(s) from multiple epistemological perspectives - innovates on research methodology while completing an empirical study - + ## General Quality Criteria diff --git a/Standards/GroundedTheory.md b/Standards/GroundedTheory.md index 02ea885..a5f8700 100644 --- a/Standards/GroundedTheory.md +++ b/Standards/GroundedTheory.md @@ -1,4 +1,5 @@ # Grounded Theory + A study of a specific area of interest or phenomenon that involves iterative and interleaved rounds of qualitative data collection and @@ -23,6 +24,7 @@ collection and analysis or do not use theoretical sampling, consider the **Case Study Standard** or the **Qualitative Survey Standard**. ## Specific Attributes + ### Essential Attributes - identifies the version of Grounded Theory used/adapted (Glaser, Strauss-Corbin, Charmaz, etc.) @@ -31,8 +33,11 @@ collection and analysis or do not use theoretical sampling, consider the - provides evidence of saturation; explains how saturation was achieved - explains how key patterns (e.g. categories) emerged from GT steps (e.g. selective coding) - provides clear chain of evidence from raw data (e.g. interviewee quotations) to derived codes, concepts, and categories - + + ### Desirable Attributes + + - explains how and why study adapts or deviates from claimed GT version - presents a mature, fully-developed theory or taxonomy - includes highly diverse participants and/or data sources (e.g. software repositories, forums) @@ -45,11 +50,14 @@ collection and analysis or do not use theoretical sampling, consider the - explains theoretical sampling vis-à-vis the interplay between the sampling process, the emerging findings, and theoretical gaps perceived therein - reflects on how researcher’s biases may have affected their analysis - explains the role of literature, especially where an extensive review preceded the GT study - + + ### Extraordinary Attributes + + - triangulates with extensive quantitative data (e.g. questionnaires, sentiment analysis) - employs a team of researchers and explains their roles - + ## Quality Criteria diff --git a/Standards/QualitativeSurveys.md b/Standards/QualitativeSurveys.md index 3943028..0d15bc8 100644 --- a/Standards/QualitativeSurveys.md +++ b/Standards/QualitativeSurveys.md @@ -1,5 +1,5 @@ # Qualitative Surveys (Interview Studies) - + Research comprising semi-structured or open-ended interviews ## Application @@ -26,11 +26,16 @@ consider the **Discourse Analysis Standard**. ## Specific Attributes ### Essential Attributes + + - explains how interviewees were selected (i.e. sampling strategy; see The Sampling Supplement) - describes interviewees (e.g. demographics, work roles) - presents clear chain of evidence from interviewee quotations to proposed concepts + + +### Desirable Attributes + -### Desirable Attributes - includes highly diverse participants - uses direct quotations extensively to support key points - EITHER: evaluates an a priori theory (or model, framework, taxonomy, etc.) using deductive coding with an a priori coding scheme based on the prior theory OR: synthesizes results into a new, mature, fully-developed and clearly articulated theory (or model, etc.) using some form of inductive coding (coding scheme generated from data) @@ -40,12 +45,15 @@ consider the **Discourse Analysis Standard**. - discusses transferability; findings plausibly transferable to different contexts - compares results with (or integrates them into) prior theory or related research - reflects on how researchers’ biases may have affected their analysis + + +### Extraordinary Attributes + -### Extraordinary Attributes - employs multiple methods of data analysis (e.g. open coding vs. process coding; manual coding vs. automated sentiment analysis) with method-triangulation - employs longitudinal design (i.e. each interviewee participates multiple times) and analysis - employs probabilistic sampling strategy; statistical analysis of response bias - + ## General Quality Criteria diff --git a/Standards/QuestionnaireSurveys.md b/Standards/QuestionnaireSurveys.md index 678f1b6..a030276 100644 --- a/Standards/QuestionnaireSurveys.md +++ b/Standards/QuestionnaireSurveys.md @@ -1,4 +1,5 @@ # Questionnaire Surveys + A study in which a sample of respondents answer a series of questions, typically through a computerized or paper form and mostly structured @@ -23,7 +24,9 @@ typically given to participants in controlled experiments (see the ## Specific Attributes -### Essential Attributes +### Essential Attributes + + - identifies the target population & defines the sampling strategy (see the Sampling Supplement) - provides questionnaire instrument (e.g. as supplemental file) - EITHER: provides study artifacts; i.e., instrument(s), code books, analysis scripts and dataset(s) (addressing potential anonymity and confidentiality issues) OR: describes in detail study artifacts and justifies why they are not provided @@ -35,8 +38,11 @@ typically given to participants in controlled experiments (see the - explains handling of missing data (e.g. imputation, weighting adjustments, discarding) - acknowledges generalizability threats; discusses how respondents may differ from target population - analyzes response rates - + + ### Desirable Attributes + + - characterizes the target population including demographic information (e.g. culture, knowledge) - defines and estimates the size of the population strata (if applicable) - accounts for the principles of research ethics (e.g. informed consent, re-identification risk) @@ -47,12 +53,15 @@ typically given to participants in controlled experiments (see the - analyzes response bias (quantitatively) - discusses possible effect of incentives (e.g. on voluntariness, response rates, response bias) if used - describes the stratification of the analysis (if stratified sampling is used) -- clearly distinguishes evidence-based results from interpretations and speculation - +- clearly distinguishes evidence-based results from interpretations and speculation + + ### Extraordinary Attributes + + - provides feasibility check of the anticipated data analysis techniques - reports on the scale validation in terms of dimensionality, reliability, and validity of measures - + ## General Quality Criteria diff --git a/Standards/SystematicReviews.md b/Standards/SystematicReviews.md index 0c15576..f4375c8 100644 --- a/Standards/SystematicReviews.md +++ b/Standards/SystematicReviews.md @@ -1,5 +1,5 @@ # Systematic Reviews - + A study that appraises, analyses, and synthesizes primary or secondary literature to provide a complete, exhaustive summary of current evidence regarding one or more specific topics or research questions @@ -15,6 +15,7 @@ regarding one or more specific topics or research questions advanced qualitative synthesis methods (e.g. meta-ethnography) ## Specific Attributes + ### Essential Attributes - presents step-by-step, systematic, replicable description of search process including search terms @@ -25,8 +26,11 @@ regarding one or more specific topics or research questions - clear chain of evidence from the extracted data to the answers to the research question(s) - presents conclusions or recommendations for practitioners/non-specialists - identifies method (e.g. systematic review, meta-analysis, mapping study, narrative synthesis, etc.) + + +### Desirable Attributes + -### Desirable Attributes - provides replication package including protocol, search terms, search results, selection process results; complete dataset, analysis scripts; examples of coding, decision rules or edge cases - mitigates sampling bias and publication bias, using some combination of: (i) manual and keyword automated searches; (ii) backward and forward snowballing searches; (iii) checking profiles of prolific authors in the area; (iv) searching both formal databases (e.g. ACM Digital Library) and indexes (e.g. Google Scholar); (v) searching for relevant dissertations; (vi) searching pre-print servers (e.g. arXiv); (iiv) soliciting unpublished manuscripts through appropriate listservs or social media; (iiiv) contacting known authors in the area. - demonstrates that the search process is sufficiently rigorous for the systematic review goals @@ -39,11 +43,14 @@ regarding one or more specific topics or research questions - uses meta-analysis methods appropriate for primary studies; does not use vote counting - integrates results into prior theory or research; identifies gaps, biases, or future directions - presents results as practical, evidence-based guidelines for practitioners, researchers, or educators + + +### Extraordinary Attributes + -### Extraordinary Attributes - two or more researchers independently undertaking the preliminary search process before finalizing the search scope and search keywords - contacted primary study authors to ensure interpretations are correct, and elicit additional details not found in the papers such as access to raw data - + ## Examples of Acceptable Deviations diff --git a/form_generator/index.html b/form_generator/index.html index 757963c..e5cd28c 100644 --- a/form_generator/index.html +++ b/form_generator/index.html @@ -3,11 +3,16 @@ Empirical Standards
-

Select all that applies:

- Systematic Reviews
- Action Research
- Grounded Theory
- Mixed Methods

+

Select all that apply:

+ Systematic Reviews

+ Action Research

+ Grounded Theory

+ Qualitative Surveys

+ Case Study

+ Questionnaire Surveys

+ Experiments

+ Engineering Research

+ None of the above