Skip to content

Commit

Permalink
Added tags to markdown files
Browse files Browse the repository at this point in the history
  • Loading branch information
arhamarshad1 committed Jan 21, 2021
1 parent 8a1835b commit 8f0cfad
Show file tree
Hide file tree
Showing 14 changed files with 150 additions and 37 deletions.
5 changes: 5 additions & 0 deletions .idea/.gitignore

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 12 additions & 0 deletions .idea/EmpiricalStandards.iml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 8 additions & 0 deletions .idea/modules.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions .idea/vcs.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 11 additions & 3 deletions Standards/ActionResearch.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Action Research
<standard name="Action Research">

Empirical research that investigates how an intervention, like the
introduction of a method or tool, affects a real-life context
Expand Down Expand Up @@ -28,6 +29,7 @@ standards.

## Specific Attributes
### Essential Attributes
<checklist name="Essential">

- describes the context or site of the intervention(s)

Expand All @@ -52,20 +54,26 @@ standards.
- reports lessons learned by the organization

- researchers reflect on their own possible biases

</checklist>

### Desirable Attributes
<checklist name="Desirable">

- uses direct quotations extensively

- uses member checking to assess resonance

- findings plausibly transferable to other contexts

- triangulation across quantitative and qualitative data

</checklist>

### Extraordinary Attributes
<checklist name="Extraordinary">

- research team with triangulation across researchers (to mitigate researcher bias)

</checklist>
## General Quality Criteria

Example criteria include reflexivity, credibility, resonance, usefulness
Expand Down
16 changes: 13 additions & 3 deletions Standards/CaseStudy.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Case Study and Ethnography
<standard name="Case Study and Ethnography">

"An empirical inquiry that investigates a contemporary phenomenon (the
"case") in depth and within its real-world context, especially when the
Expand Down Expand Up @@ -35,6 +36,8 @@ captures a large quantitative dataset with limited context, consider the
## Specific Attributes

### Essential Attributes
<checklist name="Essential">

- explains why the case study approach is appropriate for the research question

- justifies the selection of the case or site that was studied
Expand All @@ -46,8 +49,11 @@ captures a large quantitative dataset with limited context, consider the
- presents a clear and well-argued "chain of evidence" from observations to findings

- clearly answers the research question(s)

</checklist

### Desirable Attributes
<checklist name="Desirable">

- reports the type of case study (see *Types of Case Studies*, below)

- describes external events and other factors that may have affected the case or site
Expand All @@ -57,16 +63,20 @@ captures a large quantitative dataset with limited context, consider the
- cross-checks interviewee statements (e.g. against direct observation or archival records)

- uses quotations to *illustrate* findings (note: quotations should not be *the only* representation of a finding; each finding should be described independently of supporting quotations)

</checklist>

### Extraordinary Attributes
<checklist name="Extraordinary">

- multiple, deep, fully-developed cases with cross-case triangulation

- uses multiple judges and reports inter-rater reliability (cf. Gwet & Gwet 2002)

- uses direct observation and clearly integrates direct observations into results

- created a case study protocol beforehand and makes it publicly accessible

</checklist>

## General Quality Criteria

Case studies should be evaluated using qualitative validity criteria
Expand Down
15 changes: 12 additions & 3 deletions Standards/EngineeringResearch.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Engineering Methods
<standard name="Engineering Methods">

## Engineering Research (AKA Design Science)

Expand All @@ -20,6 +21,8 @@ standard is not appropriate for:
## Specific Attributes

### Essential Attributes
<checklist name="Essential">

- describes the proposed artifact in adequate detail^1^

- justifies the need for, usefulness of, or relevance of the proposed artifact^2^
Expand All @@ -38,8 +41,11 @@ standard is not appropriate for:
- assumptions (if any) are explicit; do not contradict each other or the contribution's goals; plausibly hold for the evaluation subjects

- uses notation consistently (if any notation is used)

</checklist>

### Desirable Attributes
<checklist name="Desirable">

- reviews the theoretical basis of the artifact

- provides correctness arguments of the key analytical and theoretical contributions (e.g. theorems, complexity analyses, mathematical proofs)
Expand All @@ -51,12 +57,15 @@ standard is not appropriate for:
- provides a replication package including datasets and analytical scripts and EITHER a comprehensive description of the artifact OR source code if artifact is virtual

- justifies any items missing from replication package based on practical or ethical grounds.

</checklist>

### Extraordinary Attributes
<checklist name="Extraordinary">
- contributes to our collective understanding of design practices or principles

- presents ground-breaking innovations with obvious real-world benefits

</checklist>

## General Quality Criteria

- Comprehensiveness of proposed artifact description
Expand Down
17 changes: 13 additions & 4 deletions Standards/Experiments.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Experiments (with Human Participants)
<standard name="Experiments (with Human Participants)">

A study in which an intervention is deliberately introduced to observe
its effects on some aspects of reality under controlled conditions
Expand All @@ -24,7 +25,9 @@ Science Standard** or the **Engineering Research Standard**.

## Specific Attributes

### Essential Attributes
### Essential Attributes
<checklist name="Essential">

- describes how characteristics of phenomenon under investigation relate to experimental constructs
- states formal hypotheses
- justifies use of one-sided hypotheses (if any) based on face validity or previous work
Expand All @@ -42,8 +45,11 @@ Science Standard** or the **Engineering Research Standard**.
- EITHER: shares raw, de-identified data OR: explains why sharing raw data is impractical or unethical
- discusses construct, conclusion internal and external validity
- discusses alternative interpretations of results
</checklist>
### Desirable Attributes
<checklist name="Desirable">

### Desirable Attributes
- justifies hypotheses and Bayesian priors (if applicable) based on previous studies and theory
- discusses alternative experimental designs and why they were not used (e.g. validity trade-offs)
- includes visualizations of data distributions
Expand All @@ -55,12 +61,15 @@ Science Standard** or the **Engineering Research Standard**.
- analyzes construct validity of dependent variable
- uses and reports manipulation checks
- pre-registration of hypotheses and design where venue allows
</checklist>
### Extraordinary Attributes
<checklist name="Extraordinary">

### Extraordinary Attributes
- reports multiple experiments or replications in different cultures or regions
- uses multiple methods of data collection; data triangulation
- longitudinal data collection with appropriate time-series analysis

<checklist>

## General Quality Criteria

Expand Down
15 changes: 12 additions & 3 deletions Standards/GeneralStandard.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# The General Standard
<standard name = "General Standard">

## Application

Expand Down Expand Up @@ -44,6 +45,8 @@ following.
## Specific Attributes

### Essential Attributes
<checklist name="Essential">

- states a purpose, problem, objective, or research question
- methodology is appropriate (not necessarily optimal) for stated purpose or questions
- describes in detail what, where, when and how data were collected
Expand All @@ -57,8 +60,11 @@ following.
- language is not misleading; any grammatical problems do not substantially hinder understanding
- visualizations/graphs are not misleading (see the Information Visualization Supplement)
- complies with all applicable empirical standards
</checklist>
### Desirable Attributes
<checklist name="Desirable">

### Desirable Attributes
- summarizes and synthesizes a reasonable selection of related work
- clearly describes relationship between contribution(s) and related work
- states epistemological stance (e.g. post-positivism, interpretivism, critical realism)
Expand All @@ -71,12 +77,15 @@ following.
- visualizations (e.g. graphs, diagrams, tables) advance the paper’s arguments or contribution
- clarifies the roles and responsibilities of the researchers (i.e. who did what?)
- provides an auto-reflection or assessment of the authors’ own work (e.g. lessons learned)

</checklist>
### Extraordinary Attributes
<checklist name="Extraordinary">

- applies two or more data collection or analysis strategies to the same research question (see Multimethodology Supplement)
- approaches the same research question(s) from multiple epistemological perspectives
- innovates on research methodology while completing an empirical study

</checklist>

## General Quality Criteria

Expand Down
14 changes: 11 additions & 3 deletions Standards/GroundedTheory.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Grounded Theory
<standard name="Grounded Theory">

A study of a specific area of interest or phenomenon that involves
iterative and interleaved rounds of qualitative data collection and
Expand All @@ -23,6 +24,7 @@ collection and analysis or do not use theoretical sampling, consider the
**Case Study Standard** or the **Qualitative Survey Standard**.

## Specific Attributes
<checklist name="Essential">

### Essential Attributes
- identifies the version of Grounded Theory used/adapted (Glaser, Strauss-Corbin, Charmaz, etc.)
Expand All @@ -31,8 +33,11 @@ collection and analysis or do not use theoretical sampling, consider the
- provides evidence of saturation; explains how saturation was achieved
- explains how key patterns (e.g. categories) emerged from GT steps (e.g. selective coding)
- provides clear chain of evidence from raw data (e.g. interviewee quotations) to derived codes, concepts, and categories

</checklist>
### Desirable Attributes
<checklist name="Desirable">

- explains how and why study adapts or deviates from claimed GT version
- presents a mature, fully-developed theory or taxonomy
- includes highly diverse participants and/or data sources (e.g. software repositories, forums)
Expand All @@ -45,11 +50,14 @@ collection and analysis or do not use theoretical sampling, consider the
- explains theoretical sampling vis-à-vis the interplay between the sampling process, the emerging findings, and theoretical gaps perceived therein
- reflects on how researcher’s biases may have affected their analysis
- explains the role of literature, especially where an extensive review preceded the GT study

</checklist>
### Extraordinary Attributes
<checklist name="Extraordinary">

- triangulates with extensive quantitative data (e.g. questionnaires, sentiment analysis)
- employs a team of researchers and explains their roles

</checklist>

## Quality Criteria

Expand Down
16 changes: 12 additions & 4 deletions Standards/QualitativeSurveys.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Qualitative Surveys (Interview Studies)

<standard name="Qualitative Surveys (Interview Studies)">
Research comprising semi-structured or open-ended interviews

## Application
Expand All @@ -26,11 +26,16 @@ consider the **Discourse Analysis Standard**.
## Specific Attributes

### Essential Attributes
<checklist name="Essential">

- explains how interviewees were selected (i.e. sampling strategy; see The Sampling Supplement)
- describes interviewees (e.g. demographics, work roles)
- presents clear chain of evidence from interviewee quotations to proposed concepts
</checklist>
### Desirable Attributes
<checklist name="Desirable">

### Desirable Attributes
- includes highly diverse participants
- uses direct quotations extensively to support key points
- EITHER: evaluates an a priori theory (or model, framework, taxonomy, etc.) using deductive coding with an a priori coding scheme based on the prior theory OR: synthesizes results into a new, mature, fully-developed and clearly articulated theory (or model, etc.) using some form of inductive coding (coding scheme generated from data)
Expand All @@ -40,12 +45,15 @@ consider the **Discourse Analysis Standard**.
- discusses transferability; findings plausibly transferable to different contexts
- compares results with (or integrates them into) prior theory or related research
- reflects on how researchers’ biases may have affected their analysis
</checklist>
### Extraordinary Attributes
<checklist name="Extraordinary">

### Extraordinary Attributes
- employs multiple methods of data analysis (e.g. open coding vs. process coding; manual coding vs. automated sentiment analysis) with method-triangulation
- employs longitudinal design (i.e. each interviewee participates multiple times) and analysis
- employs probabilistic sampling strategy; statistical analysis of response bias

</checklist>

## General Quality Criteria

Expand Down
Loading

0 comments on commit 8f0cfad

Please sign in to comment.