Data Types Pros and Cons
Pros and Cons of different data types
Program-Generated Data
Program data is efficient and great for tracking over time, but I’ve run into issues like missing fields or inconsistent formats. It’s a solid starting point, but I always plan to fill in the gaps with extra data collection.
• Can provide systematic information on services delivered, staff supports, or clients served, eliminating the need for original data collection.
• Often longitudinal, allowing for understanding of a program’s operation and implementation over time.
Administrative Records Data
Administrative records are a goldmine—standardized, long-term data that saves time and works well with other sources. But access delays, linking difficulties, and missing context can limit their usefulness for deeper analysis.
• Data are comprehensive, cover long periods of time, and are standardized across places or settings, eliminating the need for original data collection and allowing for analysis of longer-term outcomes.
• Can be combined with other data sources to create a richer understanding.
• May contain relevant data for both program and comparison groups.
Secondary Data
Secondary data is convenient and cost-effective, offering broad comparisons across contexts—but it rarely fits your exact needs. Timing, relevance, and detail can be off, and it often lacks the specificity or scale needed for focused projects.
• May contain information on elements not available in program-generated or administrative data (e.g., background sociodemographic information for the target population).
• Useful for comparative purposes, because the same survey instrument and measures are used across contexts (e.g., the Demographic and Health Survey contains data on the same indicators for over 90 low- and middle-income countries).
• Data can be easily accessed, often through online databases.
• Cost-effective.
Survey Data
Survey data is great because you control exactly what you ask, target the right population, and can compare results across studies. But it’s time-consuming, costly, and full of logistical hurdles—from design to data quality issues.
• Give researchers control over what information is collected (including level of detail) to ensure that data directly map onto the research questions.
• When using an existing validated or standardized measure, can compare results to those of other studies and populations.
• Collect consistent and systematic data from a specific population of direct interest (e.g., service providers, program participants).
• Can identify and survey specific individuals from the program and comparison groups, rather than creating nonrandom comparison groups from secondary or administrative data sources (which may be prone to bias).
Assessment
Assessments offer precise, standardized data that's often more accurate than self-reports and easy to compare across studies. But they can be expensive, require trained staff, and place a high burden on participants—making timing and consistency real challenges.
• Capture constructs with rigor and precision difficult to capture via self-report or proxies (e.g., biomarker data based on urine samples to measure nicotine in a body's system, compared to self-reported behaviors of cigarette smoking, which are subject to imprecision and bias).
• Can gather the same information from a large sample.
• May be quicker to analyze than qualitative assessments, particularly if data are entered electronically while the assessment is administered.
• Use of validated assessment tools (e.g., the Woodcock Johnson Tests of Achievement) allows for comparing results to other studies.
In-Depth Interview Methods
In-depth interviews offer rich, nuanced insights you can’t get from surveys, with the flexibility to explore unexpected topics. But they’re time-consuming, rely heavily on interviewer skill, and balancing structure with spontaneity can be tricky.
• Gather a large amount of detailed information from respondents.
• Allow for flexibility while collecting data (e.g., probing on a particular issue or topic in fuller depth when needed) and allow new topics to be brought up by respondents and included in analysis—topics that would not emerge if using a closed-ended survey.
• Interviewers can probe on meaning and inconsistencies that arise in an interview, which provides nuance to understanding a respondent's reasoning and behavior.
• Can probe on more sensitive topics a respondent may not answer truthfully in front of others, such as in a focus group setting.
Focus Group
Focus groups are a cost-effective way to gather diverse perspectives and spark dynamic discussions, especially for exploring group consensus or disagreement. But they offer less depth than interviews, depend heavily on a skilled moderator, and can be skewed by group dynamics or scheduling challenges.
• Cost-effective and efficient.
• Provide a breadth of perspectives and opinions.
• Respondents can "play off each other" in useful and illustrative ways, which takes advantage of the group-based dynamic in understanding an issue.
• Group-based dynamic also allows researchers to understand whether there is group consensus on a specific topic and explore when and why there might be a lack of a consensus.
Ethnographic Research Methods
Ethnographic research offers deep, immersive insight by capturing real-life experiences in context—but it’s time-intensive, costly, and demands careful attention to bias, influence, and overwhelming amounts of data. The payoff? Unmatched depth and understanding.
• Gather very rich information about a setting and participants over a period of time, across many different topics and in natural environments.
• Provide thick description (more so than in-depth interviews) about a program from participants' viewpoints, as the researcher is actually partaking in the program while collecting data.
• Allow for an understanding of how social interactions and relationships affect an individual's decisions and behaviors.
• Can shift focus or research questions to be responsive to unexpected ideas or events that emerge on the ground.
Direct Observation Methods
Direct observation lets you capture real-time behaviors and interactions—what’s actually happening, not just what’s reported. It’s flexible and rich, but time-consuming, requires careful protocols, and can be influenced by your presence. Still, it offers insights no other method can.
• Researchers can record all activities they see and document the perceived quality of those activities, giving a detailed description of settings and interactions.
• Observers gather data using prespecified fields but also have flexibility to record unexpected activities and to provide a finer description of events that emerge as more salient.
• Can be used in flexible ways. Data may be quantified (e.g., three out of five classrooms had take-home flyers for parents), but can also include narrative descriptions of settings or observed interactions.
Choose the right approach for your goals—each method has strengths and trade-offs, so match your strategy to the questions you need to answer.