AI News, Qualitative before Quantitative: How Qualitative Methods Support Better Data Science

Qualitative before Quantitative: How Qualitative Methods Support Better Data Science

Co-authored with Vicky Zhang Have you ever been embarrassed by the first iteration of one of your machine learning projects, where you didn’t include obvious and important features?

In the practical hustle and bustle of trying to build models, we can often forget about the observation step in the scientific method and jump straight to hypothesis testing.

These assumptions could lead to: In this post, we’ll explore how qualitative methods can help all data scientists build better models, using a case study of Indeed’s new lead routing machine learning model which ultimately generated several million dollars in revenue.

Quantitative methods are great for answering questions like “How much does the average small business spend on a job posting?”, “What are the skills that make someone a data scientist?”, or even “How many licks does it take to get to the center of a Tootsie roll pop?” (The answer is 3.

Three licks.) But there are some questions that quantitative methods can’t answer, such as “Why do account executives reach out to this lead instead of that lead?” or “How do small businesses make the decision to sponsor or not sponsor a job?” Or the truly deep question: “Why do you want to get to center of the Tootsie roll pop?” To answer these questions, qualitative researchers rely on methods like in-depth interviews, participant observation, content analysis and usability studies.

We route that employer to an account executive, who then reaches out and helps the employer set an advertising budget to sponsor their job.

With the intuition we gained from our qualitative studies on account executives’ behaviors and thought processes, we ultimately built a machine learning model that generated millions of dollars in annual incremental revenue.

Using the framework method for the analysis of qualitative data in multi-disciplinary health research

The Framework Method sits within a broad family of analysis methods often termed thematic analysis or qualitative content analysis.

These approaches identify commonalities and differences in qualitative data, before focusing on relationships between different parts of the data, thereby seeking to draw descriptive and/or explanatory conclusions clustered around themes.

While in-depth analyses of key themes can take place across the whole data set, the views of each research participant remain connected to other aspects of their account within the matrix so that the context of the individual’s views is not lost.

It is therefore useful where multiple researchers are working on a project, particularly in multi-disciplinary research teams were not all members have experience of qualitative data analysis, and for managing large data sets where obtaining a holistic, descriptive overview of the entire data set is desirable.

The Framework Method is most commonly used for the thematic analysis of semi-structured interview transcripts, which is what we focus on in this article, although it could, in principle, be adapted for other types of textual data [13], including documents, such as meeting minutes or diaries [12], or field notes from observations [10].

Although the Framework Method is a highly systematic method of categorizing and organizing what may seem like unwieldy qualitative data, it is not a panacea for problematic issues commonly associated with qualitative data analysis such as how to make analytic choices and make interpretive strategies visible and auditable.

It is therefore essential that studies using the Framework Method for analysis are overseen by an experienced qualitative researcher, though this does not preclude those new to qualitative research from contributing to the analysis as part of a wider research team.

would require a more inductive approach that allows for the unexpected, and permits more socially-located responses [25] from interviewees that may include matters of cultural beliefs, habits of food preparation, concepts of ‘fate’, or links to other important events in their lives, such as grief, which cannot be predicted by the researcher in advance (e.g.

It is not within the scope of this paper to consider study design or data collection in any depth, but before moving on to describe the Framework Method analysis process, it is worth taking a step back to consider briefly what needs to happen before analysis begins.

As any form of qualitative or quantitative analysis is not a purely technical process, but influenced by the characteristics of the researchers and their disciplinary paradigms, critical reflection throughout the research process is paramount, including in the design of the study, the construction or collection of data, and the analysis.

They cannot be too attached to certainty, but must remain flexible and adaptive throughout the research in order to generate rich and nuanced findings that embrace and explain the complexity of real social life and can be applied to complex social issues.

It is important to remember when using the Framework Method that, unlike quantitative research where data collection and data analysis are strictly sequential and mutually exclusive stages of the research process, in qualitative analysis there is, to a greater or lesser extent depending on the project, ongoing interplay between data collection, analysis, and theory development.

Mixed Methods: Integrating Quantitative and Qualitative Data Collection and Analysis While Studying Patient-Centered Medical Home Models

This brief focuses on using mixed methods to evaluate patient-centered medical home (PCMH) models.

The term “mixed methods” refers to an emergent methodology of research that advances the systematic integration, or “mixing,” of quantitative and qualitative data within a single investigation or sustained program of inquiry.

The evaluation of PCMHs provide an ideal opportunity for mixed methods studies to contribute to learning about best practices in how to implement a PCMH as well as PCMH effectiveness in achieving the triple aim outcomes of cost, quality, and patient experience of care.

Mixed methods research originated in the social sciences and has recently expanded into the health and medical sciences including fields such as nursing, family medicine, social work, mental health, pharmacy, allied health, and others.

These procedures include advancing rigor, offering alternative mixed methods designs, specifying a shorthand notation system for describing the designs to increase communication across fields, visualizing procedures through diagrams, noting research questions that can particularly benefit from integration, and developing rationales for conducting various forms of mixed methods studies.

The core characteristics of a well-designed mixed methods study in PCMH research include the following: This brief focuses on the potential uses of this methodology for PCMH research as well as on specific mixed methods designs in primary care research (Creswell, Fetters, and Ivankova, 2004) that offer feasible, information-rich data that can enhance traditional quantitative research approaches.

This explanatory sequential design typically involves two phases: (1) an initial quantitative instrument phase, followed by (2) a qualitative data collection phase, in which the qualitative phase builds directly on the results from the quantitative phase.

For example, findings from instrument data about costs can be explored further with qualitative focus groups to better understand how the personal experiences of individuals match up to the instrument results.

This exploratory sequential design involves first collecting qualitative exploratory data, analyzing the information, and using the findings to develop a psychometric instrument well adapted to the sample under study.

For example, sports stories frequently integrate quantitative data (scores or number of errors) with qualitative data (descriptions and images of highlights) to provide a more complete story than either method would alone.

Mixed methods studies are complex to plan and conduct.They require careful planning to describe all aspects of research, including the study sample for qualitative and quantitative portions (identical, embedded, or parallel);

Finding qualitative experts who are also comfortable discussing quantitative analyses and vice versa can be challenging in many environments.Given that each method must adhere to its own standards for rigor, ensuring appropriate quality of each component of a mixed methods study can be difficult (Wisdom, Cavaleri, Onwuegbuzie, et al., 2011).

For example, quantitative analyses require much larger sample sizes to obtain statistical significance than do qualitative analyses, which require meeting goals of saturation (not uncovering new information from conducting more interviews) and relevance.

By carefully selecting the mixed method design that best suits the evaluation’s questions and meets its resource constraints, evaluators can facilitate deeper, more meaningful learning regarding the effectiveness and implementation of PCMH models.

Fundamentals of Qualitative Research Methods: Data Analysis (Module 5)

Qualitative research is a strategy for systematic collection, organization, and interpretation of phenomena that are difficult to measure quantitatively. Dr. Leslie ...

Fundamentals of Qualitative Research Methods: What is Qualitative Research (Module 1)

Qualitative research is a strategy for systematic collection, organization, and interpretation of phenomena that are difficult to measure quantitatively. Dr. Leslie ...

Fundamentals of Qualitative Research Methods: Developing a Qualitative Research Question (Module 2)

Qualitative research is a strategy for systematic collection, organization, and interpretation of phenomena that are difficult to measure quantitatively. Dr. Leslie ...


In this example of rapid research Dr Martin Senior summarises the concept of triangulation as discussed by Denzin. He offers 4 key “take aways” that lead to ...

Psychological Research - Crash Course Psychology #2

You can directly support Crash Course at Subscribe for as little as $0 to keep up with everything we're doing. Also, if you ..

What is action research?

Here's a short description of action research. TRANSCRIPT: Teaching is a craft. It's both an art and a science, which is why great teachers always experiment ...

The difference between Concepts Models and Theories

Get my eBook "Research terminology simplified: Paradigms, axiology, ontology, epistemology and methodology" on Amazon: OR the ..

Ontology, Epistemology, Methodology and Methods in Research Simplified!

When you are just starting to learn about research it helps to have simple definitions of Ontology, Epistemology, Methodology and Methods in Research!

Combining quantitative and qualitative evidence: why, how and when?

In part 4 of the Qualitative Research Synthesis workshop hosted by KTDRR, James Thomas, PhD, Associate Director of the Evidence for Policy and Practice ...

The six steps of quantitative research

In this video, I talk briefly about the six most important steps any researcher should follow when doing quantitative research. This video is part of the course "First ...