My SAGE Webinar on this topic is available at:

http://youtu.be/oO3cspRrq4E

Primary Sources:

Schutt, Russell K. 2015. Investigating the Social World: The Process and Practice of Research, 8^th ed. Thousand Oaks, CA: SAGE Publications.

http://www.sagepub.com/books/Book242232?siteId=sage-us&prodTypes=any&q=schutt&fs=1

Schutt, Russell K. 2011. Homelessness, Housing and Mental Illness, with Stephen M. Goldfinger. Contributions by Larry J. Seidman. Cambridge, MA: Harvard University Press.

http://www.hup.harvard.edu/catalog.php?isbn=9780674051010

The 375 Webinar participants generated 23 questions that I have answered below.  You and your students might find that some of your own questions about mixed methods are addressed in this Q&A.

 

• You used Random Selection of subjects, but in the experimental piece, what were your other controls, such as statistical designs, if any?

 Actually, we didn’t use random selection of subjects. We screened all interested guests in the three mental health shelters for safety and then randomly assigned all those who were eligible for the project to either the group or independent housing conditions.  The details are in an article on “screening for safety” in the Community Mental Health Journal (see citation below). So the primary design we used to control for extraneous influences was creation of equivalent groups at the outset of the study.  See the methodological appendix in my book, Housing, Homelessness, and Mental Illness (pp. 285-289).  In addition, in most analyses (including in the book and the many project articles cited in it), I used statistical controls to adjust for subject differences, most often multiple regression analysis. 

 Goldfinger, Stephen M., Russell K. Schutt, Winston Turner, George Tolomiczenko, and Mark Abelman. 1996. “Assessing Homeless Mentally Ill Persons for Permanent Housing: Screening for Safety.” Community Mental Health Journal, 32:275-288.

 

• How did you decide on the various methods used? E.g. pairing case studies with qualitative and quantitative studies

 The answer to this question in terms of the overall project design is that our team of investigators who designed the project was very interdisciplinary, with a survey researcher, a neuropsychologist, an analyst of secondary data, an experimentalist, and a team of ethnographers.  We brought in the ethnographers to play the vital role of investigators of social processes in the group homes as they happened.  It would have been too easy to overlook changes in the group homes or difficulties in interaction if we had relied on only infrequently collected quantitative indicators.  We included qualitative questions whenever possible to allow us to reflect on what research participants meant by their answers or simply to provide a more engaging social experience.

For the most effective presentation of the analysis, I organized the book to maximize the value of having multiple methods of investigation for most of the major research questions in which I was interested.  Each of my chapters (except the first two and the last one) is on a major topic that must be considered in order to understand the experiences and outcomes of persons living in our project’s housing.  Examples include “Satisfying Wants and Meeting Needs” (on residential preferences and clinician recommendations), “Substance Abuse,” “Mental Illness,” etc.  At the start of most of these chapters, I review relevant literature, then present a quantitative analysis using the baseline data, then a qualitative analysis of related activities throughout the project, and then conclude with a quantitative analysis of change over time.  This ensures that I can enrich findings from quantitative analyses with insights from available qualitative data.  Most of the project’s qualitative data were from ethnographic notes, but there also some qualitative comments recorded by clinicians and by project case managers, as well as some qualitative data from semi-structured interviews my RAs conducted with staff in the shelters we worked with.  For one topic, “Empowerment,” we collected only qualitative data.  So my basic approach in the analysis was to mix methods whenever I had data collected about a topic using both qualitative and quantitative techniques.  The case studies emerged as I identified basic patterns in the ethnographic data and then searched for individuals who represented those patterns. 

 

• What are the challenges of using mixed methods?

There are many that researchers should consider, even while we recognize the benefits of mixed methods.  I have included a lengthy section on Strengths and Limitations of Mixed Methods in the mixed methods chapter of the 8^th edition of Investigating the Social World and I discuss in that section each of the major challenges.  Here are the highlights: (1) One of the appeals of mixed methods is that it can help us “triangulate” our measurement strategy–in effect, use different measures of the same concept to provide a more robust overall measure.  But what if the measures do not yield consistent results?  Is it due to the inadequacy of one of the measures or to another influence that prevents convergence of the alternative measures?  In some cases, qualitative methods can provide information that helps us to answer this question.  (See my example of the non-disclosure of substance abuse in the “strengths and limitations” section or in the methodological appendix to Homelessness, Housing, and Mental Illness, or the Journal of Nervous and Mental Disease article cited below).  In other cases, qualitative assessment can reveal the problem and we will need other data to figure out why it has occurred.  Another challenge arises when a qualitative comparison of quantitative relationships at different sites identifies marked differences, but we lack the detailed information necessary to explain the differences.  My discussion of cross-site differences in the multi-site NIMH-funded study of housing and homelessness is an example of how a careful review can generate reasonable explanations, but confirmation must await additional research.  In addition to the two books, you can also read about this comparison in an article available online (citation below).  Of course, a major incentive for using mixed methods is to uncover unexpected patterns and generate new research questions, so in that sense the real challenge is to keep garnering resources to allow us to refine our knowledge of social processes. 

Another major challenge for users of mixed methods is the different types of expertise required.  Most social researchers can manage adding some qualitative questions to a primarily quantitative survey, or collecting some quantitative indicators in a qualitative project and then analyzing the resulting mixed methods data to good effect.  But the time and training required to develop advanced expertise in ethnography, in in-depth interviewing, in survey research, statistical analysis, or any advanced method mean that most researchers are going to specialize.  So developing a strong mixed methods project will often require collaboration of different researchers with different types of expertise.  This requires more time and more attention to project design and management than may be necessary in a single method project.  Researchers with different methodological commitments may also have different research philosophies that make collaboration all the more challenging (see chapter 1 of Investigating).  Devoting time in advance to figuring out a management structure and team member responsibilities as well as to developing appreciation for different perspectives can help to overcome this challenge.

Goldfinger, Stephen M., Russell K. Schutt, Larry J. Seidman, Winston M. Turner, Walter E. Penk and George S. Tolomiczenko. 1996. “Self-report and Observer Measures of Substance Abuse Among Homeless Mentally Ill Persons, in the Cross Section and Over Time.” Journal of Nervous and Mental Disease, 184:667-672.

Schutt, Russell K., Richard L. Hough, Stephen M. Goldfinger, Anthony F. Lehman, David L. Shern, Elie S. Valencia, Patricia A. Wood. 2009. “Lessening Homelessness among Persons with Mental Illness: A Comparison of Five Randomized Treatment Trials.” Asian Journal of Psychiatry 2:100-105. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2788308/pdf/nihms147743.pdf

  

• Is it appropriate to quantitatively analyze qualitative data?

It depends.  It depends on the way in which the qualitative data were collected and on the type of research question posed.  Individual qualitative researchers may have very different styles of interviewing or of conducting an ethnographic investigation, so if we hope to combine qualitative data collected by different researchers we need to ensure consistent training and design and manage data collection to maintain consistency.  For example, at one point in the work that led up to Homelessness, Housing, and Mental Illness, I had hoped to construct social network measures based on ethnographic data collected by three different ethnographers in each of the six group homes we studied.  However, while each ethnographer had collected very rich data in the two homes they observed for about 18 months, their approaches varied so much that it was not possible to construct common measures.  A research question focused on the meaning participants attach to their experiences or the process by which a change occurred in an organization may not require consideration of quantitative issues, but a research question that concerns the different meanings given by different types of respondents or the number of steps involved in a causal mechanism encourages some degree of quantification.  Computer-aided qualitative data analysis software can encourage us to consider quantitative analysis; by facilitating systematic coding of qualitative data in set categories and making it easy to compare counts of codes used within and across cases, the CAQDAS approaches helps to break down the barriers between analysis approaches.  Of course whether we think it appropriate to “mix” analyses in this way also depends on our research philosophy; those who adopt a strict constructivist philosophy may not accept the legitimacy of quantification in any form.  There is more relevant discussion in the qualitative data analysis chapter (11) of Investigating.

 

• How do you combine the conclusions from quantitative method and qualitative method?

There are many different approaches to this combination and their value depends on the specific type of mixed methods design used and the research question asked (see Exhibit 15.2 in Investigating, 8^th edition).  I emphasized in my webinar the importance of the research question, so I will first review the research question in the NIMH study I focused on in the webinar and demonstrate how it shaped the way in which I combined conclusions from the mixed methods in that study. The following excerpt comes from the concluding chapter, Community Process in Context (Schutt 2011:255-256).

What remains, in this chapter, is to take account of these diverse experiences and the different dimensions of the analysis in order to reach a conclusion about the book’s central questions: Were our project’s participants able to become members of the larger communities into which they moved? Did they build communities within the group homes that we designed? Did they cross the boundary, successfully and permanently, between homelessness and housing?

That there is neither a single nor a simple answer to these questions makes an answer no less imperative. In fact it is not until we recognize the complexities that we can reasonably begin to formulate answers. Communities are complicated and changing social entities with many interdependent parts. It is as shortsighted to term communities good or bad for their participants as it is to believe that it is easy to create a community of previously unrelated individuals. Moreover because individuals are as complex as the social world in which they interact, it makes no sense to expect a particular social situation to have a similar effect on all of its participants.

Almost twenty years after he moved into a Boston McKinney Project group home, Wayne Thomas recalled the experience fondly in a follow-up interview:

It went well. We’d all take turns doing shopping and chores. People did their share—cleaning, cooking, dishes. There were no problems getting them done.

So the overall research question, about achieving community integration, led to more research questions that required different types of methods. As indicated in the first quote, above, I present some qualitative data that capture one perspective on one question, followed by quotes that capture other perspectives and then quantitative data that summarize related quantitative findings.  I go on to review the different findings about each aspect of the overarching research question and to discuss their separate and combined import.  As I say later, in the methodological appendix, “The social world that we investigated…was complex and the individuals within it even more so. …If we overlook this complexity, we consign ourselves to one-sided perspectives and simplistic conclusions.” (Schutt 2011:305)

On the other hand, taking account in our conclusions of their specific methodological sources may not be necessary if the research question is simpler and the data more limited.  You can see an example of conclusions that are developed from a mixed methods but that make no mention of this in a report that I wrote for a National Cancer Institute-funded study about community health workers and cancer clinical trials: http://www.faculty.umb.edu/russell_schutt/chw%20cct%20needs%20assessment%20report.htm#_Toc186208388.  You will see in the report extensive use of quotes from semi-structured interviews, but also tables and graphs that summarize quantitative data collected in these same interviews.  Yet in the conclusions you’ll see that I simply discuss what we learned about the research question and do not distinguish the different methods we used.

Most common, however, is the use of qualitative data to clarify or suggest a process or complexity that the quantitative findings identified.  The next two paragraphs show how I did this in the conclusions to one of the articles from this same research on community health workers (Schutt et al. 2010:421) (emphases added):

Contrary to our hypotheses, CHWs’ orientations to medical research were influenced neither by perceived bias in the health care system nor by knowledge about cancer clinical trials. This suggests that increasing support for medical research among these critical participants in the health care system should be addressed directly, rather than by relying on courses to improve knowledge or programs to reduce perceptions of bias. Our interviewees and focus group participants emphasized repeatedly the importance they attached to personal contact from researchers and to being acknowledged as representatives of their communities.

The negative association of support for medical research with seniority, independent of age, is troubling. Based on comments made in the qualitative aspect of this research, we believe that this association may reflect growing distrust resulting from more experience over time. Many participants in focus groups and intensive interviews were frustrated with being left out of medical decision-making and with not being included as full members of the health care team. Experiences like these in a research-oriented health care system may lead over time to less support for medical research.

Schapira, Lidia and Russell Schutt. 2011. “Training Community Health Workers about Cancer Clinical Trials.” Journal of Immigrant and Minority Health 13(5):891-8.

Schutt, Russell K., Lidia Schapira, Jennifer Maniatis, Jessica Santiccioli, Silas Henlon, JudyAnn Bigby. 2010. “Community Health Workers’ Support for Cancer Clinical Trials: Description and Explanation.” Journal of Community Health 35:417-422.

 

Another option is to use a diagram or other summary model to represent conclusions about the associations you have identified.  There is an example from the research by Brown and his colleagues (2013:341-343) in Schutt 2015:550-551.

 

• Are top tier journals becoming more open to publishing mixed methods research?

 

It depends on the discipline and the specific journal, but in general the answer is yes.  I delayed writing a mixed methods chapter for Investigating the Social World until the 8^th edition (published late in 2014), because the primary markets for that text are sociology and political science and communications, and I did not feel that at least in the first of those two disciplines there was enough interest in mixed methods until recently.  But now there is.  On the other hand, my coauthors and I added mixed methods chapters to earlier editions of the versions of Investigating for the disciplines of social work, education, and criminology because mixed methods were popular in those disciplines earlier.  Paul Nestor and I have still not added a mixed methods chapter to Research Methods in Psychology, because it does not seem to be of interest to reviewers of our text. I cite in ISW8 (p. 542) a study by Alise and Teddlie that finds that mixed methods articles are more common in applied fields than in academic disciplines.  Consistent with this conclusion, I find that in my own discipline of sociology most articles appearing in the top journals are based on a single method.  I believe that reflects most importantly that most researchers continue to identify as either quantitative or qualitative in their approach and conduct research in only one tradition, but I suspect that the emphasis in top tier journals on the highest quality research designs and data analyses tends to favor research produced by the most advanced specialists of a given type.  So publication chances for a mixed methods project in top tier journals, at least in sociology, may be maximized if different articles from the project are developed that draw primarily on one method.  A book may be the best outlet for the whole corpus of findings using multiple methods (as with Homelessness, Housing, and Mental Illness), and you may also consider a specifically mixed methods journal like SAGE’s Journal of Mixed Methods Research: http://mmr.sagepub.com/.  But the times are changing.

 

• When we use a qualitative research and a quantitative research, can we say it is mixed methods? Although they are not categorized in any type of mixed methods that text explained.

 

There are no sharp boundaries between the different types of mixed methods or between single method and those mixed method designs.  When we scratch the surface of so many quantitative and qualitative designs, we so often find elements of the other methodological approach.  So one reason to study mixed methods as a separate topic is to better understand why it can be so important to add some elements of one approach to research using another approach, such as cognitive (qualitative) interviewing when developing survey questions or counts of specific behaviors when observing groups.  All of the types of research designs represented in Exhibit 15.2 of Investigating (8^th edition) are true mixed methods designs, but those involving an “embedded” approach may be better understood as single method designs if the “embedded” component is minor and intended only to enhance the value of the primary method.  For example, including a few open-ended questions or a follow-up probe in a structured survey does not make it a mixed methods design.  Counting the number of individuals on street corners at different times during an ethnographic neighborhood study does not make it a mixed methods design. 

On the other hand, in the type of mixed methods design I label as a “research program” (Schutt 2015:Exhibit 15.2), different researchers may have conducted different investigations of the same research question that build on each other but that use different, single methods.  It is helpful to think of such a research program in terms of the issues that arise in a mixed methods design, but of course the specific research studies are only using single method designs.

 

• When we use a survey with rating scales (QUAN) and open-ended questions (QUAL), is it sufficient to call it a mixed methods (only with open-ended questions)?

As I mentioned in response to the preceding question, a survey that has primarily fixed choice questions (whether with or without rating scales) and just a few open-ended questions is not reasonably termed one that uses a mixed methods design.  However, two caveats are important: (1) It is valuable to consider issues and possibilities that arise in mixed methods designs when planning, collecting, and analyzing data from such a survey; (2) The more open-ended questions are included in such a semi-structured survey, the more that it begins to approximate a mixed methods study.  For example, the survey instrument that appears at the end of this report would qualify as a mixed methods survey due to the many open-ended questions (apologies for the poor formatting in the online document): http://www.faculty.umb.edu/russell_schutt/Building%20the%20Future.htm . As I mentioned previously, there are no sharp boundaries between research designs that can reasonably be termed mixed methods and those that are best thought of as single method designs.

• Can you repeat the name of the book where you mention the software you used to analyze qualitative data?

 

It was a SAGE book, of course, coauthored by my colleague, Eben Weitzman.  I see that it has not recently been updated, but you can order it at: http://www.sagepub.in/books/Book4778. 

• Could you elaborate more into the integration model of mixed method vs. Quant to Qual or Qual to Quant ? if not the same thing, what makes it different?

My table on p. 545 of Investigating, 8^th ed. (Exhibit 15.2, Types of Mixed Methods) distinguishes the “integrated” type of mixed method design as involving equal priority to and concurrent use of qualitative and quantitative methods, while I distinguish the “staged” type of mixed methods design as involving sequential use of Qual to QUAN and QUAN to qual where one of those methods is given priority, and the “research program” type of mixed methods design as involving sequential use of QUAN and QUAL methods but giving them equal priority.  Using a concurrent QUAL+QUAN integrated design requires planning in advance how to mesh the two approaches within one project and then balancing both types of methods (and researchers) at the same time.  You may learn after the fact about how data collected in one mode contributes or alters insights developed using the other mode.  When you use either of the sequential approaches, you can learn from the first effort (qual or quan) how to best design the second effort so as to address unresolved issues and/or to take into account issues that have already been resolved in the first effort.  Deciding whether to give priority to one method or another involves a different set of issues (see my answer to question 23, below).

• Is there a particular researcher (Creswell, Turner & Onwuegbuzie, Teddlie and Tashakkori, etc) that you tend to use when deciding on a design? Or a particular book you would recommend?

These are all very expert mixed methodologists and we can all learn much from them when making design decisions.  I refer frequently to books by Creswell and by Teddlie in ISW8, as well as to David Morgan’s 2014 SAGE book on integrating qualitative and quantitative methods. 

• What are the best programs to use for analyzing qualitative data? Is there any software that can analyze both qualitative and quantitative data?

 

There are several good computer programs for analyzing qualitative data and increasingly the best programs include some quantitative analysis options.  I have primarily used NVivo, by QSR, in my own research, but I also illustrate ATLAS.ti and HyperRESEARCH in Investigating, 8^th edition.  I don’t feel that I’ve been able to keep up with all the different excellent choices now available in qualitative data analysis software, so I don’t feel I can provide more specific advice.

• What is the best strategy to approach requesting funding for a mixed methods project?

Often, the best approach is to focus on complexities in conceptualization and/or causal linkages.  Reviewers used primarily to quantitative designs can readily understand the importance of probing qualitative questions to better understand how meanings and the need to investigate the mechanism by which a treatment or experience influences an outcome.  It is also important to ground the mixed methods logic in a very firm understanding of the research literature about the planned research question, so that you can make a clear argument about how the mixed methods elements are necessary to overcome prior limitations, puzzling findings, or contradictory studies.

 

• Does the use of MM mean that one or the other of the 2 approaches is less ‘rigorous?’  For example, the ethnographic method usually demands long term site work and immersion in the study site.  To what extent is that even possible?

First of all, to be honest, we have to recognize that single-method studies vary in their rigor, so I’m sure we can all agree that some MM studies will be more and some less rigorous than some SM studies.  But this is a good question, because there are only so many hours in the day, so many dollars in the bank, and so many years of possible training.  It is certainly possible that a MM study may require resources be spread across two methods in a way that lessens the rigor that could be achieved when using just one method.  But tradeoffs occur in so many ways and for so many different reasons in research that we shouldn’t let that possibility deter us from MM designs.  In fact, I think most single method designs are made more rigorous, not less so, when some use of the “other” method is at least embedded in the design.  This is perhaps most evident when we include questions reflecting both approaches (closed-ended and open-ended) in semi-structured surveys or quantitative counts or ratings in observational studies.  Historical and comparative methods also provide many good examples of designs made more rigorous with the use of both qualitative and quantitative methods (see chapter 13 in ISW8).  Nonetheless, it IS very difficult to combine intensive ethnographic methods and quantitative elements, and it may not necessarily even be desirable to do so.  Imagine how disruptive it would be for a researcher who is immersed in a community to administer a structured survey to residents.  It may also not be feasible to have a meaningful qualitative element in a large sample survey, although innovations in web-based surveying and computer-assisted survey approaches are making such combination more possible.  Research projects that are so ambitious as our NIMH-funded study of housing alternatives as to include a major ethnographic component with a quantitative experimental design require substantial funding, considerable time, and a carefully planned group of collaborators representing the different necessary forms of expertise. 

 

• When looking at survey data and qualitative research, how do we rectify contradictions in participants’ answers? Which answers are more reliable?

I discussed this issue in my answer to Question 3, above, so I’d read that answer first.  Another way to answer the question, although it is somewhat simplistic, is to note that good survey questions are carefully designed to maximize clarity for the target population and then carefully administered to ensure consistency in delivery, which is to say they are based on a process designed to maximize reliability.  It would not be consistent with a true qualitative interviewing (or observational) approach to expect such consistency in the wording or delivery of questions.  But that answer certainly doesn’t tell us how to proceed when we find that answers about the same issue are contradictory.  I suggest that in situations like this you need to consider two issues:  (1) Is one approach (usually the structured questions) missing the mark with this population?  That’s what I concluded in the housing and homelessness study in relation to an index of social support that had been developed with college students (Schutt 2011:305-306). (2) Is the difference between the two methods revealing that the concept you intended both to measure is multidimensional or otherwise more complex than you had anticipated?  This is a very common source of inter-method differences in responses.  Again, a good example emerged in my analysis of data from the NIMH study.  In our structured baseline interviews, we included a well-tested index of ability to manage the tasks of daily living.  I found that scores on that index were associated positively with strength of preference for living independently, without staff support (Schutt et al. 1992).  However, when residents were observed over time by ethnographers, and when their case managers assessed their behavior, we discovered that those who were most confident they could manage the tasks of daily living on their own were, on average, least able to do so.  (Schutt 2011:Chapter 3).  So the self-report index was actually measuring self-confidence in ability to manage the tasks of independent living and so could not be expected to be correlated with a behavior-based measure of such ability.  So the fact that qualitative and quantitative “measures” could result in apparently contradictory results is not a reason to avoid mixed methods, but rather an indication of how valuable mixing methods can be to uncovering a more complex social reality than we had anticipated. 

Schutt, Russell K., Stephen M. Goldfinger and Walter E. Penk. 1992. “The Structure and Sources of Residential Preferences Among Seriously Mentally Ill Homeless Adults.”  Sociological Practice Review, 3:148-156.

 

• What is the format for writing mixed methods results?

 

In articles, I usually intersperse qualitative quotes or observations with tables or graphs of related quantitative measures, but there is really no one format that is always preferable.  Occasionally you’ll see a presentation of quantitative findings in a separate section before or after a section on qualitative findings.  A key issue in deciding which approach to take is whether the different methods are being used to provide more insight into the same issue, or to investigate different aspects of an overarching research question.

 

• How do you determine which method is most appropriate for the given research? e.g. driven by the research question solely?

Deciding which is “most appropriate” must take into account feasibility given time and resources and capacity given training and experience, as well as the research question itself and the setting in which it can be investigated.  I emphasized in my Webinar that a mixed methods design should be considered the more the research question is original, complex, has ambiguous or conflicting implications, and suggests a challenge for an authentic identification of a causal mechanism or causal context. We can enrich the answer to any research question by mixing methods, but it is not necessarily a good idea to do so if resources are limited and the question is relatively simple, etc.  It’s also important to have sufficient expertise with each planned method and/or a plan with appropriate collaborators before deciding to adopt a mixed methods design.  But no researcher should feel that they are somehow missing the boat if they don’t use mixed methods in a particular investigation, as the insights from so much single-method research shows us.  Rather, it’s a matter to be considered in identifying limitations of a single study and avenues for further exploration.

• In considering what methods to use, is there any difference when considering evaluation questions vs research questions?

It is hard to make a firm distinction between evaluation and research projects, and hence between evaluation and research questions.  Admittedly, some Institutional Review Boards (IRBs) interpret federal guidelines as allowing evaluation research on behalf of a client to be exempt from human subjects review, so that can be one practically important difference, but that is not a key issue from the standpoint of the appropriateness of different methods.  If an evaluation question strictly involves an impact analysis of a program, or an efficiency analysis of program costs and benefits, the funder may only be interested in quantitative outcomes and predictors.  However, evaluations designed as needs assessments, evaluability assessments, or as process evaluations usually include a major qualitative component and hence are likely to involve mixed methods.  See chapter 12 of Investigating, 8^th edition, for a more detailed discussion of these types of evaluations and the value of qualitative methods in evaluation research (pp. 463-464).  On the other hand, if we are attempting to answer a research question that extends a specific prior line of research in a very specific way, only one method may be appropriate.  Think of extending the classic social psychological research by Aronson and Mills (1959) on severity of initiation and liking for a group.  Subsequent research has extended this study without any need for qualitative additions; but what if you want to explore how people interpret the initiation experience?  Again, it always makes sense to consider mixed methods, even if this leads to a decision to concentrate efforts with just one method.

 

• I have found that there are a host of mixed methods designs to choose from. What aspects of your study did you use to choose your design?

I discuss the mixed methods designs options in chapter 15 of Investigating the Social World, 8^th edition, and distinguish them in terms of the dimensions of priority and sequence in Exhibit 15.2.  Using that scheme, I categorized our NIMH-funded study as having used an “Integrated Method” design, symbolized as QUAL+QUAN.  We had both practical and scholarly reasons for choosing this type of mixed methods design.  Practical reasons were that we were able to secure a large amount of funding for the study ($3.1 million from NIMH for the research and related services and $10 million from HUD for developing the housing alternatives) and we had connections with excellent researchers having different types of expertise and ourselves represented (that is the co-investigator team) represented different disciplines.  In this way, a massively mixed methods design was made possible, with resources to collect and analyze data ranging from thousands pages of ethnographic notes to neuropsychological test results.  Scholarly reasons were that we were not so certain that our standard quantitative instruments would capture the many aspects of our research participants’ orientations and experiences that we knew could influence their success, and that we intentionally planned a process of “evolution” in the group homes that required constant observation and introduced many ambiguities.  More than anything, the need to understand people in a group context made it imperative for us to “be there” as things happened.  When you read my book (Schutt 2011), you can decide for yourself how well our mixed methods design allowed us to understand these complex issues.

• Will it make sense then to separate the mentally ill based on what knowledge we have gained into both group living and individual living or apartments based on the mixed methods approach, (quantitative and ethnographic studies)? As this is likely to increase success with our intervention program given the fact that there are individual differences among these residences.

I conclude Homelessness, Housing, and Mental Illness (Schutt 2011) with a section on Policy Recommendations (pp. 277-281).  I will quote from a few of those recommendations that bear on this question of housing alternatives. 

The most important policy recommendation from the analysis … is to give community processes their due in housing programs.  … Policies that presume direct placement into independent apartments ignore deficits in social connections at their peril…. Integration within the larger community will only succeed in the context of participation in a meaningful social community. Neither group homes nor independent apartments provide a simple path along which persons who have been homeless and diagnosed with serious mental illness can return to the community. Building social ties, overcoming social isolation, learning to identify what is needed for sustained community living, and engaging with others to meet these needs are the keys to successful community living.

 

• Also, can Dr. Schutt recommend some very useful articles for qualitative research that he thinks might help us in our learning process? Thanks so much.

 

I selected what I believe are very useful articles using mixed methods to illustrate different techniques in Investigating the Social World, 8^th edition.  I did the same for the two chapters on qualitative methods (10 and 11).  In order to gain a basic, solid understanding of qualitative methods per se, I suggest purchasing the book and reading both chapters 10 and 11.  Good articles to start with for examples would be those I selected from SAGE journals for the introductory “Research That Matters, Questions That Count” that begin the chapters of Investigating, 8^th edition.  These are:

Kato, Yuki, Catarina Passidomo, and Daina Harvey. 2013. “Political Gardening in Post-Disaster City: Lessons from New Orleans.” Urban Studies 51: 1833-1849. (chapter 10)

Rinehart, Jenny K. and Elizabeth A. Yeater. 2011. “A Qualitative Analysis of Sexual Victimization Narratives.” Violence Against Women 17(7):925–943. (chapter 11)

Testa, Maria, Jennifer A. Livingston and Carol VanZile-Tamsen. 2011. “Advancing the Study of Violence Against Women Using Mixed Methods: Integrating Qualitative Methods Into a Quantitative Research Program.” Violence Against Women 17:236–250. (chapter 15)

• Are there any kinds of research questions that you would say would not profit from mixed methods?

 

I will repeat a portion of my answer to question 18, above:

If we are attempting to answer a research question that extends a specific prior line of research in a very specific way, only one method may be appropriate.  Think of extending the classic social psychological research by Aronson and Mills (1959) on severity of initiation and liking for a group.  Some subsequent research has extended this study without any need for qualitative additions.

 

• What determines whether you begin your mixed methods study with the qualitative (discovery) vs quantitative (relationships) analysis? Is one sequence more common than the other and why?

As I discuss in relation to Exhibit 15.2 in Investigating the Social World, 8^th edition, some types of mixed methods designs do not use a sequenced approach.  However, among those that do, certainly the use of qualitative methods for an initial more exploratory investigation of a research question, followed by a structured quantitative investigation (often a structured survey) is the most common sequence.  However, it is not uncommon for researchers to conduct a quantitative survey and then identify cases from the resulting dataset that meet particular criteria for the purpose of conducting more intense qualitative interviews.  This was the approach in the research by Zhong and Arnett (2014) that I discuss in my mixed methods chapter (pp. 547-548).  Researchers are also starting to use more complex mixed methods designs that alternate between qualitative and quantitative methods.  Brown et al.’s (2013) study of sexual intimacy among homeless men in downtown LA is the example I use of this type of complex design (Schutt 2011:548-551).

 

Brown, Ryan A., David P. Kennedy, Joan S. Tucker, Daniela Golinelli, and Suzanne L. Wenzel. 2013. “Monogamy on the Street: A Mixed Methods Study of Homeless Men.” Journal of Mixed Methods Research 7:328–346.