Design

We have designed a comprehensive mixed-methods study, relying on both quantitative and qualitative methods as well as both formative and summative research to better understand the role of the TSLC experience on select students’ college trajectory. In 2011, MIT started an impact evaluation as to whether or not the STBF’s financial aid (as well as financial aid plus TSLC experience) leads to better academic achievements for recipients. We leveraged their research design for the summative research. In addition, our mixed-methods design will further contribute to the research by providing important contextual information regarding the STBF scholars’ participation in these TSLCs. The study outlined in this project adds onto MIT’s important study by understanding the complex interplay of financial aid and other forms of student support. Here we provide a brief overview of each of the four methods –longitudinal survey, focus group discussions, case study, and longitudinal qualitative cohort.

The main purpose of the longitudinal survey is to draw connections between student characteristics, traits, attitudes, and behaviors; membership in the TSLC and various outcomes. The survey begins with a pilot of key constructs. Both the 2015 and 2016 baseline component of the longitudinal survey will occur at the beginning of the academic year. This allows the research team to capture data on pre-college experiences and college expectancies prior to substantial exposure to the TSLC treatment. Each spring, we will administer the follow-up component of the longitudinal survey. The purpose of the follow-up component is to collect interim measures of growth on key outcomes and to capture data on student engagement with programs and activities believed to be related to that growth.

An additional component of the survey is a summative assessment. For the summative assessment of the project, we will use the information collected through the longitudinal survey for the two cohorts of students, and we will focus exclusively on conducting rigorous quantitative analyses to test for differences related to program participation. The research design of the summative component of the assessment focuses on a number of outcomes (i.e., academic development and performance, academic and social self-efficacy, career and major commitments, resiliency, mattering, and sense of belonging) that, according to our conceptual framework and the literature, are the ones that programs such as learning communities are designed to impact.

Focus group discussions (FGDs) are a key component of the mixed methods design and will serve three main purposes. The first purpose is to help explain survey findings. The second purpose is to help the team develop new TSLC-specific survey modules. In addition to the survey’s base components, which will be included without change over the course of the entire study, several modules will be incorporated into the survey at different points in time to ask topic-specific questions of certain groups of students. Finally, over the course of the study the team may find it necessary to conduct additional FGDs that do not directly correspond with the survey but may prove necessary to understand other study elements. For example, there is interest in understanding less engaged students and the focus groups might be the best mechanism for understanding this subgroup of students.

Case studies of the three campuses will be conducted using the following approaches: 1) Observations of various program components; 2) Interviews with faculty and staff; and 3) Document social and print media analysis. The case studies serve two main purposes. The first purpose is to better understand the program elements in the three different campuses. A second purpose is to document the ways the program operates to shape student experiences. By spending extensive time observing the program, participating in events, understanding the history of programs and viewing the ways that groups and individuals interact we can begin to better capture how and why the program works.

The second major part of the qualitative research design will be the use of digital diaries. The purpose of the diaries is to develop a deep and holistic understanding of the student experience in the program. Student digital diaries (self-created videos) allow us to obtain data continuously over time (opposed to snapshots in interviews) and from the students’ perspective since they initiate data collection (as opposed to interviews where researchers lead the questioning). We will follow two cohorts of students for three years for a total of four years of data collection. We will conduct detailed follow up interviews every 4-6 weeks to probe questions about the videos they create. In these interviews, we will ask questions that relate to their own backgrounds (for example, being a commuter student), relate to the program (ways they are getting along with their mentor or not), as well as other themes that emerge through review of the digital diaries.