Collaborative Ethnographic Assessment: An Anthropological Rubric for a Community Ecosystem

The ethnographic team of a community-based engineering project in South Bend, Indiana, continues to create modes of anthropological assessment while conducting collaborative research. The Bowman Creek Educational Ecosystem (BCe2) is a National Science Foundation-funded project designed to restore and enhance a vital but polluted St. Joseph River tributary by linking the efforts of local community groups, schools, and universities in a revitalizing small city. This paper describes the impetus and creation of an ethnographic rubric for assessing community-based anthropological research towards potential replication in future collaborations. Based on a modification of Rapid Ethnographic Assessment (REA), used widely in environmental, medical, military, and other research applications, this paper offers an REA modification called Collaborative Ethnographic Assessment (CEA).


Introduction
But when you think about it…this notion of collaboration or working together…the political climate is "America first" and 'we're going to do this on our own!" and "we're in this by ourselves" and "we're the best" and that's literally nothing about what a community looks like or what a neighborhood looks like or what a family looks like…We are doing a lot of things differently and bringing a lot of people together who wouldn't have come together naturally on their own so it makes me feel better that we are not mimicking the political climate [laughs] and just thinking that "we have all these smart kids from all these smart schools and we're going to save this stupid world." It makes me feel good [laughs]...that we're trying to make these connections and build these relationships" ("Duck," South Bend, Indiana, July 2017). i From the summer of 2016 to the present summer of 2019, the ethnographic team of a community-based engineering project in South Bend, Indiana, continues to create modes of anthropological assessment while conducting collaborative research. The Bowman Creek Educational Ecosystem (BCe2) is a National Science Foundation-funded project designed to restore and enhance a vital but polluted St. Joseph River tributary by linking the efforts of local community groups, schools, and universities in a revitalizing small city. This paper describes the impetus and creation of an ethnographic rubric for assessing community-based anthropological research towards potential replication in future collaborations.
The ethnographic team inherited several research rubrics listed in the initial grant, but none of them offered the basic ethnographic toolkit that would become vital to our research engagement with the engineering teams and advocacy with the broader community. Initially, we offered basic methodological training to the two ethnographic team interns who were anthropology majors including interview techniques, participant observation, personal experience methods, media analysis, semiotic analysis, etc. This methodological approach formed the basis for our research among the BCe2 organization in the first summer extending to community advocacy outreach in the following summer of 2017.

Bowman Creek Educational Ecosystem (BCe2) internship team creating a new rain garden, 2017
While creating project updates and grant reports, because one of the promised deliverables was to provide modified rubrics, it became structurally beneficial to retroactively frame our methodological approach within a pre-existing ethnographic rubric and then to describe how we were modifying it through practice. However, there are few ethnographic toolkits framed in referable rubric packages with one notable exception. This paper describes our organizational quest to create an ethnographic rubric, based on a modification of Rapid Ethnographic Assessment (REA), which is used widely in environmental, medical, military, and other research applications. Because of REA's applicability to landscape restoration and its emphasis on time-sensitive projects, it matches well with BCe2's ecological focus and limited ten-week summer internship schedule. Our ethnographic approach is a set of holistic frameworks that extends REA in several key areas, particularly by focusing on community engagement and neighborhood advocacy. With many parallels to Gretchen Seuss' recent research (2018), for the purposes of providing working methodological models amid funding-driven outcomes, this paper describes an REA modification called Collaborative Ethnographic Assessment (CEA).

Evaluating Collaboration and Existing Rubrics beyond Temporal Limits
In May 2016, when I was assigned to BCe2 and began to navigate the project without understanding the structural importance of rubrics and assessments or other evaluative categories such as the Accreditation Board for Engineering and Technology (ABET) accreditation measures governing engineering programs (ABET, 2017;Laporte, 2017), we designed the ethnographic team's methodology from the ground up. This approach drew from previous methods courses, techniques, and modes of analysis that seemed applicable, given our limited understanding of the project at that time. Methods such as formal interviewing, participant observation, document analysis, and personal experience served as an introductory training approach for the two ethnographic interns researching the project that summer (Adler & Adler, 1994;Denzin & Lincoln, 1994;Holstein & Gubrium, 1995;Jorgensen, 1989;Madison, 2005;Thomas, 1993). As a working ethnographic rubric, these methods also emphasized collaborative ethnography, cultural brokerage, and community advocacy (Lassiter, 2005;Lassiter, 2006, Van Willigen, 2002, which would become central to BCe2's collaborative maturation from the STEM-focused interventions that predominated at its origins. For reporting purposes, the ethnographic team was obligated to interface with the initial research rubrics attached to the project proposal, modifying and justifying what worked and what did not. Of the four rubrics provided, Dimensions of Success (DoS) is a proprietary observation tool that requires paid training, certification, and a continual cycle of recertification. DoS frames its ability to quantify STEM learning opportunities through limited binaries of strengths and weaknesses that demand analytical transcendence. Furthermore DoS-certified researchers are tasked with using "the rubrics to assign ratings for four dimensions [STEM Knowledge and Practices, Activity Engagement, etc.] that are backed up with evidence from the observation" (PEAR Institute, 2017). This approach limits the holistic applications and open-ended contributions of ethnographic observation by focusing solely on evidence confirming one of DoS's four categorical dimensions.
Similarly, two other rubrics from the BCe2 research proposal also tend to discourage organic findings from ethnographic discovery. Both of these VALUE rubrics from the Association of American Colleges and Universities (AAC&U) frame their approach to "Problem Solving" and "Leadership" through a progression of benchmarks, milestones, and capstones. Ethnographic research is particularly suited for confirming these presumed achievements, such as "identifies multiple approaches for solving the problem that apply within a specific context" (PEAR Institute, 2017), but these rubrics encourage a myopic approach of binary confirmation. From the first week of the project, the Ethnographic Team asserted that we could provide evidence that each of these rubrics were being fulfilled and stressed the importance of additional assessments that allowed for organic research discoveries.
Like the DoS observation model and the Teamwork and Problem Solving Value rubrics, the BCe2 project's three main research questions also tend towards confirmation rather than discovery. For example, Research Question 1 begins, "How do students…conceptualize STEM education and career pathways? How does exposure to an innovation ecosystem with multi-dimensional diversity change that conception?" These questions suggest an overall potential that research findings could merely confirm BCe2's programmatic design goals, its obvious application for STEM recruitment and retention. In addition to this potential for uncritical project

Bowman Creek internship team community cookout with South Bend
Mayor, Pete Buttigieg, 2016 The PEAR Institute's Dimensions of Success (DoS) proprietary rubric promotion, these core goals oppose my occupational priorities as the Director of Undergraduate Studies for an anthropology department undergoing the familiar loss of majors shared by similar programs in the continual fallout from the 2008 financial crisis (Ginsberg, 2017). Of BCe2's three research questions, part of the second seemed to offer the most holistic potential for anthropological discovery, "What factors facilitate or inhibit positive collaboration?" (Brockman, Torres, Wood & Blum, 2016). By working among the project's engineering and intern teams, this question seemed to offer a relatively rare window for critical applications of practicing anthropology.
The project description described assessment measures as, "analysis of tool quality based on triangulation with observational data" (Brockman et al., 2016) Thus, although the initial assessment measures were confined to observations, this small window of rubric analysis provided an entry point for broader ethnographic methodologies. The project deliverables included modified rubrics "drawn from the literature and STEM education practice...modified…for the differences in audience environment, and core research interests" (Brockman et al., 2016). For the engineering project leaders, the cited rubrics and assessment tools were the cutting-edge of STEM literature. Given the above critiques, however, we needed another rubric to modify. Certainly, we expected that pre-existing ethnographic rubrics would be available for assessing decades of anthropological engagement with projects that demand deliverables.

Rapid Ethnographic Assessment of planned mobile health clinics in Mozambique, 2013 photo credit: Makini Boothe (2002)
One of the difficulties of a literature search for "ethnographic rubrics" is the tendency to yield classroom models for evaluating introductory ethnographic writing. Rubrics using holistic anthropological toolkits might exist under the blanket of this educational literature, but generally, ethnographic methodologies tend towards booklength descriptive exercises. In BCe2, these core texts informed our shared understanding of ethnographic methodologies (Atkinson, 2001;Emerson, Fretz & Shaw, 2011;Faubion & Marcus, 2009;Hammersley & Atkinson, 1995). The crucial, but subtle concept that anthropological methods are not just a set of techniques, but are an application of theory to fieldwork is difficult to represent in shorter works and research descriptions. Nevertheless, our funding obligations and collaborative translations for BCe2's engineering-centered STEM framework still required a modified ethnographic rubric. Subsequent searching revealed that the phrase "ethnographic assessment" offered an efficient path around the vast swath of classroom-based literature for evaluating student ethnographies. This refocused search highlighted Rapid Ethnographic Assessment (RAE) emerging from public health as Rapid Assessment, Response and Evaluation (RARE), park preservation as "They are offering us the experience to be leaders…taking a project and breaking it down into pieces and building it up again." Focused Ethnographic Study (FES) or Participatory Rapid Assessment (PRA), and agricultural studies as Participatory Rural Appraisal from under the same acronym (Williams, Ramos, Brown, Juárez & Associates, 1997). RAE's interdisciplinary context, characterized by a three-week to four-month limited timeframe, triangulated methods, community engagement, and collaborative research teams made it ideal for BCe2's ethnographic team (Beebe, 2014). Its origins in environmental preservation, community programs, and military organizations mirrors BCe2's operational mission and tendency towards "security ecology," a totalizing framework of everyday securitization (Beebe, 2001;Taplin, Scheld & Low, 2002) The centrality of compressed time and pressing concerns in REA methodologies separates them from longer forms of ethnographic investigation primarily in terms of scale. In a related comparison, Rapid Situational Awareness emerged out of REA and WWII tactical history, creating a method to train US Army soldier's assessment of unfamiliar cultural encounters (Nolan, LaTour & Klafehn, 2014). Like other military applications of REA (Schultz, Arsdale & Knop, 2009) this situational awareness model emphasizes rapid perceptions from months of immersion to minutes and seconds of field encounters (Nolan et al., 2014). Yet subverting these pervasive temporal compressions is a necessary aspect of all contemporary ethnographic fieldwork methodologies. Even with the rare luxury of traditional immersion of a year or more, fieldwork routinely encounters temporal scarcity. Thus, a de-emphasis on rapidity in RAE variants balanced with an emphasis on collaboration signals a crucial modification in BCe2's modes of ethnographic assessment, particularly given the projects' emphasis on sustained community engagement.
Collaborative Ethnographic Assessment (CEA) recognizes the limited timeframes and pressing issues of contemporary ethnographic research. Each annual reformation of BCe2's ethnographic team conducts their work within the constraints of the ten-week summer internships. The inaugural 2016 team focused on the project proposal's three research questions, quickly confirming many of the more binary STEM-based outcomes (Blum, Barnes, Huggins & Haanstad, 2018). For example, the nearly rhetorical second research question, "Does this experience build the 21 st Century skills needed for STEM professionals?" was demonstrated in the first project team meeting. Part of that same question, "What factors facilitate or inhibit positive collaboration?" offered a more open-ended opportunity for ethnographic exploration (Brockman et al., 2016). The 2017 team built on these holistic explorations by focusing more specifically on community collaboration. The 2018 team continued to highlight collaborative engagement while building its capacities for neighborhood advocacy and community liaisons.
While CAE is designed to fulfill requisite project-based research questions including the potential for rubric modifications, its core presupposes condensed timeframes while highlighting holistic evidence of collaboration across participants, communities, and other partners. Documenting collaborative moments, from the profoundly mundane to the structurally formal, is central to CAE methodology. Data collection includes minute observations such as the congenial body language of comfortable group dynamic and the use of "we" and "us" in project conversations. It also includes less nuanced, but more complex appearances of collaborative experiences such as the fomentation of local community advocacy between project teams and neighborhood residents and the coalescing of leadership roles among cooperative groups. Interview data can be coded as part of CAE for keywords such as "community," "collaboration," "integration," "partners," etc. Data can also be analyzed for its collaborative opposite, particularly when compared with initial moments of unfamiliarity, discomfort, and distance. Analyzing a holistic range of ethnographic data sources in light of the increasing appearance of collaborative encounters forms the underlying basis for CAE's evaluative capacities.

Conclusion
I've never had a home before this…I feel connected…the work I did is meaningful. I like to look at murals and think "I helped with that" and I like to see the rain gardens and say "I know how those work" and to build a park and visit and think "This helped the community" ("Cal," South Bend, Indiana, July 2017).

BCe2's Collaborative "informal place-based learning in an authentic environment" South Bend 2017
In his canonical handbook for ethnographic methods, Bernard writes, "if you have a clear question and a few, clearly defined variables you can produce quality work in lot less time than you would imagine" (Bernard, 2011, p. 265). Beebe describes rapid assessments as differing from standard qualitative research in their reliance on multidisciplinary team interaction, which produces faster results (Beebe, 1995, p. 42). These approaches suggest that although rapidity is a critical factor, focused questions and collaborative work are equally important. Accelerating temporal restrictions in most contemporary anthropological research make featuring rapidity somewhat redundant as a central methodological feature. The BCe2's research trajectory continues to highlight the crucial importance of collaboration through multidisciplinary teams of engineers, educators, and anthropologists and through our ongoing emphasis on community liaisons who are part of the neighborhood from the outset (Haanstad, Robinson, & Webb, forthcoming). At the beginning of the project, our engineering partners asserted that they wanted to do research with the community not for the neighborhood. By emphasizing collaboration's centrality to the ethnographic team's modes of assessment, BCe2 can incorporate the longstanding anthropological commitment to community advocacy. Thus, Collaborative Ethnographic Assessment can generate research both with and for the neighborhoods we work within while continuing to navigate the temporal compressions that characterize hyper-modern existence.