Advancing software evaluation rubrics in the era of big data: The value of genre and activity theories

Date of Award

May 2020

Degree Type

Dissertation

Degree Name

Doctor of Professional Studies

Department

School of Information Studies

Advisor(s)

Michelle Kaarst-Brown

Keywords

activity theory, data analysis, genre theory, information systems, practice, sociomateriality

Subject Categories

Social and Behavioral Sciences

Abstract

Approved for Public Release; Distribution Unlimited. Public Release Case Number 20-1046

Government and other large organizations are moving from traditional information technology infrastructure and custom, proprietary analytic solutions to cloud environments, shared data, and large-scale data analytics (“big data”). This shift highlights the potential for organizational actors to experience conflict in managing competing internal and external demands. As important information technology stakeholders, data analysts are situated at a crossroads. Strategic, enterprise imperatives to promote efficiencies and sharing collide with situated, mission-oriented analytic practices. The ability to understand and predict how and where analysts must grapple with internal cultural and historical forces to navigate these external demands offers important benefits. These benefits include a more holistic understanding of what the work of analysis entails and technology’s role in analysis, and the creation of important information for planning for, and facilitating, successful technological change.

The literature concerned with information technology for intelligence analysis commonly frames analysis as an individual, internal, mental activity. On more rare occasions, the literature may take the stance that intelligence analysis is a social activity. My research takes a broader view, proposing intelligence analysis is a historically and culturally informed activity comprised of a dynamic network of people, artifacts, and social objects, such as rules. These two different perspectives on intelligence analysis can impact the systems development process, including the evaluation of “built to order” or purchased software.

Given the value proposition of framing intelligence analysis as a sociotechnical process, the broad research question is, how can sociotechnical theories contribute to identifying the value of information technology in analytic practice? In particular, how can genre theory compared to activity theory contribute to evaluation rubrics for analytic technologies? While not designed to study how analytic software is evaluated or to evaluate any particular analytic software, the purpose of this study is unpack the practice of data analysis through the lenses of genre and activity theories in order to explore the role of context in an analytic team’s work with various tools.

This research reports on a field study of one analytic group, organizationally situated and actively working to create analytic products, using multiple software and artifacts. Data collection employed semi-structured interviews, observation of the site, and document analysis. Data analysis consisted of deductive coding based on genre and activity theory; inductive coding to identify themes emerging from the data; and, synthesis of findings from each theoretical lens to identify tacit tool-relationships integral to analytic practice.

Although a few studies have used genre and activity theories together, this study is unique in that it applies each theory to the case to provide two views of the relationship between IT tools and context in analytic practice and then compares and synthesizes these to obtain a more nuanced view of the way IT tools attach to other components sustaining the web of practice. Each framing positions IT as one of many components working together in a complex sociotechnical network, as opposed to reified artifacts for insertion or evaluation. The theoretical views, when used separately or together, stand to provide new insight into analytic work and identify opportunities for improvements in analytic tools, as well as a more holistic approach to IT evaluation in analytic and other data-centric environments.

Access

SURFACE provides description only. Full text may be available to ProQuest subscribers. Please ask your Librarian for assistance.

This document is currently not available here.

Share

COinS