The original paper is under review, citation format:
Shaokang Jiang and Michael Coblenz. An Analysis of the Costs and Benefits of Autocomplete in IDEs. In review, ACM International Symposium on Foundations of Software Engineering (FSE 2024)
Key points:
- Worked with Michael Coblenz on the usability analysis of autocomplete.
- Designed and executed an experiment with 32 participants using an eye tracker to evaluate the costs and benefits of IDE-based autocomplete features to programmers who use an unfamiliar API; analyzed data using JMP; and wrote a paper for the study.
- Found that participants who used autocomplete learned more about the API while spending less time reading the documentation; found autocomplete did not significantly reduce the number of keystrokes required to finish tasks.
- Acquired fundamental skills in conducting empirical research studies and learned various methods for handling and understanding the implications of eye-tracking data.
- Successfully designed an experiment with well-formed hypotheses and a comprehensive knowledge test suite to accurately capture the benefits and costs of autocomplete. Additionally, designed procedures and produced scripts to analyze data from various sources, including a substantial amount of data from a consumer-level eye tracker.
In more detail:
Upon starting my master’s program, I took a usability analysis of programming languages course. Realizing Python type hints could be useful in checking types, and providing more context to IDE for generating useful information, all these properties could be helpful for programmers to write better code. The reduced number of typing errors could potentially assist competitive programmers more by minimizing time spent on fixing typing errors. I investigated the impact of Python type hints feature on the performance of competitive programmers (N=5). Our findings revealed that Python type hints did not significantly enhance their task completion performance (𝑝 ≈ 0.51, 𝑑 ≈ 0.81). Survey results indicated that programmers tended to dislike type hints and autocomplete suggestions, citing them as annoying.
Together with my previous teaching experiences, where I observed programmers grappling with autocomplete suggestions and observed eyes and mouse drifting during active autocomplete use, I aimed to understand the genuine benefits and costs of autocomplete. Considering that autocomplete could potentially be harmful to learning due to the ease of access to knowledge, we conducted an experiment (N=28) to measure the learning impact of autocomplete when programmers face new materials and gauge the productivity effect during its usage. The results are expected to benefit all programmers using modern IDEs, with potential applicability to the general public relying on browser autocomplete.
During the experiment, I utilized an eye tracker to detect users’ gaze, aiming to understand the specific parts and autocomplete options that programmers focused on. I developed a novel monitoring platform based on VSCode, integrating eye-tracking at the source code level. Additionally, I designed a knowledge test suite to measure new knowledge acquisition after tasks and introduced a post-study survey for capturing individuals’ opinions.
I found the main benefit of autocomplete speeding up information acquisition by significantly reducing the time spent reading documentation (𝑝 < 0.01, 𝑑 = 1.76). It also significantly increases the knowledge testing score (𝑝 < 0.01, 𝑑 = 1.1). An insignificant result in the total number of keystrokes across experimental conditions (𝑝 ≈ 0.65, 𝑑 = 0.18) helps us assert that autocomplete does not reduce typing time, thus not making programmers more productive. From designing to executing this study, I learned a lot, including ways of determining the area of interest in eye-tracking studies, methods of designing participant-friendly tasks that align with the project’s goals, quantitative analysis techniques, and methods for hypothesis testing.
I faced numerous challenges during my research project. Integrating an eye-tracker allowed me to delve deeper into the usefulness of autocomplete and the involved cognitive processes, but it also presented challenges. Given the limitations of the previous monitoring platform, including a restricted area of interest, a less popular platform, high latency, and artificial coding restrictions, I developed a new monitoring platform based on the more popular VSCode, integrating eye-tracking at the source code level. I tested various eye trackers, wrote code to evaluate fixation algorithms, and consulted with I-trace experts for optimization insights.
Experiment design posed another significant challenge, requiring the creation of a robust, controllable, and effective experiment that could reveal the benefits or drawbacks of autocomplete. After proposing various experiment versions and conducting pilot experiments, I selected a design utilizing a task suite based on common-sense knowledge and a modern API library. This design included a knowledge test suite, an eye tracker, and a post-study survey. The successful execution of the entire experiment suggests that I effectively overcame these challenges, showcasing my ability to be both persistent and resourceful in graduate school.