This goal of this course is to introduce and discuss concepts in research methodology, empirical analysis, and the scientific enterprise in computing. This course will prepare students for conducting research by examining how to plan, conduct, and report on empirical investigations. The course will cover techniques applicable to each of the steps of a research project, including formulating research questions, theory building, data analysis (using both qualitative and quantitative methods), building evidence, assessing validity, and publishing. The course will cover the principal research methods used to study human interaction with computer technology: controlled experiment, case studies, surveys, archival analysis, action research and ethnographies. We will also cover topics in peer review, ethical obligations involving human subjects research, how to give a scientific presentation, and how to write research papers, survey papers, and funding proposals.
Prerequisites
Enrolled as a Graduate Student in CSE or by instructor permission.
Logistics
Class Information
Lecture:
T/R 9:30am – 10:45am
206 Debartolo Hall
Instructor
Dr. Tim Weninger (tweninge@nd.edu)
Office Hours:
Tue 11:00am-12:00pm in 380 Fitzpatrick Hall
or by appointment
Teaching Assistants
None
Course Format and Activities
| Week | Date | Topic | Discussion Leaders | Pre-Reading | Assignments |
|---|---|---|---|---|---|
| 1 | 01/16 | Introduction | None | ||
| 1 | 01/18 | History and Philosophy of Science | Okasha, Ch 1-3 | ||
| 2 | 01/23 | Critical Reading of Research | Chandrasekharan, Eshwar, et al. “You can’t stay here: The efficacy of reddit’s 2015 ban examined through hate speech.” CSCW (2017): 1-22. | ||
| 2 | 01/25 | Critical Reading of Research | Muchnik, L., Aral, S., & Taylor, S. J. (2013). Social influence bias: A randomized experiment. Science, 341(6146), 647-651. | ||
| 3 | 01/30 | Peer Review | Bohannon, John. “Who’s afraid of peer review?.” Science. (2013): 60-65. Tomkins, A., Zhang, M. and Heavlin, W.D., 2017. Reviewer bias in single-versus double-blind peer review. Proceedings of the National Academy of Sciences, 114(48), pp.12708-12713. http://blog.mrtz.org/2014/12/15/the-nips-experiment.html | ||
| 3 | 02/01 | How to Write a Peer Review | |||
| 4 | 02/06 | Morphology of a Paper and Technical Writing | Tim Weninger PPT | Weekly Review: Glenski, M., Stoddard, G., Resnick, P., & Weninger, T. (2018). Guessthekarma: A game to assess social rating systems. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), 1-15. | Weekly Review Due |
| 4 | 02/08 | LaTeX and BibTeX | Weninger Example | ||
| 5 | 02/13 | How to Write your Research | |||
| 5 | 02/15 | How to Write a Survey | |||
| 6 | 02/20 | How to Make a Research Presentation | |||
| 6 | 02/22 | How to Make a Research Presentation | PPT1 PPT2 PPT3 Discussion Leader: Byron Dowling | Weekly Review: Boyd, A., Tinsley, P., Bowyer, K., & Czajka, A. (2021). Cyborg: Blending human saliency into the loss improves deep learning. arXiv preprint arXiv:2112.00686. | |
| 7 | 02/27 | Revising and Publishing Research | Discussion Leader: Tasha Januszewicz | Weekly Review: Roy Chowdhury, A., Ding, B., Jha, S., Liu, W., & Zhou, J. (2022, November). Strengthening order preserving encryption with differential privacy. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security (pp. 2519-2533). | Literature Review Due |
| 7 | 02/29 | Computing as a Discipline | Discussion Leader: Ellen Joyce | Double Blind Who’s Harry Potter? Approximate Unlearning in LLMs. (under review) | |
| 8 | 03/05 | Research Funding and Proposal Writing | Discussion Leader: Maria Dhakal | Weekly Review: Ahmed, T., & Devanbu, P. (2022, October). Few-shot training LLMs for project-specific code-summarization. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering (pp. 1-5). | |
| 8 | 03/07 | IRB, Ethics, and Research Malpractice | Discussion Leader: Emelia Hughes | Bhandari, Aparajita, and Sara Bimo. 2022. Why’s Everyone on TikTok Now? The Algorithmized Self and the Future of Self-Making on Social Media. Social Media + Society 8(1) | |
| 9 | 03/12 | Spring Break | |||
| 9 | 03/14 | Spring Break | |||
| 10 | 03/19 | Basics of Research, Theory Building | Discussion Leader: Robert Wallace | Weekly Review: Su, C. Y., & McMillan, C. (2024). Distilled GPT for source code summarization. Automated Software Engineering, 31(1), 22. | |
| 10 | 03/21 | Study Design | Discussion Leader: Liu Liu | Wu, L., Sharifi, R., Lenjani, M., Skadron, K., & Venkat, A. (2021, June). Sieve: Scalable in-situ DRAM-based accelerator designs for massively parallel k-mer matching. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA) (pp. 251-264). IEEE. | Introduction Due |
| 11 | 03/26 | Experiment Design, Controls, Confounders | Discussion Leader: Shivani Vyas | Weekly Review: Agrawal, A., Abraham, S. J., Burger, B., Christine, C., Fraser, L., Hoeksema, J. M., … & Cox, S. (2020, April). The next generation of human-drone partnerships: Co-designing an emergency response system. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (pp. 1-13). | |
| 11 | 03/38 | Laboratory, quasi and natural experiments | None | ||
| 12 | 04/02 | What do we mean when we we say that we know a thing? | Discussion Leader: Jin Zhou | Weekly Review: Qiu, H., Banerjee, S. S., Jha, S., Kalbarczyk, Z. T., & Iyer, R. K. (2020). FIRM: An intelligent fine-grained resource management framework for SLO-Oriented microservices. In 14th USENIX symposium on operating systems design and implementation (OSDI 20) (pp. 805-825). | |
| 12 | 04/04 | What do we mean when we we say that we know a thing? pt2 | Discussion Leader: Niloofar Sayadi | Schelble, B. G., Flathmann, C., McNeese, N. J., Freeman, G., & Mallick, R. (2022). Let’s think together! Assessing shared mental models, performance, and trust in human-agent teams. Proceedings of the ACM on Human-Computer Interaction, 6(GROUP), 1-29. https://towardsdatascience.com/lessons-from-how-to-lie-with-statistics-57060c0d2f19 | Research Design Due |
| 13 | 04/09 | Distributions and when statistics lie | Discussion Leader: Saiful Islam | Weekly Review: Zhou, D., & Tamir, Y. (2022). RRC: Responsive Replicated Containers. In 2022 USENIX Annual Technical Conference (USENIX ATC 22) (pp. 85-100). | |
| 13 | 04/11 | Distributions and when statistics lie | Discussion leader: Daniel Worae | Zahan, H., Al Azad, M. W., Ali, I., & Mastorakis, S. (2023). IoT-AD: A Framework To Detect Anomalies Among Interconnected IoT Devices. IEEE Internet of Things Journal. | |
| 14 | 04/16 | OLS | Discussion Leader: Mariana Fernández-Espinosa | Weekly Review: Calacci, D., & Pentland, A. (2022). Bargaining with the black-box: Designing and deploying worker-centric tools to audit algorithmic management. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2), 1-24. | |
| 14 | 04/18 | My results are State of the Art, and other lies we tell ourselves. | Discussion Leader: Ning Zheng | Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., … & Horvitz, E. (2019, May). Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems (pp. 1-13). | |
| 15 | 04/23 | How to Evaluate AI Systems | Final Paper Due | ||
| 15 | 04/25 | Class Cancelled | |||
| 16 | 04/30 | How to Evaluate AI Systems | Reviews Due | ||
| 16 | 05/02 | Reading Day | |||
| 17 | TBD | Final Exam | TBD | Final Exam |
This course will draw materials from research literature as well as lessons accumulated over decades of experience in computing research. Students will attend weekly classes, complete frequent readings and reviews, and formulate a short research review article.
This term we will be using Canvas for class discussion. The system is highly catered to getting you help fast and efficiently from classmates and myself.
Lectures and Class Participation
Students should attend all classes. Effective lectures rely on students’ participation to raise questions and contribute in discussions. We will strive to maintain interactive class discussions if possible.
Questions, Discussions, and Help
If you have any questions or need clarification of class material, what should you do? First, try to post your question to the Canvas forum whenever possible, or otherwise email the instructor. The forum is for you and your peers to discuss class-related materials and to help one another. The forum will be monitored closely, but please be aware that we may not be able to answer all questions on the forum in a timely manner, due to the overwhelming number of questions that such a forum sometimes generates. Also, there are obviously things that are not appropriate for the forum, such as solutions for assignments as well as comments or requests to the staff.
In any case, for more thorough discussion, come to our office hours if you can! Don’t be shy. Use our office hours to their fullest extent to help your study.
Requirements
Coursework
Most class meetings will require pre-reading selected by discussion leaders. Those readings will be discussed during class.
Each weekly reading will result in a short writeup.
Discussion leaders will give a talk at the beginning of each class. Discussion leaders for each week are exempt from the readings.
Signup here: first come first served.
Pre-Candidacy Proposal
A term paper is due at the end of the term with several milestones throughout the semester.
Final Exam
A final exam covering the topics in this course will be administered during finals week.
Grade Breakdown
| Discussion Leaders | 10 |
| Weekly Readings/Reviews | 15 |
| Literature Review | 20 |
| Introduction | 10 |
| Research Design | 5 |
| Final Paper | 10 |
| Peer Review | 5 |
| Final Exam | 25 |
Grades
This table indicates minimum guaranteed grades. Under certain limited circumstances (e.g., an unreasonably hard exam), we may select more generous ranges or scale the scores to adjust.
Total Grade
90-100 A-, A
80-89 B-, B, B+
70-79 C-, C, C+
60-69 D
Polices
Textbooks
Textbooks are required, but generally very cheap or free.
Salganik, Matthew J. Bit by bit: Social research in the digital age. Princeton University Press, 2019.
Okasha, Samir. Philosophy of Science: Very Short Introduction. Oxford University Press, 2016.
Lectures
Students should attend all classes. Effective class meetings rely on students’ participation to raise questions and contribute in discussions. We will strive to maintain interactive class discussions if possible.
Lecture capture and Zoom will not be provided.
Regrading
All requests to change grading of any course work must be submitted to the instructor in writing within one week of when the grades are made available. Requests must be specific and explain why you feel your work deserves additional credit. Do not ask for a regrade until you have studied and understood our sample solution.
Late Work
All scheduled due dates/times are US Eastern Time. Homework is typically due at the beginning of class on the due date, but check each the assignment for specifics.
Due date/time will be strictly enforced. Missing or late and/or unannotated work gets zero credit. If you are unable to complete an assignment due to illness or family emergency, we will understand but please see the instructor as soon as possible to make special arrangements. All such exceptional cases must be fully documented.
Academic Integrity
Notre Dame Students are expected to abide by Academic Code of Honor Pledge:
As a member of the Notre Dame community, I acknowledge that it is my responsibility to learn and abide by principles of intellectual honesty and academic integrity, and therefore I will not participate in or tolerate academic dishonesty.
Authorship effort on any submitted work must be accurately documented and properly cited. Artificial Intelligence tools like ChatGPT represents a new paradigm in academic and scholarly writing. Use of such tools on submitted work must be documented.