Research Methods

This goal of this course is to introduce and discuss concepts in research methodology, empirical analysis, and the scientific enterprise in computing. This course will prepare students for conducting research by examining how to plan, conduct, and report on empirical investigations. The course will cover techniques applicable to each of the steps of a research project, including formulating research questions, theory building, data analysis (using both qualitative and quantitative methods), building evidence, assessing validity, and publishing. The course will cover the principal research methods used to study human interaction with computer technology: controlled experiment, case studies, surveys, archival analysis, action research and ethnographies. We will also cover topics in peer review, ethical obligations involving human subjects research, how to give a scientific presentation, and how to write research papers, survey papers, and funding proposals.

Prerequisites

Enrolled as a Graduate Student in CSE or by instructor permission.

Logistics

Course Format and Activities

Week	Date	Topic	Discussion Leaders	Pre-Reading	Assignments
1	01/17	Introduction		None
1	01/19	History and Philosophy of Science		Okasha, Ch 1-3
2	01/24	Critical Reading of Research		Chandrasekharan, Eshwar, et al. “You can’t stay here: The efficacy of reddit’s 2015 ban examined through hate speech.” CSCW (2017): 1-22.
2	01/26	Critical Reading of Research		Muchnik, L., Aral, S., & Taylor, S. J. (2013). Social influence bias: A randomized experiment. Science, 341(6146), 647-651.
3	01/31	Peer Review		Bohannon, John. “Who’s afraid of peer review?.” Science. (2013): 60-65. Tomkins, A., Zhang, M. and Heavlin, W.D., 2017. Reviewer bias in single-versus double-blind peer review. Proceedings of the National Academy of Sciences, 114(48), pp.12708-12713. http://blog.mrtz.org/2014/12/15/the-nips-experiment.html
3	02/02	How to Write a Peer Review		Donahue, C., McAuley, J. and Puckette, M., 2018. Adversarial audio synthesis. ICLR. arXiv preprint arXiv:1802.04208.
4	02/07	Morphology of a Paper and Technical Writing	Tim Weninger	Weekly Review: Li, M., Xu, R., Wang, S., Zhou, L., Lin, X., Zhu, C., … & Chang, S. F. (2022). Clip-event: Connecting text and images with event structures. CVPR (pp. 16420-16429).	Weekly Review Due
4	02/09	LaTeX and BibTeX
5	02/14	How to Write your Research		Weekly Review: Rotabi, R., Danescu-Niculescu-Mizil, C. and Kleinberg, J., 2017, April. Competition and selection among conventions. TheWebConf (pp. 1361-1370).
5	02/16	How to Write a Survey
6	02/21	How to Make a Research Presentation	Charles Chiang Meghan McClain	Weekly Review: Zhang, C., Song, D., Huang, C., Swami, A. and Chawla, N.V., 2019, July. Heterogeneous graph neural network. KDD (pp. 793-803).
6	02/23	How to Make a Research Presentation	Aydin Wells PPT1 PPT2 PPT3
7	02/28	Revising and Publishing Research	Zephan Enciso	Weekly Review: Ando, K., Ueyoshi, K., Orimo, K., Yonekawa, H., Sato, S., Nakahara, H., … & Motomura, M. (2017). BRein Memory: A Single-Chip Binary/Ternary Reconfigurable in-Memory Deep Neural Network Accelerator Achieving 1.4 TOPS at 0.6 W. IEEE Journal of Solid-State Circuits, 53(4), 983-994.	Literature Review Due
7	03/02	Computing as a Discipline	Brianna Wimer Ozioma Oguine
8	03/07	Research Funding and Proposal Writing		Weekly Review: House, B., Malkin, J., & Bilmes, J. (2009, April). The VoiceBot: a voice controlled robot arm. CHI (pp. 183-192).
8	03/09	IRB, Ethics, and Research Malpractice
9	03/14	Spring Break
9	03/16	Spring Break
10	03/21	Basics of Research, Theory Building	Prabina Sharma Rasel Ahmed Bhuiyan	Weekly Review: House, B., Malkin, J., & Bilmes, J. (2009, April). The VoiceBot: a voice controlled robot arm. CHI (pp. 183-192).
10	03/23	Study Design	Aarohi Srivastava		Introduction Due
11	03/28	Experiment Design, Controls, Confounders	Noah Ziems Annapurna Puttaswamy	Weekly Review: Tay, Y., Tran, V. Q., Dehghani, M., Ni, J., Bahri, D., Mehta, H., … & Metzler, D. (2022). Transformer memory as a differentiable search index. NeurIPS.
11	03/30	Laboratory, quasi and natural experiments
12	04/04	What do we mean when we we say that we know a thing?	Adnan Hoq Nandini Banerjee Hy Dang	Weekly Review: Jeong, S., Baek, J., Park, C., & Park, J. C. (2021). Unsupervised Document Expansion for Information Retrieval with Stochastic Text Generation. NAACL, 7.
12	04/06	What do we mean when we we say that we know a thing? pt2	Mohammed Latif Siddiq Kehan Guo Siyu Yang	https://towardsdatascience.com/lessons-from-how-to-lie-with-statistics-57060c0d2f19	Research Design Due
13	04/11	Distributions and when statistics lie	Aditya Joshi Mohammed Almutairi	Weekly Review: Jiang, J., Hu, Y., Li, X., Ouyang, W., Wang, Z., Fu, F., & Cui, B. (2022, August). Analyzing Online Transaction Networks with Network Motifs. SIGKDD (pp. 3098-3106).
13	04/13	Distributions and when statistics lie	Yunhao Xing Chaoran Chen
14	04/18	Class Cancelled
14	04/20	OLS	Ruiyang Qin Shang Ma	Weekly Review: Davis, M. G., Smith, E., Tudor, A., Sen, K., Siddiqi, I., & Iancu, C. (2020, October). Towards optimal topology aware quantum circuit synthesis. QCE (pp. 223-234). IEEE.
15	04/25	My results are State of the Art, and other lies we tell ourselves.	Tawfiq Chowdhury Shivani Vyas		Final Paper Due
15	04/27	How to Evaluate AI Systems
16	05/02	How to Evaluate AI Systems			Reviews Due
16	05/04	Reading Day
17	05/10	Final Exam		10:30am-12:30pm	Final Exam

This course will draw materials from research literature as well as lessons accumulated over decades of experience in computing research. Students will attend weekly classes, complete frequent readings and reviews, and formulate a short research review article.

This term we will be using Canvas for class discussion. The system is highly catered to getting you help fast and efficiently from classmates and myself.

Lectures and Class Participation

Students should attend all classes. Effective lectures rely on students’ participation to raise questions and contribute in discussions. We will strive to maintain interactive class discussions if possible.

Questions, Discussions, and Help

If you have any questions or need clarification of class material, what should you do? First, try to post your question to the Canvas forum whenever possible, or otherwise email the instructor. The forum is for you and your peers to discuss class-related materials and to help one another. The forum will be monitored closely, but please be aware that we may not be able to answer all questions on the forum in a timely manner, due to the overwhelming number of questions that such a forum sometimes generates. Also, there are obviously things that are not appropriate for the forum, such as solutions for assignments as well as comments or requests to the staff.

In any case, for more thorough discussion, come to our office hours if you can!  Don’t be shy. Use our office hours to their fullest extent to help your study.

Requirements

Polices

Late Work

All scheduled due dates/times are US Eastern Time. Homework is typically due at the beginning of class on the due date, but check each the assignment for specifics.

Due date/time will be strictly enforced. Missing or late and/or unannotated work gets zero credit. If you are unable to complete an assignment due to illness or family emergency, we will understand but please see the instructor as soon as possible to make special arrangements. All such exceptional cases must be fully documented.

Academic Integrity

Notre Dame Students are expected to abide by Academic Code of Honor Pledge:

As a member of the Notre Dame community, I acknowledge that it is my responsibility to learn and abide by principles of intellectual honesty and academic integrity, and therefore I will not participate in or tolerate academic dishonesty.

Authorship effort on any submitted work must be accurately documented and properly cited. Artificial Intelligence tools like ChatGPT represents a new paradigm in academic and scholarly writing. Use of such tools on submitted work must be documented.