Research Methods

This goal of this course is to introduce and discuss concepts in research methodology, empirical analysis, and the scientific enterprise in computing. This course will prepare students for conducting research by examining how to plan, conduct, and report on empirical investigations. The course will cover techniques applicable to each of the steps of a research project, including formulating research questions, theory building, data analysis (using both qualitative and quantitative methods), building evidence, assessing validity, and publishing. The course will cover the principal research methods used to study human interaction with computer technology: controlled experiment, case studies, surveys, archival analysis, action research and ethnographies. We will also cover topics in peer review, ethical obligations involving human subjects research, how to give a scientific presentation, and how to write research papers, survey papers, and funding proposals.

Prerequisites

Enrolled as a Graduate Student in CSE or by instructor permission.

Logistics

Class Information

Lecture:
T/R 9:30am – 10:45am

232 Debartolo Hall

Instructor

Dr. Tim Weninger (tweninge@nd.edu)

Office Hours:
Tue 11:00am in 353 Fitzpatrick Hall
or by appointment

Teaching Assistants

None

Course Format and Activities

WeekDateTopicDiscussion LeadersPre-ReadingAssignments
101/11IntroductionNone
101/13Class Cancelled
201/18History and Philosophy of ScienceOkasha, Ch 1-3
201/20Critical Reading of ResearchChandrasekharan, Eshwar, et al. “You can’t stay here: The efficacy of reddit’s 2015 ban examined through hate speech.” CSCW (2017): 1-22.
301/25Peer ReviewBohannon, John. “Who’s afraid of peer review?.” Science. (2013): 60-65.

Tomkins, A., Zhang, M. and Heavlin, W.D., 2017. Reviewer bias in single-versus double-blind peer review. Proceedings of the National Academy of Sciences, 114(48), pp.12708-12713.

http://blog.mrtz.org/2014/12/15/the-nips-experiment.html
301/27How to Write a Peer Review Donahue, C., McAuley, J. and Puckette, M., 2018. Adversarial audio synthesis. ICLR. arXiv preprint arXiv:1802.04208.
402/01Morphology of a Paper and Technical WritingTim Weninger [PPT]Weekly Review: 
Gutierrez, J. and Schrum, J., 2020. Generative Adversarial Network Rooms in Generative Graph Grammar Dungeons for The Legend of ZeldaarXiv preprint arXiv:2001.05065.
Weekly Review Due
402/03LaTeX and BibTeXhttps://tobi.oetiker.ch/lshort/lshort.pdf Ch 1
502/08How to Write your ResearchWeekly Review:
Rotabi, R., Danescu-Niculescu-Mizil, C. and Kleinberg, J., 2017, April. Competition and selection among conventions. In Proceedings of the 26th International Conference on World Wide Web (pp. 1361-1370).
502/10How to Write a Survey
602/15How to Make a Research PresentationWeekly Review:
Gonzalez, J.E., Low, Y., Gu, H., Bickson, D. and Guestrin, C., 2012. Powergraph: Distributed graph-parallel computation on natural graphs. In USENIX Symposium on Operating Systems Design and Implementation (OSDI 12) (pp. 17-30).
602/17 How to Make a Research PresentationPPT1
PPT2
PPT3
702/22 Revising and Publishing ResearchLiterature Review Due
702/24Computing as a DisciplineJustin Dulay
Scheirer, W.J., Anthony, S.E., Nakayama, K. and Cox, D.D., 2014. Perceptual annotation: Measuring human vision to improve computer visionIEEE transactions on pattern analysis and machine intelligence36(8), pp.1679-1686.
803/01Research Funding and Proposal WritingSamuel Pasmann
Weekly Review:
Farmer, J. and Roy, S., 2020. A quasi-Monte Carlo solver for thermal radiation in participating media. Journal of Quantitative Spectroscopy and Radiative Transfer242, p.106753.

(Sections 1-4.1 and 5 only)
803/03 IRB, Ethics, and Research MalpracticeKat DearstyneComito, C., Forestiero, A. and Pizzuti, C., 2019. Bursty event detection in twitter streamsACM Transactions on Knowledge Discovery from Data (TKDD)13(4), pp.1-28.
903/08 Spring Break
903/10Spring Break
1003/15Basics of Research, Theory BuildingGuangyu Meng
Weekly Review:
Chen, Q., Zhao, B., Wang, H., Li, M., Liu, C., Zheng, Z., Yang, M. and Wang, J., 2021. SPANN: Highly-efficient Billion-scale Approximate Nearest Neighborhood SearchAdvances in Neural Information Processing Systems34.
1003/17Study DesignSimret Gebereegziabher
Khadpe, P., Krishna, R., Fei-Fei, L., Hancock, J.T. and Bernstein, M.S., 2020. Conceptual metaphors impact perceptions of human-ai collaborationProceedings of the ACM on Human-Computer Interaction4(CSCW2), pp.1-26.Introduction Due
1103/22Experiment Design, Controls, ConfoundersLouisa Conwill
Weekly Review:
Fribourg, R., Peillard, E. and Mcdonnell, R., 2021, October. Mirror, Mirror on My Phone: Investigating Dimensions of Self-Face Perception Induced by Augmented Reality Filters. In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) (pp. 470-478). IEEE.
1103/24Laboratory, quasi and natural experimentsYihong MaShen, L., Li, Z. and Kwok, J., 2020. Timeseries anomaly detection using temporal hierarchical one-class networkAdvances in Neural Information Processing Systems33, pp.13016-13026.
1203/29What do we mean when we we say that we know a thing?Deeksha Arun
Weekly Review:
Trokielewicz, M., Czajka, A. and Maciejewicz, P., 2018. Iris recognition after death. IEEE Transactions on Information Forensics and Security14(6), pp.1501-1514.
1203/31What do we mean when we we say that we know a thing? pt2 Brendan O’HandleyPicoreti, R., do Carmo, A.P., de Queiroz, F.M., Garcia, A.S., Vassallo, R.F. and Simeonidou, D., 2018, August. Multilevel observability in cloud orchestration. In 2018 IEEE DASC/PiCom/DataCom/CyberSciTech (pp. 776-784). IEEE.

https://towardsdatascience.com/lessons-from-how-to-lie-with-statistics-57060c0d2f19 
Research Design Due
1304/05Distributions and when statistics lieHaoran GangMishra, P., Lehmkuhl, R., Srinivasan, A., Zheng, W. and Popa, R.A., 2020. Delphi: A cryptographic inference service for neural networks. In 29th USENIX Security Symposium (USENIX Security 20) (pp. 2505-2522).
1304/07OLSAnnalisa Szymanski
Langevin, R., Lordon, R.J., Avrahami, T., Cowan, B.R., Hirsch, T. and Hsieh, G., 2021, May. Heuristic evaluation of conversational agents. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1-15).
1404/12My results are State of the Art, and other lies we tell ourselves.Kristina RadivojevicVosoughi, S., Roy, D. and Aral, S., 2018. The spread of true and false news onlineScience359(6380), pp.1146-1151.
1404/14Fantastic Statistics and Where to Find Them
1504/19Fantastic Statistics and Where to Find Them: Pt 2, the Crimes of ElsevierFinal Paper Due
1504/21Survey Bias
1604/26this page intentionally left blank
1604/28Reading Day
1705/03Finals Week Reviews Due
1705/05Finals Week

This course will draw materials from research literature as well as lessons accumulated over decades of experience in computing research. Students will attend weekly classes, complete frequent readings and reviews, and formulate a short research review article.

This term we will be using Canvas for class discussion. The system is highly catered to getting you help fast and efficiently from classmates and myself.

Lectures and Class Participation

Students should attend all classes. Effective lectures rely on students’ participation to raise questions and contribute in discussions. We will strive to maintain interactive class discussions if possible.

Questions, Discussions, and Help

If you have any questions or need clarification of class material, what should you do? First, try to post your question to the Canvas forum whenever possible, or otherwise email the instructor. The forum is for you and your peers to discuss class-related materials and to help one another. The forum will be monitored closely, but please be aware that we may not be able to answer all questions on the forum in a timely manner, due to the overwhelming number of questions that such a forum sometimes generates. Also, there are obviously things that are not appropriate for the forum, such as solutions for assignments as well as comments or requests to the staff.

In any case, for more thorough discussion, come to our office hours if you can!  Don’t be shy. Use our office hours to their fullest extent to help your study.

Requirements

Coursework

Most class meetings will require pre-reading selected by discussion leaders. Those readings will be discussed during class.

Each weekly reading will result in a short writeup.

Discussion leaders will give a talk at the beginning of each class. Discussion leaders for each week are exempt from the readings.

Signup here: first come first served.

Pre-Candidacy Proposal

A term paper is due at the end of the term with several milestones throughout the semester.

Grade Breakdown

Discussion Leaders10
Weekly Readings/Reviews20
Literature Review20
Introduction10
Research Design5
Final Paper30
Peer Review5

Grades

This table indicates minimum guaranteed grades. Under certain limited circumstances (e.g., an unreasonably hard exam), we may select more generous ranges or scale the scores to adjust.

Total Grade
90-100 A-, A
80-89 B-, B, B+
70-79 C-, C, C+
60-69 D

Polices

Textbooks

Textbooks are required, but generally very cheap or free.

Salganik, Matthew J. Bit by bit: Social research in the digital age. Princeton University Press, 2019.

Okasha, Samir. Philosophy of Science: Very Short Introduction. Oxford University Press, 2016.

Lectures

Students should attend all classes. Effective lectures rely on students’ participation to raise questions and contribute in discussions. We will strive to maintain interactive class discussions if possible.

Lecture capture and Zoom will not be provided.

Regrading

All requests to change grading of any course work must be submitted to the instructor in writing within one week of when the grades are made available. Requests must be specific and explain why you feel your work deserves additional credit. Do not ask for a regrade until you have studied and understood our sample solution.

Late Work

All scheduled due dates/times are US Eastern Time. Homework is typically due at the beginning of class on the due date, but check each the assignment for specifics.

Due date/time will be strictly enforced. Missing or late and/or unannotated work gets zero credit. If you are unable to complete an assignment due to illness or family emergency, we will understand but please see the instructor as soon as possible to make special arrangements. All such exceptional cases must be fully documented.

Academic Integrity

Notre Dame Students are expected to abide by Academic Code of Honor Pledge:

As a member of the Notre Dame community, I acknowledge that it is my responsibility to learn and abide by principles of intellectual honesty and academic integrity, and therefore I will not participate in or tolerate academic dishonesty.

All course work that you submit must be efforts of your own (if it is an individual assignment) or of your approved team (if it is a group assignment). Discussion of homework problems is encouraged, but writing solutions together or looking at other students’ solutions is not allowed. Much of the material in this class can be found online. You may look to online sources for guidance, but you must always cite your source(s).