1 of 67

Research Practicum in Artificial Intelligence

Overview

CS371W: Research Practicum in Artificial Intelligence

This course is designed to equip students with the essential skills and knowledge to conduct rigorous and impactful research in Artificial Intelligence (AI). As AI continues to evolve from a futuristic concept into a foundational technology that permeates every aspect of modern life, its role in advancing scientific discovery, innovation, and societal progress has become increasingly critical. Understanding the breadth and depth of AI and its potential to address complex challenges across various domains is vital for any aspiring researcher in this field.

Course Objectives

Comprehensive Survey of AI Domains: You will explore a broad spectrum of AI areas, delving into the latest advancements and understanding the implications of these developments in both theoretical and applied contexts.
Idea Development: The course will focus on nurturing the ability to generate innovative and compelling research ideas that resonate with and engage the broader research community, addressing both current and emerging challenges.
Methodological Innovation: You will be guided in designing cutting-edge methods that push the boundaries of existing knowledge, fostering breakthroughs and new perspectives in AI research.
Experimental Rigor: A key emphasis will be placed on conducting experiments with meticulous attention to detail, ensuring that the analyses not only capture the essence of the research question but also contribute to the broader understanding of AI's role and impact.
Effective Communication of Research: The ability to present research findings in a clear, persuasive, and intellectually stimulating manner is crucial. You will develop the skills necessary to communicate your work to both academic and general audiences, enhancing the reach and impact of their research.

Course Requirements

Collaborative Team Project: Engage in a group project that leverages collective expertise and promotes interdisciplinary collaboration, reflecting the collaborative nature of AI research.
Research Paper Writing: Work with your team to author a research paper that reflects the depth and originality of your collective insights, contributing to the broader AI research community.
Peer Review: Critically evaluate and provide constructive feedback on the work of your peers, fostering a collaborative learning environment and enhancing your own research acumen.
Public Presentations: Deliver presentations that showcase your research findings, demonstrating your ability to effectively communicate complex ideas and insights to diverse audiences.

This course satisfies the Continuing Communication Requirement.

Syllabus

Fall 2024

Information

Course

Prerequisites: CS325: Artificial Intelligence or CS334: Machine Learning
Webpage: https://emory.gitbook.io/ai-research-practicum
Class Hours: MW 2:30 AM - 3:45 PM
Class Location: MSC W303

Instructors

Jinho Choi
- Associate Professor of Computer Science, QTM, and Linguistics
- Office Hours: MW 4 PM - 5 PM @ WH 218
- Email: jinho.choi@emory.edu
Nayoung Choi
- Ph.D. Student in Computer Science and Informatics
- Office Hours: TuTh 10 AM - 11:30 AM @ WH 100
- Email: nayoung.choi@emory.edu

Notes

This course is conducted in person, and no virtual links or digital recordings will be provided.
It is recommended that you bring your laptop as there will be many in-class exercises.

Overall Grading

Criteria

Homework Assignments (10 total, including 6 writing and 1 peer-review assignments): 70%
Checkpoint Presentations (4+2 total): 9%
Team Project: 21%

Policies

Late submissions will be accepted up to one week after the original deadline. However, a grading penalty of 15% will be applied to the submitted work. Submissions made after the one-week grace period will not be accepted under any circumstances.
All work submitted in this course is expected to adhere to the Emory Honor Code. Any violations of the honor code will be reported to the Emory College Honor Council for appropriate action.
If you encounter significant personal circumstances such as health issues or family emergencies that affect your ability to perform in the course, you may request accommodations. Submit a letter from the Office for Undergraduate Education supporting documentation to facilitate appropriate adjustments to your coursework or deadlines.

Team Grading

For certain team assignments, you will be required to indicate the contribution percentage of each team member, which will directly impact the individual grades for the assignment.

Contribution

If your team of two members receives 4 out of 5 points for an assignment, for example, and you indicate that your contribution was 60% while your teammate's was 40%, the points will be distributed as follows:
This approach ensures that the grading reflects the effort and input of each team member, promoting fairness and accountability.

Consensus

Each team is required to submit a single, agreed-upon chart detailing the contribution percentages of all members for each team assignment. This means that you and your teammates must reach a consensus on the contribution rates before submitting your work.
Open communication and transparency are essential in this process. Disagreements should be resolved within the team, ensuring that the final submission reflects the true division of labor and contributions.

By adhering to these guidelines, you will not only produce a strong research paper but also develop key skills in teamwork and fair assessment of contributions.

Writing Assignments

Throughout the semester, each team will collaboratively write a full research paper, with each section of the paper developed as a separate homework assignment. This process will guide you through the various stages of research writing, ensuring that you develop a comprehensive and high-quality paper.

You and your teammates will be responsible for collectively drafting each section of the research paper. These sections will be assigned as individual assignments, allowing you to focus on specific aspects of the research process.
Effective collaboration is crucial. Regular communication and coordination within your team will help ensure that each section is coherent and contributes to the overall quality of the final paper.
All writing must be your team’s original work. You are expected to synthesize ideas, data, and arguments in your own words. This includes not only the main text but also any related work sections, tables, and figures.
Direct copying or close paraphrasing from other sources, including previously published papers or external materials, is strictly prohibited. Any such instance will be considered plagiarism and will be subject to academic penalties in accordance with the university's policies.

Grading for the writing assignments will be conducted according to the Team Grading policy.

Checkpoint Presentations

Presentations

Each team will present their intermediate work to the class 4 times throughout the semester, which are crucial for sharing your progress, receiving feedback, and refining your project.
Every member of your team will receive the same grade for each presentation, reflecting the collective effort and quality of the work presented.
Active participation and equal contribution from all team members are expected to ensure that your presentations are comprehensive and well-prepared.

Attendance

Attendance at all presentations, including those not presented by your team, is mandatory:

Attendance will be graded individually, separate from your team's presentation score. Your engagement during others' presentations is as critical as presenting your own work.
You are allowed to miss one out of six presentations without penalty. This absence will be automatically excused, and you will not lose any points for it.
Beyond the first excused absence, any additional missed presentations will result in a score of 0 points for those attendances. It’s important to manage your time and responsibilities to ensure consistent participation.

By adhering to these guidelines, you will contribute to a supportive and dynamic classroom environment while ensuring that your team’s work is accurately and fairly evaluated.

Peer Reviews

As part of the final stages of your team’s research paper, your work will undergo a peer review process. The final draft of your paper will be evaluated by three peers from other teams. Similarly, you will be responsible for reviewing three research papers written by students outside of your own team.

This exercise aims to help you learn and practice proper research paper review techniques. By critically evaluating the work of your peers, you will gain insights into what makes a strong research paper, including aspects such as clarity, originality, methodology, and overall impact.
The feedback you provide will play a crucial role in helping authors improve the quality of their research. Your constructive criticism and suggestions will enable them to refine their arguments, strengthen their methodologies, and enhance the overall coherence and impact of their work.

By engaging fully in the peer review process, you will not only contribute to the research development of your peers but also enhance your own skills in research evaluation and writing.

Team Project

Your team is required to maintain frequent meetings on a weekly basis. As a team, you will be expected to submit the following towards the end of the semester:

Oral Presentation
Poster Presentation
Final Draft of Your Research Paper

Presentations

Your team will be responsible for preparing and delivering both an oral presentation and a poster presentation. These presentations will showcase the results of your research, demonstrating the depth of your analysis and the significance of your findings.
Each team member must actively participate in both presentations. All team members will receive the same grade for the oral and poster presentations unless an individual fails to participate, in which case that member will receive a zero.
Attendance is mandatory for all presentations, including those presented by other teams. Active participation in these presentations is crucial for fostering a collaborative learning environment and for gaining insights from your peers' work.
If you fail to attend any of the presentations, you will receive 0 points for that presentation. This policy emphasizes the importance of your engagement in the entire course process, not just your own team's work.

Research Paper

Collaboratively write a complete research paper that meets the standards for publication as a preprint or in a peer-reviewed venue. Your paper should be a rigorous and original contribution to the field, showcasing the results of your team’s research efforts.
The final draft of your research paper will be graded according to the Team Grading policy.

Peer Ranking:

Team projects will be ranked by peers, which will be taken into account in the final evaluation of the projects, offering an additional perspective on the quality and impact of your work.
The grading for the peer ranking will also adhere to the Team Grading policy.

Policy on LLM-based Systems

Instructors will rigorously utilize large language model (LLM)-based systems, such as GPT or Claude, to review and evaluate your individual writings.
You are highly encouraged to leverage LLM-based systems to revise and review the content you have already written, enhancing its quality and coherence.
You are not allowed to rely on LLM-based systems to generate any content from scratch. For example, you cannot use an LLM to create an introduction by providing only a title or to write additional paragraphs that may plausibly follow a given paragraph.
Engaging LLM-based systems to generate fabricated data, experimental results, or analyses is strictly prohibited. Any violation of this policy will result in a referral to the Honor Council for further investigation.
While you are permitted to utilize LLM-based systems to discover relevant resources (e.g., references), it is your responsibility to ensure the accuracy and reliability of those resources.

Schedule

Fall 2024

Date

Topic

Assignment

(W): Writing assignments.

Discussions

During our first class, we will engage in a series of discussions aimed at clarifying key distinctions and concepts that are foundational to both academic and professional work in AI research. These discussions will cover the following topics:

Research vs. Coursework
Research vs. Engineering
Scientific Paper vs. Technical Report
Journal vs. Conference vs. Workshop
Individual Project vs. Team Project

Research vs. Coursework

Research

Nature and Purpose: Research is an open-ended process of inquiry aimed at generating new knowledge or deepening understanding in a specific area. Unlike coursework, which typically follows a structured curriculum with predefined learning outcomes, research involves exploring uncharted territories, asking new questions, and developing novel solutions or theories.
Skills and Approach: Research requires critical thinking, creativity, and independence. It demands that students engage deeply with their subject matter, often requiring them to design experiments, analyze data, and draw conclusions that advance the field. The process is iterative, with frequent revisiting of ideas and methodologies based on new findings or challenges.
Outcome: The goal of research is to contribute original insights to the academic or professional community, often culminating in publications such as journal articles, conference papers, or theses.

Coursework:

Nature and Purpose: Coursework is designed to impart specific knowledge and skills through a structured program of study. It typically includes lectures, assignments, exams, and projects, all of which are aimed at mastering the content of the course.
Skills and Approach: Coursework emphasizes learning established knowledge and techniques, with a focus on understanding and applying concepts in a controlled environment. It is usually more guided and directive, with clear expectations and criteria for success.
Outcome: The primary objective of coursework is to build a solid foundation in a particular subject, preparing students for further study, professional practice, or research.

Research vs. Engineering

Research

Objective: Research seeks to explore new ideas, generate hypotheses, and create theoretical frameworks. It is driven by curiosity and the desire to expand the boundaries of knowledge. Research often deals with abstract concepts and fundamental questions that may not have immediate practical applications.
Process: Research involves experimentation, data analysis, and theoretical modeling. It is often exploratory and may not lead to immediate or tangible outcomes. The focus is on understanding, discovery, and explanation.
Outcome: The primary output of research is knowledge, which can take the form of publications, theories, models, or prototypes. This knowledge may later be applied in practical settings, but its immediate purpose is to contribute to the body of academic or scientific understanding.

Engineering

Objective: Engineering focuses on applying existing knowledge to solve practical problems. It is concerned with designing, building, and optimizing systems, products, or processes to meet specific needs or goals.
Process: Engineering involves the application of scientific principles, mathematics, and technology to develop functional solutions. The process is often iterative, with an emphasis on design, testing, and refinement to ensure that the final product meets desired specifications.
Outcome: The primary output of engineering is a working system, product, or process that addresses a real-world problem. Engineering solutions are typically evaluated based on criteria such as efficiency, reliability, cost-effectiveness, and user satisfaction.

Scientific Paper vs. Technical Report

Scientific Paper

Purpose: A scientific paper is intended to communicate original research findings to the academic community. It undergoes a rigorous peer-review process to ensure the validity, originality, and significance of the research.
Structure: Scientific papers generally follow a standardized structure: Abstract, Introduction, Related Work, Approach, Experiments, Analysis, and Conclusion. Each section serves a specific purpose in detailing the research process and findings.
Audience: The primary audience for a scientific paper is other researchers and scholars in the field. The language and content are typically technical and assume a certain level of expertise from the reader.
Publication: Scientific papers are published in academic journals and are often considered a significant contribution to the field. They are cited in other scholarly works and are used to advance the academic discourse.

Technical Report

Purpose: A technical report is a document that describes the progress, methods, and outcomes of a technical or engineering project. It is often more detailed and practical than a scientific paper and may include comprehensive documentation of design processes, experiments, or implementation details.
Structure: Technical reports may vary in structure but often include sections such as Introduction, Background, Methodology, Results, Analysis, and Recommendations. The focus is on providing clear and thorough documentation of technical work.
Audience: The audience for a technical report may include engineers, project managers, clients, or other stakeholders who need a detailed understanding of the work. The language is typically more practical and accessible, focusing on the technical aspects rather than theoretical contributions.
Publication: Technical reports are usually internal documents, though they may be published by research institutions, government agencies, or companies. They are not typically peer-reviewed and may not be widely disseminated.

Journal vs. Conference vs. Workshop

Journal

Purpose: Journals are periodical publications that present in-depth research articles. They are the primary medium for disseminating comprehensive and significant research findings. Journals often focus on specific disciplines or subfields and are considered prestigious outlets for research.
Review Process: Articles submitted to journals undergo a rigorous peer-review process, which can be lengthy. The review ensures that the research is original, methodologically sound, and contributes to the field.
Impact: Journal publications are often considered more impactful and prestigious than conference papers. They contribute to a researcher’s reputation and are widely cited in academic work.

Conference

Purpose: Conferences are events where researchers present their latest work, often in the form of papers, posters, or presentations. They provide a platform for immediate feedback, networking, and collaboration with peers.
Review Process: Conference papers are usually peer-reviewed, but the process is often less rigorous than journal reviews. The emphasis is on timely dissemination of new ideas rather than in-depth analysis.
Impact: Conference papers are valuable for sharing preliminary findings or emerging ideas and for engaging with the research community. While they are generally considered less prestigious than journal articles, they are crucial for staying current in fast-evolving fields like AI.

Workshop

Purpose: Workshops are smaller, more focused events often held in conjunction with conferences. They provide a forum for discussing specific topics, emerging areas of research, or new methodologies. Workshops often encourage more interaction and discussion than conferences.
Review Process: Papers or presentations submitted to workshops may undergo a peer-review process, but the criteria are often more flexible. The focus is on fostering dialogue and exploring new ideas.
Impact: Workshops are valuable for networking and gaining feedback on early-stage research. They may not carry the same weight as conference or journal publications but are instrumental in shaping research directions and building collaborations.

Individual Project vs. Team Project

Individual Project

Scope and Responsibility: An individual project allows a student to take full ownership of a research problem, from conceptualization to execution and reporting. It provides an opportunity to develop deep expertise in a specific area and fosters independence and self-motivation.
Learning Outcomes: Through individual projects, students learn to manage their time, resources, and research process. They gain a sense of responsibility and accomplishment by completing a project on their own.
Challenges: The main challenge of an individual project is the lack of collaboration and peer support. Students must rely on their own skills and knowledge to overcome obstacles, which can be both rewarding and demanding.

Team Project

Scope and Responsibility: A team project involves collaboration among multiple students, allowing for the division of labor and the integration of diverse skills and perspectives. Team projects often tackle more complex or multidisciplinary problems that benefit from a collective approach.
Learning Outcomes: Team projects teach collaboration, communication, and leadership skills. Students learn to work together, resolve conflicts, and leverage each other’s strengths to achieve common goals.
Challenges: The primary challenges of team projects include coordinating efforts, managing different working styles, and ensuring equitable contribution. Effective teamwork requires strong communication and the ability to navigate interpersonal dynamics.

Speed Dating

During this class hour, you will have the opportunity to meet half of your classmates in a series of brief, focused conversations. The goal is to identify potential teammates whose research interests and skills complement your own. This is not about finding your exact clone, but rather discovering peers whose strengths and passions could contribute to a well-rounded and dynamic project team.

Be concise, be curious, and most importantly, be yourself. After this session, you will have two weeks to form a team for collaboration on a research project. This is your chance to make meaningful connections that could shape your semester-long project and perhaps even your future in AI research.

Exercise

You will participate in 10 rounds of speed dating, with each round lasting 5 minutes and a 1-minute transition time between rounds.
During each round:
- 2 minutes for you to present and answer questions
- 2 minutes for your partner to present and answer questions
- 1 minute for mutual questions and note-taking
After each round, find a new partner you have not interacted with yet.

Profiles

Fall 2024

The following section contains the profiles of your potential project partners, as submitted for the HW1: Speed Dating assignment.

Students

Faculty Interests

During this class hour, you will work in small teams to explore a chosen faculty member's expertise, review recent publications, and develop original research ideas. Through a combination of collaborative exploration and individual contribution, you will gain practical insight into the research process and deepen your understanding of specific AI subfields relevant to the chosen faculty members, ultimately preparing you for potential future research collaborations or projects in the field.

Exercise

Navigate to [People → Faculty Interests*] in Canvas and form a team of 2-3 members who share interest in the same faculty member.
Explore the selected professor's website to identify their primary research areas and current focus.
Review the professor's recent publications (last 2-3 years) to understand their ongoing research projects and latest contributions to the field.
Each group member should develop two potential research ideas that align with the professor's areas of expertise as well as Interest you personally (for potential collaboration).
Create a concise slide presentation summarizing the above activities (one per group), and submit it to [Assignments → Exercises → Faculty Interests] in Canvas.
Prepare to deliver a short (5-minute) presentation of your slides in class.

AI Faculty

Fall 2024

The following list presents faculty members actively engaged in AI research within the at Emory University.

Professors

Eugene Agichtein

Research Areas: Information Retrieval, Web Search, Conversational AI, Natural Language Processing
Publications: ,
Websites: ,
Joined Emory in Fall 2006

Yana Bromberg

Research Areas: Bioinformatics, Protein Structure and Function, Genome Variation Analysis, Personalized Medicine, Metagenomics
Publications: ,
Website:
Joined Emory in Spring 2023

Li Xiong

Research Areas: Privacy Enhancing Technology, AI for Health, Mobility Data
Publications: ,
Websites:
Joined Emory in Fall 2005

Associate Professors

Jinho Choi

Research Areas: Conversational AI, Natural Language Processing, Computational Linguistics
Joined Emory in Fall 2014

Joyce Ho

Research Areas: Data Mining, Healthcare Informatics
Joined Emory in Spring 2016

Chinmay Kulkarni

Research Areas: Human-AI interaction, Social Computing
Joined Emory in Fall 2022

Fei Liu

Research Areas: Large Language Models, Automatic Summarization, Generative AI, Natural Langauge Processing
Joined Emory in Fall 2022

Yolanda Rankin

Research Areas: Human-Computer Interaction, CS Education, Gaming
Website: TBA
Joined Emory in Fall 2023

Andreas Züfle

Research Areas: Spatial Databases, Data Mining, Geosimulation, Spatial Computing
Joined Emory in Fall 2022

Assistant Professors

Wei Jin

Research Areas: Data-Efficient Learning, Graph Neural Networks, Trustworthy AI
Joined Emory in Fall 2023

Joon-Seok Kim

Research Areas: Prescriptive Analytics, Spatio-temporal Databases, Data Mining, Agent Based Modeling
Joined Emory in Fall 2024

Kai Shu

Research Areas: Data Mining, Trustworthy AI, Social Computing, Misinformation
Joined Emory in Fall 2024

Shengpu Tang

Research Area: Reinforcement learning, Healthcare
Joined Emory in Fall 2024

Emily Wall

Research Areas: Visual Analytics, Information Visualization, Cognitive Bias, Decision Making
Joined Emory in Fall 2021

Kristin Williams

Research Areas: Human-Computer Interaction, Physical Computing
Joined Emory in Fall 2022

Carl Yang

Research Areas: Graph Data Mining, Knowledge Graphs, Federated Learning
Joined Emory in Fall 2020

Liang Zhao

Research Areas: Spatial Data Nining, Graph Neural Networks, Generative AI
Joined Emory in Fall 2020

Research Areas

Explore diverse research areas within the domain of Artificial Intelligence.

In this class, you will dive deep into specific AI research areas, gaining insights into current trends and state-of-the-art approaches. This hands-on approach allows you to:

Familiarize yourself with the current state of AI research
Develop skills in analyzing academic conferences and literature
Practice identifying promising research directions
Enhance your ability to synthesize and present complex information

Exercise

Navigate to [People → Research Areas*] in Canvas and form a team of 2-3 members who share interest in the same .
Share your findings from with your teammates.
Discuss and synthesize your individual findings by identifying common trends, approaches, and challenges across your research.
Select 2-3 compelling papers to highlight, and provide a brief overview and the key aspects of each paper.
Create a concise slide presentation summarizing the above activities (one per group), and submit it to [Assignments → Exercises → Research Areas] in Canvas.

AI Conferences

The following is a list of research areas along with their associated top-tier conferences from recent years. This list highlights key fields of study and the leading conferences where the most influential and cutting-edge research has been presented.

Artificial Intelligence

AAAI Conference on Artificial Intelligence (AAAI) : 2024, 2023, 2022
International Joint Conference on Artificial Intelligence (IJCAI) : 2024, 2023, 2022

Computer Vision

Conference on Computer Vision and Pattern Recognition (CVPR) : 2024, 2023, 2022
European Conference on Computer Vision (ECCV) : 2024, 2022
International Conference on Computer Vision (ICCV) : 2023, 2022

Data Mining and Management

ACM International Conference on Information and Knowledge Management (CIKM) : 2024, 2023, 2022
ACM International Conference on Knowledge Discovery and Data Mining (KDD) : 2024, 2023, 2022
IEEE International Conference on Data Mining (ICDM) : 2024, 2023, 2022

Human-Computer Interaction

ACM CHI Conference on Human Factors in Computing Systems (CHI) : 2024, 2023, 2022

Information Retrieval

ACM SIGIR International Conference on Research and Development in Information Retrieval (SIGIR) : 2024, 2023, 2022
ACM International Conference on Web Search and Data Mining (WSDM) : 2024, 2023, 2022
ACM Web Conference (WWW) : 2024, 2023, 2022

Machine Learning

International Conference on Machine Learning (ICML) : 2024, 2023, 2022
International Conference on Learning Representations (ICLR) : 2024, 2023, 2022
Conference on Neural Information Processing Systems (NeurIPS) : 2024, 2023, 2022

Natural Language Processing

Annual Meeting of the Association for Computational Linguistics (ACL) : 2024, 2023, 2022
Conference on Empirical Methods in Natural Language Processing (EMNLP) : 2024, 2023, 2022
Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) : 2024, 2022

Useful Links

Task Selection

Task selection and defining research objectives are critical steps in any research project, providing focus, direction, relevance, and a feasibility check. This section aims to develop essential skills for academic research, including collaboration, literature review, critical thinking, project planning, and communication. You will learn to narrow broad interests into manageable projects, identify gaps in current knowledge, and clearly articulate their research ideas.

Exercise

Navigate to [People → Task Selection*] in Canvas and form a team of 2-3 members who share interest in the same research area.
Find three papers closely related to your team's research project that can serve as baselines. For each paper, provide an overview of its key methods, results, and main findings.
For each paper, provide an overview of its key methods, results, and main findings.
Select a specific task for your team and describe the types of data and/or domains you plan to focus on.
Outline the primary research objectives of your project, which may include:
- Devising an innovative methodology that outperforms previous approaches on specific or relevant datasets.
- Constructing a novel dataset that facilitates task adaptation to new domains or languages.
- Conducting a comprehensive analysis of state-of-the-art models, focusing on comparative aspects to gain insights into their strengths and weaknesses.
- Developing a practical application employing existing resources to advance translational research.
Create a concise slide presentation summarizing the above activities (one per group), and submit it to [Assignments → Exercises → Task Selection] in Canvas.

The novelty of your research will be assessed based on these objectives. Ensure that your objectives are designed to address the challenges outlined in the motivation.

Introduction

This chapter provides guidance on selecting a research topic and task, as well as how to compose the Introduction section to effectively motivate the readers and offer an overview of your work.

Motivation

Describe the motivation behind your selected task.

Broad Impacts

If the selected task has been popular in recent years, highlight its prevalence and reference recent seminal works related to the study:

TASK has recently gained lots of interests because REASONS (CITATIONS).

You may want to make separate citations per reason if appropriate e.g., REASON 1 (CITATIONS), REASON 2 (CITATIONS)

If the selected task is new or has not been widely recognized in recent times, explain its significance and cite relevant works that provide evidence of its importance:

TASK is important because REASONS (CITATIONS), yet it has been underexplored.

What are the advantages and disadvantages of opting for novel tasks compared to popular ones? Should you ever choose tasks that are neither novel nor popular?

Intellectual Merit

Identify the challenges present in this task that you aim to enhance, and provide citations to previous works that have addressed these challenges:

Despite the great advancement, CHALLENGE is challenging because REASONS (CITATIONS).

The scope of references here should encompass a broader range than the ones cited in the section.

Concisely describe the major approaches that have been developed to tackle these challenges and cite the corresponding works:

Several stuides have proposed APPROACH to overcome this issue (CITATIONS).

It should serve as an overarching description of previous works that have employed similar approaches.

Explain the limitations associated with those approaches.:

However, it is still limited by REASONS.

If your work addresses multiple challenges, you may consider dedicating a paragraph to each challenge, where each paragraph follows all the steps outlined in this section.

Overview

Provide the overview of your project.

Approach

Elaborate on your approach and reference the section:

In this paper, we present APPROACH to improve CHALLENGE (Section #).

You may need to describe multiple approaches if your work serves multiple objectives.

Explain the hypothesis underlying your approach by specifying which methods are expected to address particular issues:

We hypothesize that METHOD can resolve ISSUE because REASONS.
Our approach uses METHOD to resolve ISSUE because REASONS.

What distinguishes "approach" from "method"?

How do we differentiate between "challenge" and "issue"?

Findings

Specify the types of data and/or models used to evaluate your approach, explain the reasons for using them, and reference the section for detailed information on this aspect:

Our approach is evaluated on DATA with MODELS (Section #).

You may not want to mention the exact names of the datasets or models, but instead refer to their types to emphasize the generalizability of your work.

Provide a high-level description of the primary findings from the analysis that offer supporting evidence for your hypothesis, and reference the section where a comprehensive breakdown of the results can be found:

Our experiments show that FINDINGS (Section #).

At this stage, you do not know the outcomes of the experiments. Thus, write your expected findings here.

Contributions

List specific contributions of your work:

This work makes THREE main contributions as follows:
We create a new dataset for TASK.
We introduce a new model for TASK.
MODEL achieves the state-state-of-the-art performance on TASK.
A real system is built for TASK.

Indicate the primary contribution of your work (if any):

To the best of our knowledge, this is the first work that does APPROACH for TASK.
This work will facilitate the development of ...

Exercise

Learn how to structure the Introduction section.

Find your team members.
Select 3-5 papers related to your task.
For each paper, excerpt writings in the introduction section for the following categories:
- Broad Impacts
- Intellectual Merit
- Approach
- Findings
- Contributions
Identify parts of the writings that do not fit into any of the categories above and propose appropriate categories for them.
Make slides summarizing the above analysis for a short presentation (one per group) and submit them to [Assignments → Exercises → Introduction] in Canvas.

Related Work

This chapter provides guidance for writing the related work section by conducting a comprehensive literature review and differentiating your work from existing research.

Writing an extensive related work section is important because:

It demonstrates your comprehension of the relevant field.
It provides supporting evidence for your hypothesis.
Reviewers appreciate relevant citations of their work.

Literature Review

Conduct a survey of previous work relevant to your research.

Categorization

It is common to survey the literature for the following three categories:

Task
Methodology
Data

These are general categories; feel free to make your own categorization as needed.

Task

Conduct a comprehensive survey of the most recent works (e.g., last 2-3 years) that show state-of-the-art results in your task. The survey should not be limited by a certain domain or dataset and should encompass a broad range of approaches and methodologies:

Begin by examining the latest research papers and proceed by exploring their Related Work and Experiment sections for additional insights.
Useful sites: Papers with Code, ACL Anthology.

If you are the first to address this task, explore recent works related to similar tasks.

Many previous studies have shown remarkable results on YOUR TASK.
Only a few works have tackled YOUR TASK.
Although YOUR TASK has been underexplored, several works have been done on similar tasks.

Conduct comparative studies among previous works to identify their strengths and weaknesses. Describe each work briefly in 1-2 sentences, highlighting its unique contributions and differences from other approaches:

CITATION was the first to adapt APPROACH to TASK.
CITATION presented APPROACH/MODEL that showed the state-of-the-art results on DATA.

Use /citet instead of /cite in LaTex, which allows you to indicate the authors without parentheses.

Describe the limitations of the previous works.

Despite the great work, these models have CHALLENGES.

Explain how your work is distinguished from theirs and can potentially overcome such challenges:

Our work is distinguished because REASONS that handle ISSUES better.

Avoid citing a preprint version (e.g., arXiv) of a paper if it has been published in a peer-reviewed venue; instead, cite the published venue to ensure credibility, as papers on arXiv are not peer-reviewed and may have limited credentials.

Methodology

If you intend to apply existing methods or techniques from other tasks to your own, conduct a survey of significant works that have employed such methodologies across various tasks:

YOUR METHOD has been sucessfully adapted to TASKS.

If you are the first to introduce this methodology, find papers using similar methods.

Provide a brief 1-2 line description of each work and elucidate how these methods have contributed to the improvement of their respective tasks.

CITATION used METHOD and signficantly improved ASPECTS of TASK.

Explain the reasons why these methods are likely to enhance specific aspects of your task:

Given the great success of METHOD, we beileve it can enhance ASPECTS of YOUR TASK.

Data

If you plan to adapt existing tasks or methods to a new domain or language, survey renowned works in that particular domain/language, irrespective of the tasks they address:

Many recent studies have focused on DATA.

Provide a concise 1-2 line description for each work and highlight the key challenges or contributions they encountered while dealing with data from the specified domain/language:

CITATION presented METHOD to tackle TASK on DATA and showed promising results.
CITATION tackeled TASK on DATA and found CHALLENGES.

Clarify the importance of applying these findings to your task in the new domain/language, emphasizing the potential benefits and insights that can be gained from tackling such a cross-domain or cross-language challenge:

Given the growing interest, many people will benefit if there is a robust model for YOUR TASK on DOMAIN/LANGUAGE.

Exercise

Conduct an extensive and thorough survey related to your task.

Assemble your team members.
Survey recent papers relevant to your task, categorizing them based on criteria such as task, methodology, data, etc.
For each paper, provide a concise overview of the main approach, highlighting its strengths and weaknesses.
Explain how your work is distinguished from the existing works, emphasizing the advantages and unique contributions of your approach.
Make slides summarizing the above analysis for a short presentation (one per group) and submit them to [Assignments → Exercises → 3.2. Exercise] in Canvas.

A practical way of finding relevant papers is to start with one recent paper, collect the state-of-the-art papers cited in the paper (usually found in the Experiment section), branch out to those cited papers, and repeat this process recursively.

Approach

This chapter guides you to write the approach section.

Content

Typically, it is better to write the approach section as abstract as possible so your methods become generalizable for many tasks. For example, even if you use as an encocder but your approach can take any transformer as an encoder, it is better to denote that your method uses a transformer as an encoder instead of BERT.

Algorithm Development

This chapter discusses how to develop new algorithms and write them in pseudocode.

Task

Your task is to design an algorithm that takes a post with the title and its comments with replies from a discussion forum (e.g., Reddit) and converts them into a multi-turn one-to-one dialogue.

Input

A post with the title (https://www.reddit.com/r/college/comments/v7h9rs):

How do you focus when you’re depressed?
I have so many assignments due, with exams coming up too. Life's just keeps hitting me recently and I'm finding it really hard to sit down and take information in. Writing is hard, listening and paying attention is hard. Even if I manage to listen or read none of the information stays in my head. Any help is very appreciated!!

Comments and replies:

Get up early everyday and use the library to study if you have one, idk if you're like me but as soon as I get home I'm kinda done for the day so it helps to stay somewhere where you can't really relax.
Thank you, I’ll give this a try tomorrow

What helps me is embracing when I'm feeling down and allowing myself to take a deserved break. Sometimes I confuse my depressive episodes with burnout and it's important to know your limits. The biggest pro tip to not be overwhelmed with so much to do all at once is doing something every day. Dedicating simply 30 minutes to an hour a day of intense studying goes a long way over time vs cramming at the end. If you're able to do more than 1 hour then great! But know that you don't have to do 6-7 intense studying hours a day to be successful. Be intentional with your time and work smarter vs harder. Your future self will thank you.
This is so nice to hear, and very helpful, thank you!

Hardest part is starting to study, once I have like 15 minutes into my study session that’s my only focus and just forget everything else.

Overview

Give a comparison overview of your algorithms with key features:

We introduce two algorithms for the reddit-to-dialogue generation: the baseline algorithm considers every sentence in the post an utterance of Speaker 1 and each comment an utterance of Speaker 2 (Section 3.1), whereas the advanced algorithm finds an appropriate span of sentences from the post to form an utterance for Speaker 1 and an appropriate span of any comment to form an utterance for Speaker 2 (Section 3.2).

Indicate the objective of your algorithm(s):

The main objective is to generate a multi-turn dialogue using a post, its comments, and replies that flows naturally in context.

Describe what the input and output data should be (possible with a figure) that are commonly applied to all algorithms:

All algorithms assume that the number of sentences in the input post is less than or equal to the number of comments. The generated dialogues involve two speakers where utterances of Speakers 1 and 2 are extracted from the post and comments, respectively.

Baseline

Objective

The title or each sentence in the post is considered an utterance of Speaker 1 (S1).
For each utterance $u_1$ of S1, find a comment that is the most relevant and make it the response to $u_1$ from Speaker 2 (S2).

Output

S1 : How do you focus when you’re depressed?
S2 : What helps me is embracing when I'm feeling down and allowing myself to take ...
S1 : I have so many assignments due, with exams coming up too.
S2 : Get up early everyday and use the library to study if you have one, ...
S1 : Life's just keeps hitting me recently and I'm finding it really hard to sit down and take information in.
S2 : Hardest part is starting to study, once I have like 15 minutes into my study session that’s my only focus and just forget everything else.

Algorithm

Illustrate the baseline algorithm in pseudocode. Create helper methods if they help the readability and/or generalizability of your algorithm.
Give a brief overview of the algorithm by explaining what each line of the code does.
Describe helper methods (if any) in detail.

Overview

Define the input:

Let $P = [p_1, .., p_n]$ be an input post where $p_i$ is the $i$ 'th sentence in $P$ , and $\mathbb{C} = \{C_1, .., C_m\}$ be a set of $P$ 's comments such that $C_j = [c_{j1}, .., c_{j\ell}]$ where $C_j$ is the $j$ 'th comment in $\mathbb{C}$ and $c_{jk}$ is the $k$ 'th sentence in $C_j$ .

Is the input correctly described according to the objective?

Initialize the output and auxiliary data structures:

Let $D$ be the list of utterances representing the output dialogue (L1) and $T$ be a set of segments created from $\mathbb{C}$ (L2).

Describe the loop:

The algorithm visits every sentence $p_0 \in P$ (L3) and appends it to $D$ (L4). It then finds the most-relevant segment $\hat{t} \in T$ (L5) and adds $\hat{t}$ to $D$ (L6). $T$ gets trimmed with $\hat{t}$ (L7).

Return the output:

Finally, it returns $D$ as the output (L8).

Helper Methods

Describe the $\textit{first}$ method:

The $\textit{first}$ method removes and returns the first sentence in $P$ .

Describe the $\textit{segment}$ method:

The $\textit{segment}$ method makes each comment a segment s.t. $\textit{segment}(\mathbb{C}) = \{C'_1, \ldots, C'm\}$ , where C'j = c_{j1}\,^\frown ...^\frown c_{j\ell} ( $^\frown$ : text concatenation).

Describe the $\textit{ranker}$ method:

The $\textit{ranker}$ method takes $D$ comprising all previous utterances and $p_i$ , then estimates the likelihood of $t$ being the next utterance.

How do you estimate such likelihoods?

Describe the $\textit{trim}$ method:

the $\textit{trim}$ method removes $\hat{t} = C'_j$ from $T$ such that $\textit{trim}(T, \hat{t}) = T \setminus {C'_j}$ .

Advanced (Exercise)

Objective

Any span of consecutive sentences is considered an utterance of S1.
For each utterance $u_1$ of S1, find a span of any consecutive sentences in comments that is the most relevant and make it the response to $u_1$ from S2.

Output

S1 : How do you focus when you’re depressed? I have so many assignments due, with exams coming up too.
S2 : What helps me is embracing when I'm feeling down and allowing myself to take a deserved break.
S1 : Life's just keeps hitting me recently and I'm finding it really hard to sit down and take information in.
S2 : Get up early everyday and use the library to study if you have one, idk if you're like me but as soon as I get home I'm kinda done for the day so it helps to stay somewhere where you can't really relax.
S1 : Writing is hard, listening and paying attention is hard.
S2 : Hardest part is starting to study, once I have like 15 minutes into my study session that’s my only focus and just forget everything else.
S1: Even if I manage to listen or read none of the information stays in my head. Any help is very appreciated!!
S2: Sometimes I confuse my depressive episodes with burnout and it's important to know your limits. The biggest pro tip to not be overwhelmed with so much to do all at once is doing something every day.

Model Design

Encoder Challenge

Given the input text $W = \{w_1, \ldots, w_n\}$ where $w_i$ is the $i$ 'th token in $W$ , a contextualized encoder (e.g., BERT) takes $W$ and generates an embedding $e_i \in \mathbb{R}^{1 \times d}$ for every token $w_i \in W$ using $w_i$ as well as its context. The challenge is that this encoder can take only up the $m$ -number of tokens such that it cannot handle any input where $n > m$ .

What are the ways to handle arbitrarily large input using a contextualized encoder?

Baseline

One popular method is called the "Sliding Window", which splits the input into multiple blocks of text, generates embeddings for each block separately, and merges them at the end.

Let W = W_1 \cup \cdots \cup W_k \where $W_h = \{w_{(h-1)m+1}, \ldots, w_{hm}\}$ if $hm < n$ ; otherwise, $W_h = \{w_{(h-1)m+1}, \ldots, w_{n}\}$ such that $km \leq n$ . Then, the encoder takes each $W_h$ and generates $E_h = \{e_{(h-1)m+1}, \ldots, e_{hm}\}$ for every token in $W_h$ . Finally, the embedding matrix $E \in \mathbb{R}^{n \times d}$ is created by sequentially stacking all embeddings in $W_{\forall h}$ .

What are the potential issues with this baseline method?

The baseline method does not have enough context to generate high-quality embeddings for tokens on the edge of each block.

Advanced (Exercise)

Modify the baseline method such that a block has overlapped tokens with its surrounding blocks (both front and back). Once all blocks are encoded, each overlapped token should have two embeddings. Create an average embedding of those two embeddings and make it the final embedding for the overlapped token.

Decoder Challenge

In a sequence-to-sequence model (aka, an encoder-decoder model), a decoder takes an embedding matrix $E \in \mathbb{R}^{m \times d}$ and predicts what token should come next. It is often the case that this embedding matrix is also bounded by a certain size, which becomes an issue when the size of the matrix becomes larger than $m$ (for the case above, $E \in \mathbb{R}^{n \times d}$ where $n > m$ ). One common method to handle this issue is to use an attention matrix for dimensionality reduction as follows:

The embedding matrix $E \in \mathbb{R}^{n \times d}$ is first transposed to $E^T \in \mathbb{R}^{d \times n}$ then multiplied by an attention matrix $A \in \mathbb{R}^{n \times m}$ such that $E^T \cdot A \rightarrow D \in \mathbb{R}^{d \times m}$ . Finally, the transpose of $D$ , that is $D^T \in \mathbb{R}^{m \times d}$ gets fed into the decoder.

Would the following method be equivalent to the above method?

An attention matrix $A \in \mathbb{R}^{m \times n}$ is multiplied by the embedding matrix $E \in \mathbb{R}^{n \times d}$ such that $A \cdot E \rightarrow D \in \mathbb{R}^{m \times d}$ . Finally, $D$ gets fed into the decoder.

Data Creation

When you create a dataset, the followings need to be clearly described:

Data collection (e.g., sources of the data).
Preprocessing if performed (e.g., scripts that you write, existing tools used).
Annotation scheme and guidelines if conducted with justification.
People involved in this process (e.g., annotators, survey subjects).
Quality of the created data (e.g., inter-annotator agreement).
Statistics and analysis of the original, preprocessed, annotated data.

Here are a few papers presenting new datasets:

Competence-Level Prediction and Resume & Job Description Matching Using Context-Aware Transformer Models, Li et al., EMNLP 2020 (see Section 3).
FriendsQA: Open-Domain Question Answering on TV Show Transcripts, Yang and Choi, SIGDIAL, 2019 (see Section 3).

Research Challenges

You will engage in a critical discussion about the challenges and progress of your ongoing projects. This activity aims to encourage reflection, problem-solving, and strategic thinking about your research directions.

Please discuss the following topics within your team for 15 minutes
Prepare a 5-minute presentation summarizing your discussion.
Submit your slides to [Assignments → Exercises → Research Challenges] in Canvas.

Topic/Task Changes

Describe your original topic/task and how it has evolved.
Explain the reasons for any changes in research direction.
Reflect on what you'd do differently when selecting future research topics.

Approach Feasibility

Evaluate the feasibility of your current approach to produce meaningful results by the semester's end.
Detail your methods, including data, algorithms, models, and computing resources.
If needed, propose revisions to make your project more feasible.

Impact and Contributions

Compare your original expected contributions with your current goals.
Based on your comprehensive literature survey, identify what additional work might be needed for potential publication in top-tier conferences.

Future Directions

Honestly assess your interest in extending this research beyond the semester.
If interested, outline potential ways to further advance the research.
Suggest entry points for newcomers interested in this topic.

Experiments

This chapter guides you how to write the experiments section

Datasets

This section describes datasets used for the experiments.

Data Selection

If there are datasets for your task that have been widely used by previous work, you need to test your models on them for fair comparisons. For each dataset, briefly describe the dataset:

For our experiments, DATASET is used (CITATION) to evaluate our MODEL.

If it is not widely used but appropriate to evaluate your approach, explain why it is selected:

For our MODEL, the DATASET is used (CITATION) because REASON(S).

If it is a new dataset that you create, briefly mention it and reference the section describing how the dataset is created (e.g., ):

All our models are evaluated on the EXISTING DATASET (CITATION) as well as YOUR DATASET (Section #).

It is always better to experiment with multiple datasets to validate the generalizability of your approach.

Data Split

Your work is not comparable to previous work unless it uses the same data split. Indicate which previous work you follow to split the training, development, and evaluation sets:

The same split as CITATION is used for our experiments.

During the development (including hyper-parameter tuning), your model should be trained on the training set and tested on the development set (aka. validation set).

While you are training, you should frequently check the performance of your model (usually once every epoch) and save the model that gives the highest performance on the development set.

Once the training is done, your best model (on the development set) is tested on the evaluation set, and the results are reported in the paper.

If a new dataset is used, you need to create your own split. If the dataset has:

[1K, 10K] instances, use the 75/10/15 split for the training/development/evaluation sets.
> 10K instances, use the 80/10/10 split.

It is important to create training, development, and evaluation sets that follow similar distributions (in terms of labels, etc.) as the entire data.

Cross-validation is typically used for development, and for evaluation. However, when the data is not sufficiently large, the evaluation set becomes too small, which can cause many variants in the model performance tested on the set. Thus, cross-validation is used to average out the variants.

Once you split the data, create a table describing the distribution and statistics of each set. This table should include all necessary statistics to help researchers understand your experimental results (e.g., how many entities per sentence for named entity recognition, how many questions per document for question answering).

If it is unclear how the data is split from the previous work, contact the authors. If the authors do not respond, create your own split and describe the datasets.

Models

This section describes models used for comparative study.

Descriptions

Describe existing models or frameworks commonly adopted by your models (if any):

All our models adopt MODEL (CITATION) as the encoder.

List all models used for your experiments. Give a brief description of each model by referencing specific sections explaining the core methods used by the model:

The following three models are experimented:
BASELINE: DESCRIPTION (Section #)
ADVANCED: BASELINE + METHOD (Section #)
BEST: ADVANCED + METHOD (Section #)

It is important to design models in a way that clearly shows key differences in methods.

Evaluation Metrics

Because a neural model produces a different result every time trained, you need to train it 3 ~ 5 times and report its average score with the standard deviation ().

Why would a neural model produce a different result every time it is trained?

Thus, indicate how many times each model is trained and what is used as the evaluation metric(s):

Every model is trained 3 times and its average F1-score and the standard deviation is used as the evaluation metric.

If you experiment on datasets used in previous work, you must use the same evaluation metrics for fair comparisons. Even if you present a new metric, you still need to evaluate both the old and new metrics to show the advantage of your new metric.

If you use a non-standard metric that has not been used in previous work because:

The task is new,
The new aspect introduced for this task has never been tested before,
You find a better way of evaluating this, which has not been used in the previous work

explain why you cannot apply standard metrics to evaluate this task and describe the new metric:

Since TASK has not been evaluated on ASPECT(S) in prevoius work, we introduce new metrics ...

If your new evaluation metric is novel that requires bigger attention, explain it in the approach section so it would be considered one of the main contributions.

Experimental Settings

Describe hyper-parameters used to build the models (e.g., epoch, learning rate, hidden layer, optimizer, batch size):

MODEL is trained for # epochs using the learning rate of FLOAT, ...

Explain anything special that you do for training:

Early stop is adopted to control the number of epochs if the score on the development set does not improve over two epochs.

Describe computing devices used for the experiments:

Our experiments use NVIDIA Titan RTX GPUs, which takes 10/20/30 hours for training the BASELINE/ADVANCED/BEST MODELS, respectively.

It is important to describe the experimental settings for replication, although they are often put in the appendix due to the page limit for the actual paper submission.

Development

If you observe enhanced training efficiency (e.g., your new loss function requires a fewer number of epochs to train), create a figure (e.g., x-axis: epochs, y-axis: accuracy) describing the training processes of the baseline and the enhanced models.

Our ENHANCED MODEL reaches the same accuracy (or higher) than the BASELINE model after only a third of epochs.

If you experience unusual phenomena during training (e.g., results on the development set are unstable), describe the phenomena and analyze why they are happening:

Results

This section presents experimental results.

Tables

Create a table displaying experimental results from your on each and evaluation metric. The table should also include results from previous work directly comparable to yours.

If the table is too large (e.g., taking more than 1/3 of the page), it may overwhelm the readers. In this case, shrink it by including only the critical results and put the rest in the appendix.

Here are a few tips for creating the result table:

Expand it to the full page if it consists of many columns.
Use acronyms for the header titles if too long, and explain them in the caption.
Highlight the key results by making them bold.

Sometimes, it makes more sense to use multiple tables to present your results (e.g., working on multiple tasks), in which case, use a consistent scheme across the tables so they can be easily compared.

Interpretations

Once the result table is presented, you need to give an interpretation of the results. First, summarize the overall observations:

Each model shows an incremental improvement over its predecessor.
MODEL 2 shows a noticeable improvement over MODEL 1, indicating the effectiveness of our METHOD.
The ADVANCED MODEL shows a significant improvement of #.#% from the BASELINE MODEL.

Then, describe any key findings:

It is interesting that MODEL 2 shows better performance over MODEL 1 on DATASET 1 but the results are opposite on DATASET 2.

It is likely because METHOD works well for ASPECTS in DATASET 1, but not necessairly for ASPECTS in DATASET 2 (Section #.#).

In general, high-level interpretations are provided in the Experiments section whereas more detailed analyses are provided in the Analysis section. These two sections, however, can be merged into one if the space is limited.

Finally, explain any additional results that are not included in the table but help readers interpret this work better:

It it worth mentioning that we also experimented with METHOD 1, which showed a similar result as METHOD 2.

The interpretation should not be simply reading the table. The main goal of this interpretation is to provide insights that are not so obvious to the readers by reading the table, but you learn from the period of this study.

5.4. Homework

Individual Writing

Write the Experiments section in your individual overleaf project.
Recommended length: 100 - 200 lines (including tables and figures).
Submit the PDF version of your current draft up to the Experiments section.

Rubric

Data Section: are the sources and choices of the datasets reasonably explained? (1 point)
Data Split: are the statistics of training/development/evaluation or cross-validation sets distinctly illustrated? (2 points)
Model Descriptions: are the models designed to soundingly distinguish differences in methods? (2 points)
Evaluation Metrics: are the evaluation metrics clearly explained? (2 points)
Experimental Settings: are the settings described in a way that readers can replicate the experiments? (1 point)
Model Development: is the model development progress depicted? (1 point)
Result Tables: are the experimental results evidently summarized in tables? (2 points)
Result Interpretations: are the key findings from the results convincingly interpreted? (2 points)

Analysis

This chapter guides you how to write the analysis section

This section aims to provide an in-depth look at what your models can and cannot do so that researchers can make informed decisions in the following step and the best practice in applications.

Performance Analysis

This section gives a detailed analysis in performance.

Once you have shown the overall performance of your models in the section, you need to give a more detailed analysis.

You should consider any quantitative analysis not involving manual annotation in this section that can emphasize the significance of your novel approach. Qualitative analysis is provided in the section.

By Labels

If your task involves multi-class classifiction, it is good to show how your model performs for each label. Label distributions play an important role in this analysis because most machine learning algorithms tend not to perform well for labels with small distributions. Once you present the table, explain why your model does not perform well on certain labels.

Although the above example shows results from one model, it is better to show results from multiple models in comparisons.

Label analysis can be presented by a confusion matrix if certain pairs of labels are getting confused more often than the others. Once you present the matrix, explain why certain labels are confused more than the others.

By Categories

If your data can be categorized into meaningful groups (e.g., challenging input, complex output), then provide the model performance on each category and explain why your models perform well (or not well) on certain groups.

Speed Analysis

If the model speed is one of your contributions, group data with respect to different sizes and explain for which groups your model shows strength. The model speed should be adequately profiled by making sure it is not interrupted by other processes.

You should measure the speed multiple times, remove outliers, and present the average speeds to ensure the robustness.

If the model speed can be measured theoratically by counting involved operations, show the plots and explain the overal trend.

Error Analysis

This section provides a qualitative error analysis.

It is important to understand what errors are produced by your model, how often those errors occur on which occasions, and what causes such errors. Error analysis should be done both quantitatively and qualitatively.

Sample instances that your model makes errors for. Focus on data portions that your performance analysis has questions about.
Categorize errors by analyzing them (manually) and pick signature examples.
Provide distributions of the categorized errors in a figure (see below).
Explain why the model makes certian errors with examples.

Discussions

This section discusses limitations and the future direction of this research.

Limitations

Once you complete the analysis, you know what your approach can or cannot do for the task. The limitations can be thought of in multiple dimensions:

Technicality: what are the technical challenges for enhancing the performance in terms of accuracy or scalability?
Robustness: how reliable and generalizable are your results?
Applicability: what hinders your approach from being adapted by practical applications?

Future Directions

As the owner of this work, you are responsible for suggesting future research directions to the community. Explain what researchers can do to overcome the limitations.

6.4. Homework

Individual Writing

Write the Analysis section in your individual overleaf project.
Recommended length: 100 - 200 lines (including tables and figures).
Submit the PDF version of your current draft up to the Analysis section.

Rubric

Performance Analysis: Is the model performance analyzed in-depth (e.g., by labels and categories)? Is the efficiency of the model illustrated (if applicable)? (5 points)
Error Analysis: Are the errors meaningfully categorized, interpreted, and illustrated with examples? (3 points)
Discussions: Are the limitations and future directions clearly described? (2 points)

Conclusion & Abstract

This section guides you how to write the conclusion and the abstract sections.

Conclusion

Summary

The summary in the conclusion should be distinguished from the and or the in the introduction in the sense that it should be conclusive.

First, indicate the overall contribution of your system:

We present YOUR SYSTEM that improves ... YOUR TASK.

Then, conclude the key contributions of specific methods:

Our METHOD shows the state-of-the-art accuracy on ASPECTS.
Our METHOD is found to be more effective in ASPECTS.

If the impact of your work reaches the point that it can make a significant contribution to the field, describe its broader impacts:

Our model allows FIELD to OVERCOME EXISTING CHALLENGES.

Finally, mention how you plan to release relevant resources (if possible):

All our resources including the dataset, models, and source codes are available through our open source project in URL.

Future Work

The future work in the conclusion should be more explicit than the future directions of in the analysis in the sense that it should specify your next steps of this research.

First, indicate the key challenges that you would address as the next steps:

Despite great peformance, our model still lack in CHALLENGES.

Then, illustrate your plan to handle such challenges:

We plan to tackle CHALLENGES using METHODS.

Title & Abstract

Title

It is a common practice to have the main title (with catchy words) describing the overall objective and the secondary title (with specific terms) indicating the key methods.

Abstract

The abstract is the face of the paper that will get readers interested in your work. Make sure that the writing in the abstract does not seem redundant to the one in the introduction although their contents may overlap.

The abstract is a standalone section such that it should be comprehensive on its own. Also, the introduction should be not a continuation of the abstract; in other words, the introduction should not assume that the reader already read the abstract.

Describe the importance and/or progress of your task:

YOUR TASK has been heavily explored due to its IMPORTANCE.
APPROACH has achieved remarkable results on YOUR TASK.

Indicate the main challenges you address in this work:

However, CHALLENGES still have not been resolved.

Introduce the main gist of your work

This paper presents YOUR APPROACH that uses METHODS.

Describe how your methods can overcome the challenges:

METHOD 1 and METHOD 2 are used to overcome CHALLENGE 1.
METHOD 3 are used to overcome CHALLENGE 2.

Group methods addressing the same challenge and give more specifics if needed.

Illustrate the experimental design:

Our models are experimented on DATASETS.

Summarize the key findings:

Our best MODEL achieves the state-of-the-art accuracy of ##.#% and shows strengths in ASPECTS.

Emphasize the main contribution of the paper:

To the best of our knowledge, it is the first work to ...
Our resource is useful for ...

Peer Review

Presentations

Two types of presentations are often given at conferences, oral and poster.

Oral Presentation

The oral presentation is provided in a lecture setting where you present slides to a large audience for 10 - 15 minutes followed by questions. Although your interaction with the audience is limited, it gives you an opportunity to appeal your work to the research community and sometimes leads them to contact you later for more discussions.

Cover Page

The cover page must include the followings:

Title (in a large font)
Venue (e.g., CS385: Research Practicum - Emory University)
Location (e.g., Atlanta, GA, USA) and date
Authors and their affiliations (e.g., Jinho D. Choi - Department of Computer Science, Emory University)

Team Projects

Fall 2024

Projects

Learning LESS: Logical Equivalence for Structured Queries in Text-to-SQL Models : Darren Ni, Sherry Rui, Jonathan Zhang (promo)
Enhancing Information Retrieval with a Hybrid Model Combining Sparse and Dense Retrievers : Kei Nie, Daisy Wang, Rex Zhang (promo)
Predicting Effects of ncSNVs : Zechary Chou, Brian Hakam, Kihoon Kang (promo)
Identifying Hispanic Speech by Linguistic Features with Transformer Models : Ivan Martinez-Kay, Raphael Palacio (promo)
Analysis on Optimal Time Span of Trading Data for Artificial Intelligence Development : Olivia Kim, Thalia Papageorgiou (promo)
How Polluted Exemplar Undermines Large Language Models’ Numerical Prowess : Andrew Chung, William Hao (promo)
Multimodal Deception Detection in Presidential Debates: Promoting Accountability and Informed Citizenship : Yutong Hu, Ryan Meng, Harutoshi Okumura (promo)

Fall 2023

Projects

Unlocking Market Mysteries: Pushing GPT to its Limit on Stock Market Prediction
: Adedoyin Aromolaran, Harry He, Harry Jeon (oral, poster, paper, comments)
The Political Climate: Speech Identification and the Discourse of Environment
: Louis Mullarkey, Latifa Tan (oral, poster, paper, comments)
Discerning Distress: A Framework for Crisis Identification in Digital Dialogue using LLMs
: Darin Kishore, Ellie Paek (oral, poster, paper, comments)
Back to the Basics: Evaluating Information Retrieval and Generative Models for Chest X-Ray Synthesis: Alex Belov, Karam Khanna (oral, poster, paper, comments)
Evaluating OpenAI’s Approach to Diversifying DALL-E 3’s Image Generation
: Gianluis Hernandez, Fred Hinojosa (oral, poster, paper, comments)

Fall 2022

Projects

Exploring Users’ Customization and Personalization Preferences for Conversational AI
: Catherine Baker and Tom Gu (paper, slides, poster).
ECGNet: A Generative Adversarial Network (GAN) Approach to the Synthesis of 12-lead ECG Signals from Single Lead Inputs
: Max Bagga, Alex Issokson, and Harry Jeon (paper, slides, poster).
What is Your Favorite Gender, MLM?: Evaluating Gender Bias in Multilingual Masked Language Models : Jacob Choi, Seong Kim, and Jeongrok Yu (paper, slides, poster).
HPC for ML for HPC: Exploration on Possibility of Improving High Performance Computing (HPC) using HPC-Aided-Machine Learning (ML) : James Song and Richard Yang (paper, slides, poster)
Diarize from Text: Text-based Speaker Diarization with Adaptive Metrics : Cheng Gong and Peilin Wu (paper, slides, poster)
Improving Narrative Relationship Embeddings by Training with Additional Inverse-Relationship Constraints : Mikolaj Figurski (paper, slides, poster)
ScholasticQA: Evaluating the Text Completion Capabilities of Pretrained Language Models : JP Selame Fernandez and Anthony Wang (paper, slides, poster)

Comments

1. Baker and Gu (2)

I picked project 1 because the emoji part of the research gave an interesting result which disapprove some part of the hypothesis. Both parts of the research would be much better if they extend to a bigger scale.
I am fascinated by the authors' dedication to designing the chat box. The huge amount of work put into the chat box is crazy. I also firmly believed in the use of this project. It not only has research usage, but also can be use in business area. I believe this is very important in computer science research since a lot of computer sicence research can be realized in to business. Based on my personal experience with Siri, the results of this research are proven.

2. Bagga, Issokson, Jeon (6)

I selected this group because their overall research methodologies were sound. They had an advantage because they had already brought in existing research, but even though they were not able to achieve their overarching goal of predicting different diseases using these leads, they followed each step research step thoroughly.
Compared to other projects, their projects are more completed.
Although I am not a biology major, the work is simply incredible for the gan model. As a project manager who manages data, I am fascinated by the ways they show their data. I learnt a lot from them on ways to clean data show data and select data from their research experience.
The area of research sounds very beneficial. The oral presentation seemed that they are very well prepared, and the purpose of the project was explained very well to me during the poster presentation.
Very complete project, with large dataset pulled together to support. Good to see critical analysis of the evaluation step even
Has real life applications and remarkable results

3. Choi, Kim, Yu (5)

I am choosing this project secondly because a I find it interesting. The findings are not what I would expect and also highlight a chance for improvement in MLM. From their presentation and poster, it sounds like the authors found an area in MLM that has largely been ignored/not properly developed and conducted experiments that identified that problem, proposed two solutions, and then found the resulting best solution. They also incorporated multiple languages into their research which is impressive. While some other projects like could likely have a bigger impact on the research community due to it's involvement in chronic disease and health, this work, I think, was a better overall project because of their research and proposed solutions. I think this work will end up holding more weight within the research community than the other papers because these three actually solved a real problem in MLM despite the time and resource constraints.
I think this project is a well constructed project with interesting result. This research has lots of meanings in terms of the "must do" of basically all NLP researches in the future. Also, the general structure of the research is clear whether in the presentation or on the poster.
Really enjoyed listening to both the slides and poster presentation for this project as the relevance of the contributions was very clear to me—current gender bias metrics are not robust, and it's important to correct these biases for masked language modeling as the applications are widespread (e.g., Google search, YouTube recommendations, autocomplete, etc.). It was also interesting that the work was tested across four Indo-European languages (English, Spanish, Portuguese, German), and I would be curious to see how the results turn out for Asian languages such as Mandarin or Korean—they may be significantly different due to cultural variations and the corpus that these masked language models are trained on.
Very complete project with tons of implicit extensibility that the authors are very aware of and solid basis for the method being applied.
Was easy to understand and novel

4. Song and Yang (1)

I think this project is good because the clearly organized experiment and model design. Although they did not get the desired result that will reflect a sigmoid curve, the future improvement of this project is also clear. In addition, AI + HPC as a research direction is very potential.

5. Gong and Wu (9)

I selected this project because I thought their design and methods were very complex and interesting. I think takings risks should be rewarded and because I felt their topic and research was quite advanced and successful, I think they had some of the highest quality.
I picked this project because it provides a novel direction for the existing evaluation matrix. It can be used to improve speech to text into another level!
I believe this group was very sound because they also followed research methodologies soundly. They were able to create their own algorithms and work on a legitimate NLP task that others had not done. Their steps are clear to follow, and they are able to achieve results and move forward in an area that is untouched.
Explanation makes me understand well about their project
For this project, I think the model that created was very informative. They compared the Amazon with another product, and it did generate useful results. The paper is very novel as well.
I could clearly see that they have worked really hard to come this far. Their poster was the best one in the class!
I thought that this project was of high quality because the findings were conveyed in a clear, concise manner. I could easily understand why this topic is important and why their specific contributions with developing a new metric (other than the word error rate) is pivotal, because the big-picture was well-highlighted through both the presentation slides and poster. I also thought that the algorithm described was very innovative and I wonder what the next steps are from here for this group—one option might be working on multiparty speaker diarization, since their data is currently only focused on having two parties right now.
The purpose of the project was explained very well to me. I felt this is an interesting project.
I selected this project because the novelty of creating the TDER evaluation metric, along with implementing the Needleman-Wunsch algorithm and their own 3-d matrix was very unique. I also believe that it was the closest project to being fully completed especially since they achieved results and performances above the other metrics. Furthermore, I believe that it was also presented very concisely when compared to other presentations which also contributes to its rankings.

6. Figurski (3)

I selected this project because the model and algorithm this work used was really cool and different. I give special credit to him for doing as much, if not more, work than everyone else for just one person. Both his powerpoint and poster presentation had extremely high quality and delivered.
First of all, the author did the whole paper himself. This is very impressing. His model about assumption was impression. I was fascinated by his result.
I chose this group because their quality is good, given this is a single-person group.

7. Fernandez and Wang (4)

The authors did great work. Their presentation was clean and clear and their poster was very well-made. Furthermore their work has the promise to give major contributions to the field of NLP's and language models. It also highlights interesting contrasts between the performance of the different forms of BERT and I'd love to see their data compared with high school seniors scores on the GER. Simply put, their project has the credibility of novelty and aiding the NLP field incredibly while also being interesting.
They have shown the most progress throughout the semester, and they had a clear, concise oral presentation.
I chose this group due to their research values. They created a dataset for sentence completion tasks, and they evaluated each popular pre-trained model on their dataset, not only giving the quality of their dataset but also evaluating those models' ability in sentence completion tasks.
I selected this project because I believe that their methodology is very sound and that with a larger data set, their methods can produce novel results. I also believe that their work could advance the NLP field (I am no expert though so take this with a grain of salt) with a substantial result because it could hint at the weaknesses of transformer models at performing tasks and thus perhaps hint at what exactly the transformer models capture in their predictions. I also believe that during the poster presentations, they did the best job explaining how their project worked.

Assignments

HW1: Speed Dating

This assignment aims to prepare you for the upcoming Speed Dating session, where you will seek compatible project partners for the course. By completing this preparation, you will:

Articulate your research interests and skills concisely
Develop preliminary project ideas
Identify your work preferences and availability

This groundwork will enable you to:

Engage in focused, productive conversations during the session
Quickly assess compatibility with potential teammates
Make informed decisions about project partnerships

Tasks

Research Interests

Prepare a concise (1-2 minute) verbal summary of your primary AI research interests.
Identify 2-3 specific AI subfields or application areas you are most passionate about.

Skills and Experience

List your technical skills relevant to AI (e.g., PyTorch, Huggingface, OpenAI).
Briefly outline any previous AI projects you have experienced.

Project Ideas

Come up with 1-2 potential project ideas within your areas of interest.
Be ready to explain these ideas succinctly, including the problem to be solved and potential approaches.

Availability and Work Style

Be prepared to discuss your typical availability for team work.
Consider your preferred work style (e.g., structured vs. flexible, in-person vs. remote collaboration).

Questions for Potential Partners

Prepare 2-3 questions to ask potential teammates to assess compatibility.

Submission

Compile your responses to all the above sections into a concise, well-organized PDF document.
Ensure your name is clearly visible at the top of the document.
Keep the total length to 1-2 pages.
Use clear headings for each section (e.g., Research Interests, Skills and Experience).
Submit your PDF to HW1: Speed Dating on Canvas.
Note that your submitted profile will be publicly visible on the Profiles page. This visibility is intended to facilitate the speed dating process and help in forming project groups.

Rubric

Research Interests (0.2 points): Are the research interests clearly defined and relevant to the field of AI?
Skills and Experience (0.2 points): Are the necessary skills and relevant experience effectively demonstrated?
Project Ideas (0.2 points): Are the proposed project ideas innovative and feasible within the scope of this course?
Availability and Work Style (0.2 points): Are the availability and the work style well aligned with a collaborative approach?
Questions for Potential Partners (0.2 points): Are the questions insightful in assessing potential collaboration and project goals?

HW2: Research Areas

This assignment aims to deepen your understanding of a specific AI research area by:

Analyzing recent conference proceedings
Identifying current trends
Exploring state-of-the-art approaches

This preparation will enable you to contribute meaningfully to discussions and collaborative activities related to the in-class exercise in the Research Areas section.

Tasks

Choose one research area from the AI Conferences page.
Explore at least three top-tier conferences relevant to your chosen area.
Analyze the proceedings from the most recent editions of these conferences (past 3 years), focusing on keynote speeches, best paper awards, workshops and tutorials, and top-cited papers (if available).
Identify and summarize 3-5 major trends or themes in current research, 2-3 state-of-the-art approaches or methodologies, and any significant challenges or open problems highlighted.
Select one specific paper that you find particularly interesting or innovative. Provide a brief summary of, the problem it addresses, its methodology, and key results and implications.

Submission

Prepare a comprehensive report (1000-1500 words) that includes:
- Your analysis of recent conference proceedings (Task 3)
- Summary of major trends, state-of-the-art approaches, and challenges (Task 4)
- Detailed review of one selected paper (Task 5)
- A bibliography listing all conference proceedings and papers referenced in your report.
Save your report in a PDF file, and submit it to HW2: Research Areas on Canvas.

Rubric

Task 3: Conference Analysis (1 points): Does the analysis effectively cover the key aspects of the selected conferences?
Task 4: Trends and Methods (1 points): Are the major trends, approaches, and challenges clearly identified and accurately summarized?
Task 5: Paper Review (0.5 points): Does the review provide a clear and concise summary of the selected paper?
Bibliography (0.5 points): Is the bibliography complete and accurate?

HW3: Team Promotion

Selection

Form a group of 2-3 members for your team project.
Go to [People → Team Projects*] in Canvas and add your team members to the same group.

Promotion

Make team promotion slides for a 5 minutes presentation, including the following:
- An overview of your project
- The motivation behind your project and its broad impact
- Key contributions and the novelty of your project
- Proposed methods, such as algorithms, models, and data
- Anticipated challenges and expected results
- Weekly plans that outline individual responsibilities
Submit the slides as a PDF file to PT1: Team Promotion on Canvas.

Report

Sign up for an Overleaf account with your Emory email address. This will grant you access to the Professional Plan once you verify your email address in the Account Settings.
Login to Overleaf and duplicate this project by clicking [Menu] at the top-right corner and selecting "Copy Project".
Name your project as CS371-Lastname1_Lastname2_Lastname3 (replace the last names with your team members' in alphabetical order).
Provide a high-level title that reflects your potential team project.
Update the author information with your details and add additional authors if needed.
Share the Overleaf project with your teammates and instructors, granting them "Can Edit" permissions (refer to the instructor information in the syllabus).
Export your paper as a PDF file and submit it to HW3: Team & Topic Selection on Canvas.
Provide the URL of your Overleaf project as a comment in HW3: Team & Topic Selection.

LaTeX is widely regarded as the gold standard for writing scientific articles in many fields, including computer science. In this course, you will be required to use LaTeX to write your research papers. If you are not already familiar with LaTeX, it is essential that you review the LaTeX Guidelines and take the time to become comfortable with its usage.

Rubric

Selection (0.5 points): Was the team formed by the deadline?
Presentation (2 points): Do the slides clearly and comprehensively describe the proposed task?
Report (0.5 points): Does the report have a compelling title and accurate author information?

HW4: Introduction

Writing

Compose the Introduction section of your research paper in your team's Overleaf project.
Aim for 50 - 100 lines.
Upload a PDF of your draft, including the Introduction section, to Canvas.

Contribution

Provide a breakdown of each team member's contribution to this writing on Canvas as follows:

Use the Text Entry feature on Canvas to submit your response.
List all team members who contributed to the writing.
Next to each team member's name, indicate their contribution as a percentage of the total effort.
Ensure that the percentages for all team members sum up to 100%.

Rubric

Broad Impacts (1 point)
- Articulate the significance and relevance of your research topic.
- Explain why your study matters in a broader context.
Intellectual Merit (1 point)
- Outline the key challenges or complexities of your research problem.
- Highlight any gaps in current knowledge that your study addresses.
Approach (1 point)
- Provide a concise overview of your main research methodology or approach.
- Explain how your chosen method addresses the research problem.
Findings (1 point)
- Summarize your expected outcomes.
- If the study is ongoing, describe anticipated findings based on your hypothesis.
Contributions (0.5 points)
- State the novel contributions of your work to the field.
- Emphasize how your research advances current understanding.
References (0.5 points)
- Appropriately cite relevant literature and prior work.
- Use in-text citations to support your claims and contextualize your research.

HW5: Related Work

Writing

Compose the Related Work section of your research paper in your team's Overleaf project.
Aim for 50 - 100 lines.
Upload a PDF of your draft, including the Related Work section, to Canvas.

Contribution

Provide a breakdown of each team member's contribution to this writing on Canvas as follows:

Use the Text Entry feature on Canvas to submit your response.
List all team members who contributed to the writing.
Next to each team member's name, indicate their contribution as a percentage of the total effort.
Ensure that the percentages for all team members sum up to 100%.

Rubric

Comprehensiveness (2 points)
- Covers a wide range of relevant literature in the field.
- Includes both seminal works and recent developments.
Appropriateness (1 point)
- Cited works are directly relevant to the research task.
- Selected literature is significant within the field of study.
Categorization (1 point)
- Related works are organized into logical, coherent categories.
- Categories reflect meaningful distinctions or themes in the literature.
Description (2 points)
- Clear and concise explanations of each work's methodology.
- Accurate representation of each work's main contributions to the field.
Limitation (1 point)
- Identifies and discusses limitations or gaps in existing works.
- Provides balanced critique of previous research.
Distinction (2 points)
- Clearly articulates how the current work differs from previous research.
- Highlights the unique contributions or advancements of the current study.

HW6: Approach

Writing

Write the Approach section in your team overleaf project.
Recommended length: 150 - 300 lines (including algorithm boxes, figures, and tables).
Submit the PDF version of your current draft up to the Approach section.

Rubric

Overview and Structure (2 points)
- Is the overall architecture of the approach clearly demonstrated, from input to output?
- Is the approach well-organized and logically structured?
Methodology (2 points)
- Are at least two methods (e.g., baseline and advanced) described in detail?
- Is there a clear conceptual comparison between the methods?
Technical Soundness (2 points)
- Are the descriptions of all methods clear and comprehensible?
- Are the descriptions of the methods technically sound and accurate?
Visual and Supplementary Elements (2 points)
- Are appropriate supplementary materials (e.g., figures, tables, algorithms) provided to enhance understanding?
Justification and Rationale (2 point)
- Are the choices of methods well-justified with clear reasoning?
- Is there a discussion of why these methods are appropriate for the research question?
Generalizability (1 point)
- Is the approach described using generalizable methods that could be applied to similar problems?
Novelty (1 point)
- Does the approach demonstrate originality or innovative elements?

HW7: Experiments

Writing

Write the Experiments section in your team overleaf project.
Recommended length: 100 - 200 lines (including tables and figures).
Submit the PDF version of your current draft up to the Experiments section.

Rubric

Data Section (1 point)
- Are the sources and choices of the datasets clearly explained and justified?
- Is the data preprocessing pipeline well-documented (if any)?
- Are the dataset characteristics adequately described?
Data Split (1 point)
- Is the division of data into training/development/evaluation sets clearly specified?
- Are the statistics for each data split comprehensively reported?
- If using cross-validation, is the procedure properly explained?
Model Descriptions (2 points)
- Are the models described with sufficient references to the approach section?
- Are the differences between methods clearly distinguished?
- Is there a clear connection between model design choices and research objectives?
Evaluation Metrics (2 points)
- Are all evaluation metrics clearly defined with proper mathematical notation?
- Is the choice of each metric well-justified?
- Are the limitations of the chosen metrics discussed (if any)?
Experimental Settings (1 point)
- Are the hyperparameters and implementation details fully specified?
- Is the hardware/software environment clearly described?
- Are the experiments documented in a way that enables replication?
Model Development (1 point)
- Is the model development process clearly documented?
- Are the key decisions and modifications during development explained?
- Are the challenges encountered and their solutions discussed?
Result Tables (2 points)
- Are the experimental results presented in well-formatted, readable tables?
- Are all tables properly labeled with units and descriptions?
- Are baseline as well as existing state-of-the-art comparisons included and clearly marked?
Result Interpretations (2 points)
- Are the key findings from the results clearly identified and explained?
- Are the strengths and weaknesses of the results discussed?
- Are the implications of the results connected to the research questions?

HW8: Analysis

Writing

Write the Analysis section in your team overleaf project.
Recommended length: 100 - 200 lines (including tables and figures).
Submit the PDF version of your current draft up to the Analysis section.

Rubric

Performance Analysis (4 points)
- Is there a comprehensive breakdown of model performance across different categories?
- Is there a comparative analysis with baseline, state-of-the-art, and your methods?
- Are there ablation studies to validate design choices?
- Is the analysis supported by appropriate visualizations?
Error Analysis (3 points)
- Are the errors systematically categorized and analyzed?
- Are specific error cases examined in detail?
- Is there a meaningful interpretation of error sources?
Discussions (2 points)
- Are the limitations clearly identified and analyzed?
- Are future research directions properly outlined?

HW9: Conclusion & Abstract

Writing

Write the Conclusion section and the Abstract in your team overleaf project.
Recommended length: 20 - 40 lines for the conclusion, and 20 - 35 lines for the abstract.
Submit the final draft of your individual writing in PDF.

Rubric

Summary: Does the conclusion well-summarize the contributions of key methods? (2 points)
Future Work: Are the next steps of this research clearly motivated and described? (2 points)
Abstract: Is the abstract written in a compelling and comprehensive manner? (3 points)

HW10: Peer Review

Peer Reviews

Check the paper assignments in the comment section of [Grades → Peer Reviews].
Complete reviews for the three assigned papers using this form.
Make sure to fill out the form three times, one for each paper.

Rubric

Summary: Are the key aspects proficiently pointed out and described (1 point)?
Strengths: Are the major contributions objectively assessed (1.5 points)?
Weaknesses: Are the main issues soundingly addressed (1.5 points)?
Feedback: Are the comments and suggestions helpful (1 point)?

Team Project

Supplementary

LaTex Guidelines

This chapter guides you how to write research papers in LaTex.

Most reputable journals and conference proceedings recommend LaTex for scientific writing. This chapter helps you keep consistent formatting in your writings, which has the following benefits:

Readers can easily follow your content since they do not need to learn a new formatting style.
You will spend less time writing since you already know what formatting you want to use to present the content.
Your writings will look professional since the guidelines suggested here have been used to publish numerous papers.

Getting Started
File Structure
Packages
References
Paragraphs
Labels
Tables
Figures
Lists

Getting Started

Installation

Follow the instructions provided by the LaTex team.

Editors

The following editors are recommended to write in LaTex:

Overleaf: an online editor that is widely used in collaborative writing.
TeXstudio: an offline editor that supports Windows, Mac OS, and most Linux distributions.

Template

Throughout the guidelines, this template is used to show examples, which is the official template for ACL conferences.

File Structure

Top Level

Our template includes 4 files on the top level:

acl_latex.tex: contains the main content.
acl_natbib.bst: defines the bibliography format.
acl.sty: defines the paper format.
custom.bib: contains all references.

Tex Folder

Except for acl_latex.tex, all the other tex files are saved under the tex folder, making it easier to manage extended contents. There are 9 files under the tex folder:

abstract.tex
introduction.tex
related-work.tex
approach.tex
experiments.tex
analysis.tex
conclusion.tex
acknowledgements.tex
appndeix.tex

These files should be added in acl_latex.tex:

\input{tex/abstract}
\input{tex/introduction}
\input{tex/related-work}
\input{tex/approach}
\input{tex/experiments}
\input{tex/analysis}
\input{tex/conclusion}
\input{tex/acknowledgments}

% References come after the acknowledgment section
\bibliography{custom}

% Appendix comes after the references and must start at a new page
\cleardoublepage\appendix
\input{tex/appendix}

Image Folder

Additionally, the img folder contains all image files included as figures in the paper.

Packages

Required Packages

Our template uses acl.sty to format the paper, indicated at the top of acl_latex.tex:

\usepackage[final]{acl}

If you replace final to review, it turns into the anonymous mode.

The standard packages include the followings:

\usepackage{times}
\usepackage{latexsym}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}

Fonts

The following packages add useful fonts:

\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{amssymb}       % enable \mathbb, \mathcal
\usepackage{bold-extra}    % enable \texttt{\textbf{}}
\usepackage{bm}            % enable bold font in math mode
\renewcommand{\UrlFont}{\ttfamily\small}  % define fonts for URLs

Recommended Packages

The following packages are recommended:

\usepackage{graphicx}      % import graphical images
\usepackage{multirow}      % merge rows in tables
\usepackage{booktabs}      % borders in tabular
\usepackage{enumitem}      % list margins
\usepackage{subcaption}    % create sub-tables and sub-figures
\usepackage[hang,flushmargin]{footmisc}  % minimize footnote indentation
\newcommand{\LN}{\linebreak\noindent}    % manage inline spacing

References

Bibliography Style

Our template uses acl_natbib.bst to format the bibliography, indicated in acl.sty:

\bibliographystyle{acl_natbib}

Bib Entries

Our template uses custom.bib, is indicated at the bottom of acl_latex.tex, for adding references to be included in the paper:

\bibliography{custom}

Any preprint must be checked whether or not it has been published to a peer-reviewed venue. If it has, use the reference from the peer-reviewed venue instead of the preprint source such as arXiv.

Keep the following conventions to add entries in the bib file:

- For key, make sure there is no duplicate.

- For title, surround the text with curly braces; otherwise, the title will be lowercased in print:

@inproceedings{marcus-etal-1993-building,
    Title = {{Building a Large Annotated Corpus of English: The Penn Treebank}},

@inproceedings{marcus-etal-1993-building,
    Title = "{Building a Large Annotated Corpus of English: The Penn Treebank}",

- For booktitle or journal, do not use acronyms but the full venue names. For instance, the following is good:

Booktitle = {Proceedings of the Annual Conference of the Association for Computational Linguistics},

whereas the following is bad:

Booktitle = {Proceedings of ACL},

Also, do not append the acronym in parentheses at the end of the venue name. For instance, remove (ACL) from the following::

Booktitle = {Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL)},

Use the series field rather to indicate the acronym.

- For series, use the format acronym'year (e.g., ACL'20), where acronym is the acronym of the venue and year is the last two digits of the published year.

- For pages, put two dashes between the first and the last pages (e.g., 1--10).

- For url, add the link to the original source of the paper (e.g., ACL Anthology).

Citations

Use \citet when the reference is used in context:

\citet{devlin-etal-2019-bert} introduced BERT.
BERT was introduced by \cite{devlin-etal-2019-bert}.

Use \cite when the reference is used outside of the context:

BERT is a transformer-based contextualized embedding model \cite{devlin-etal-2019-bert}.

Use \citealt when the reference is used inside of parentheses:

Bidirectional Encoder Representations from Transformers (BERT; \citealt{devlin-etal-2019-bert}) is used to generate token embeddings.

Paragraphs

Write each sentence in a separate line which makes it easier to comment out selective sentences. For instance, write as follows:

This is the first sentence.
This is the second sentence.

instead of writing as follows:

This is the first sentence. This is the second sentence.

which allows the following to comment out the second sentence:

This is the first sentence.
% This is the second sentence.

Put \noindent on any paragraph that

Starts at the top of the page,
Follows tables, figures, or algorithm boxes.

Labels

Labels must not include any space. Use the following prefixes to name labels:

Section: sec:
Subsection: ssec:
Subsubsection: sssec:
Tables (including sub-tables): tab:
Figures (including sub-figures): fig:
Algorithms: alg:

Here are a few examples:

The label of the introduction section → sec:introduction
The label of a subsection describing a decoding strategy → ssec:decoding-strategy
The label of the table showing data statistics → tab:data-stats

Tables

Tables in Column

Use the following template to include tables in the column:

\begin{table}[htbp!]
\centering\small{ %\resizebox{\columnwidth}{!}{
\begin{tabular}{c|ccc} 
\toprule
\bf A & \bf B & \bf C & \bf D \\
\midrule
0 & A0 & B0 & C0 \\
1 & A1 & B1 & C1 \\
2 & A2 & B2 & C2 \\
3 & A3 & B3 & C3 \\
\bottomrule
\end{tabular}}
\caption{Description of this table.}
\label{tab:name}
\end{table}

Use the following options for all tables:

\begin{table}[htbp!]
\centering\small{ %\resizebox{\columnwidth}{!}{

If the table exceeds the column width, put the tabular inside a \resizebox instead:

\centering\resizebox{\columnwidth}{!}{
\begin{tabular}{c||c|c|c}
...
\end{tabular}}

Make sure values in the header row are always center-aligned regardless of the configuration. You can specify this by using \multicolumn:

\begin{tabular}{c||r|r|r} 
\bf A & \multicolumn{1}{c|}{\bf B} & \multicolumn{1}{c|}{\bf C} & \multicolumn{1}{c}{\bf D} \\

Make sure the label starts with the prefix tab:.

Tables in Page

Use the following template to include tables that expand to the full page.

\begin{table*}[htbp!]
\centering\small{ %\resizebox{\textwidth}{!}{
\begin{tabular}{c|ccc} 
\toprule
\bf A & \bf B & \bf C & \bf D \\
\midrule
0 & A0 & B0 & C0 \\
1 & A1 & B1 & C1 \\
2 & A2 & B2 & C2 \\
3 & A3 & B3 & C3 \\
\bottomrule
\end{tabular}}
\caption{Description of this table.}
\label{tab:name}
\end{table*}

If the table exceeds the page width, use \textwidth instead of \columnwidth for \resizebox.
See the other explanations for the Tables in Column.

Sub-Tables

Use the following template to create sub-tables.

\begin{table}[htbp!]
\centering

\begin{subtable}{\columnwidth}
\centering\small{ %\resizebox{\columnwidth}{!}{
\begin{tabular}{c|ccc} 
\toprule
\bf A & \bf B & \bf C & \bf D \\
\midrule
0 & A0 & B0 & C0 \\
1 & A1 & B1 & C1 \\
2 & A2 & B2 & C2 \\
3 & A3 & B3 & C3 \\
\bottomrule
\end{tabular}}
\caption{Sub-table 1.}
\label{tab:name-1}
\end{subtable}
\vspace{0.5em}

\begin{subtable}{\columnwidth}
\centering\small{ %\resizebox{\columnwidth}{!}{
\begin{tabular}{c|ccc} 
\toprule
\bf A & \bf B & \bf C & \bf D \\
\midrule
0 & A0 & B0 & C0 \\
1 & A1 & B1 & C1 \\
2 & A2 & B2 & C2 \\
3 & A3 & B3 & C3 \\
\bottomrule
\end{tabular}}
\caption{Sub-table 2.}
\label{tab:name-2}
\end{subtable}

\caption{Description.}
\label{tab:name}
\end{table}

The subtable environment requires the subcaption package.
Put \vspace{0.5em} at the end of every sub-table except for the very last one.
See Tables in Page to create sub-tables that expand to the full page.

Figures

Image Format

Before including figures, make sure the followings:

Export all images to PDF format. Converting image files (e.g., PNG) to PDF does not render well; you must export the vectorized images to PDF.
Crop white margins around the image. If you use Mac OS or Linux, you can simply type the following command in a terminal, which will crop image.pdf and save it to image-crop.pdf:
```
$ pdfcrop image.pdf image-crop.pdf
```

Figures in Column

Use the following template to include figures in column:

\begin{figure}[htbp!]
\centering
\includegraphics[width=\columnwidth]{img/image.pdf}
\caption{Description.}
\label{fig:name}
\end{figure}

Use the following options for all figures:

\begin{figure}[htbp!]
\centering

The width can be configured proportionally. The following example sets the width to 0.9 * \columnwidth:

\includegraphics[width=0.9\columnwidth]{img/image.pdf}

If the figure seems too large, use scale instead of width as the option for \includegraphics:

\includegraphics[scale=0.5]{img/image.pdf}

Make sure the label starts with the prefix fig:.

Figures in Page

Use the following template to include tables that expand to the full page.

\begin{figure*}[htbp!]
\centering
\includegraphics[width=\textwidth]{img/image.pdf}
\caption{Description.}
\label{fig:name}
\end{figure*}

Use the \textwidth option instead of \columnwidth for \resizebox.
See the other explanations for the Figures in Column.

Sub-Figures

Use the following template to create sub-figures.

\begin{figure}[htbp!]
\centering

\begin{subfigure}{\columnwidth}
\centering
\includegraphics[width=\columnwidth]{img/image.pdf}
\caption{Sub-figure 1.}
\label{tab:name-1}
\end{subfigure}

\begin{subfigure}{\columnwidth}
\centering
\includegraphics[width=\columnwidth]{img/image.pdf}
\caption{Sub-figure 1.}
\label{tab:name-1}
\end{subfigure}

\caption{Description.}
\label{tab:name}
\end{figure}

The subfigure environment requires the subcaption package.
Put \vspace{0.5em} at the end of every sub-figure except for the very last one.
See Figures in Page to create sub-tables that expand to the full page.

Lists

Left Margin

Use the following command if you do not want any indent in a list environment:

\begin{itemize}[leftmargin=*]
\item A
\item B
\end{itemize}

If you want to set this globally, uncomment the following configurations in acl_latex.tex:

\setenumerate[1]{leftmargin=*}    % no enumerate indentation
\setitemize[1]{leftmargin=*}      % no itemize indentation

Vertical Spacing

Use the following command to control the vertical spacing in a list environment:

\begin{itemize}
\setlength\itemsep{0em}
\item A
\item B
\end{itemize}

Writing Tips

Pronoun

It is better to avoid the pronoun "we" in your writing (the possessive pronoun "our" is ok). If you want to give an overview of your paper or approach, use "this paper" or indicate a specific section instead:

We introduce ... -> This paper introduces ...
-> This section introduces ...
We present our transformer model in Section 3.
-> Section 3 presents our transformer model.

Use passive voice to describe your methodology or findings:

We use a transformer model to ...
-> A transformer model is used to ...
We collect 1,000 posts from ...
-> 1,000 posts are collected from ...

You may want to use the pronoun "we" to distinguish discussions or notes that are helpful but do not necessarily contribute to the general knowledge:

Note that we also experimented with ... that did not show a better performance.

Tense

Use the present tense to describe your work:

Three models were developed for ...
-> Three models are developed for ...

Use the past tense to describe previous work:

Chen and Choi, 2016 introduced ...
In our preliminary study, an accuracy of 92% was achieved for ...

Progress Reports

This chapter provides a schedule and rubric for progress reports.

Submission

Please refer to the syllabus for detailed information about progress reports.
Submit your presentation slides in PDF by the selected date to Canvas.

Rubric

Motivation (0.5 points): It assesses the clarity and effectiveness of the project's motivation. It examines how well you present the reasons and justifications for undertaking the research.
Objectives (0.5 points): It evaluates the scope and specificity of the project's objectives. It focuses on how well you define the intended outcomes and goals of the research.
Specific Methods (1 point): It examines your explanation and demonstration of the particular methods used in the project. It assesses your ability to articulate the techniques, tools, or approaches employed to address the research objectives.
Experimental Design (1 point): It evaluates the quality and soundness of the experimental design used in the project. It assesses how well you plan and structure the experiments to develop models and achieve the research objectives.
Analysis Plan (0.5 points): It assesses your proposed or implemented analysis plan for the experimental results. It evaluates your ability to outline the techniques and procedures used to analyze and interpret the research findings.
Challenges (1 point): It examines your identification and discussion of challenges encountered during the project. It assesses your capacity to identify potential obstacles and provide strategies or plans to overcome them.
Member Contributions (0.5 points): It evaluates the individual contributions of team members to the progress report. It assesses the extent to which each team member actively participated and contributed to the project.

Schedule

Phase 1

10/2

10/4

Phase 2

10/18

10/23

Phase 3

11/1

11/6

Team Promotion

Present your team project to the class, highlighting its key aspects and demonstrating its value and significance.

Exercise

Connect with your teammates.
Make promotion slides for a 5 minutes presentation, including the following:
- An overview of your project
- The motivation behind your project and its broad impact
- Key contributions and the novelty of your project
- Proposed methods, such as algorithms, models, and data
- Anticipated challenges and expected results
- Weekly plans that outline individual responsibilities
Submit your presentation slides in PDF to [Assignments → Exercises → Team Promotion] in Canvas (one per team).