Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
This chapter delves into a range of research areas in AI and provides an overview of the available resources based on faculty interests.
This chapter provides an overview of the course and establishes the research environment for students.
Explore diverse research areas within the domain of Artificial Intelligence.
Go to [People → 1.1. Research Areas*] in Canvas and select a research area.
Visit the conference websites relevant to the selected research area and examine their programs and proceedings to categorize the pertinent fields of study.
Analyze the latest trends in your specific research fields of interest and explore the state-of-the-art approaches and methodologies employed in those fields.
For each group member, conduct a brief literature survey and identify two potential research ideas based on the survey.
Make slides summarizing the above activities for a short presentation (one per group) and submit them to [Assignments → Exercises → 1.1. Research Areas] in Canvas.
Homework 0: Getting Started
Sign up for an Overleaf account with your Emory email address. This will grant you access to the Professional Plan once you verify your email address in the Account Settings.
Login to Overleaf and duplicate this project by clicking [Menu] at the top-right corner and selecting "Copy Project".
Name your project as CS371-Firstname_Lastname
(replace "Firstname" and "Lastname" with your own first name and last name).
Update the first author's information with your own details and remove any additional author entries, leaving yourself as the sole author.
Share your project with the instructors, granting them "Can Edit" permissions:
Jinho Choi: jinho.choi@emory.edu
Benjamin Ascoli: benjamin.ascoli@emory.edu
Submit the URL of your shared Overleaf project to Canvas.
Familiarize yourself with the Latex Guidelines.
Add a new reference to the custom.bib
file and cite it appropriately in Section 2.
Insert a PDF image as a figure in Section 3, ensuring that it expands to fit the entire page.
Include a table in Section 4 that consists of two sub-tables, distinct from the existing one provided in the template.
Complete the research interest form by providing the required information.
Overleaf (0.2 points): Is the individual Overleaf project shared?
Latex:
Author (0.2 points): Is the author information updated correctly?
Citation (0.2 points): Is a new reference cited appropriately in Section 2?
Figure (0.2 points): Is a PDF image expanding to fit the entire page added to Section 3?
Table (0.2 points): Is a new table consisting of two sub-tables added to Section 4?
Familiarize yourself with the ongoing research interests and focus areas of the faculty members at Emory CS.
Go to [People → 1.2. Faculty Interests*] in Canvas and select a professor.
Visit the selected professor's website to ascertain one's research areas of focus.
Review the professor's publications to gain insights into one's ongoing research projects.
For each group member, identify two potential research ideas that align with the professor's areas of expertise and are of interest to you (for potential collaboration).
Make slides summarizing the above activities for a short presentation (one per group) and submit them to [Assignments → Exercises → 1.2. Faculty Interests] in Canvas.
Eugene Agichtein
Director of the Intelligent Information Access Lab
Publications: Google Scholar, Semantic Scholar
Webpage: http://cs.emory.edu/~eugene
Joined Emory in Fall 2006
Director of the Assured Information Management and Sharing Lab
Publications: Google Scholar, Semantic Scholar
Webpage: http://cs.emory.edu/~lxiong/
Joined Emory in Fall 2005
Director of the Natural Language Processing Lab
Publications: Google Scholar, Semantic Scholar
Webpage: http://cs.emory.edu/~choi
Joined Emory in Fall 2014
Director of the Practical Data Mining & Exploration Lab
Publications: Google Scholar, Semantic Scholar
Webpage: https://joyceho.github.io
Joined Emory in Spring 2016
Research Area: Human-Computer Interaction
Publications: Google Scholar, Semantic Scholar
Webpage: https://cs.cmu.edu/~chinmayk
Joined Emory in Fall 2022
Research Area: Natural Language Processing
Publications: Google Scholar, Semantic Scholar
Webpage: https://www.cs.emory.edu/~fliu40
Joined Emory in Fall 2022
Research Area: Human-Computer Interaction
Publications: Google Scholar, Semantic Scholar
Joined Emory in Fall 2023
Research Area: Spatio-Temporal Mining
Publications: Google Scholar, Semantic Scholar
Webpage: https://www.zuefle.org
Joined Emory in Fall 2022
Research Area: Data-Centric AI
Publications: Google Scholar, Semantic Scholar
Webpage: http://cse.msu.edu/~jinwei2
Joined Emory in Fall 2023
Director of the Cognition and Visualization Lab
Publications: Google Scholar, Semantic Scholar
Webpage: http://emilywall.github.io
Joined Emory in Fall 2021
Research Area: Human-Computer Interaction Lab
Publications: Google Scholar, Semantic Scholar
Webpage: http://www.cs.emory.edu/~kwil271
Joined Emory in Fall 2022
Research Area: Graph Data Mining
Publications: Google Scholar, Semantic Scholar
Webpage: http://cs.emory.edu/~jyang71
Joined Emory in Fall 2020
Research Area: Machine Intelligence for Complex Data
Publications: Google Scholar, Semantic Scholar
Webpage: https://cs.emory.edu/~lzhao41
Joined Emory in Fall 2020
CS371: Research Practicum in Artificial Intelligence
This course guides students in developing the necessary skills to conduct high-quality research in Artificial Intelligence (AI). AI has emerged as a mainstream component of human endeavors, pivotal in facilitating translational research across diverse disciplines. AI has transitioned from a futuristic concept to an integral and indispensable part of our daily lives. Thus, it is vital to comprehend the various domains of AI and their potential applications in serving humanity.
The primary goals of this course include:
Surveying various areas of AI and exploring state-of-the-art advancements in AI research.
Developing impactful ideas that effectively engage and captivate research communities.
Designing innovative and intellectually stimulating methodologies that push the frontiers of knowledge, fostering new insights and advancements in the field.
Conducting experiments with thorough analyses that aptly capture the core essence and significance of the research.
Presenting research findings in a persuasive and intellectually captivating manner.
Throughout the course, each student will be expected to accomplish the following:
Write an individual research paper.
Review papers authored by peers.
Collaborate within a group to execute a team project.
Deliver public presentations to showcase your work.
This course satisfies the Continuing Writing Requirement.
Jinho D. Choi - Associate Professor of Computer Science, Emory University.
Fall 2023
Course Webpage: https://emory.gitbook.io/ai-research-practicum/
Prerequisites: CS325: Artificial Intelligence or CS334: Machine Learning
Class Hours: MW 11:30 AM - 12:45 PM
Class Location: MSC N304
Jinho Choi Associate Professor of Computer Science Office Hours: MW 4 PM - 5 PM → W302F @ Math & Science Center
Benjamin Ascoli Ph.D. Student in Computer Science and Informatics Office Hours: TuTh 11:30 AM - 12:45 PM → W302B @ Math & Science Center
8 writing assignments: 60%
1 progress report: 5%
3 peer reviews: 12%
1 team project: 23%
Late submissions will be accepted within a one-week grace period; however, a grading penalty of 15% will be applied. No submissions will be accepted beyond the one-week grace period.
Your work in this course is subject to the Emory Honor Code. Any honor code violations will be reported to the Emory College Honor Council for appropriate action.
In the event of personal circumstances (e.g., health issues, family emergencies) that have a substantial impact on your course performance, you can submit a letter from the Office for Undergraduate Education as supporting documentation for making proper accommodations.
Each of you will individually write a full research paper by the end of the semester. You are expected to write each section of the paper as a separate assignment.
Your paper should concentrate on the specific parts of the team project for which you have been assigned responsibility.
Since all members of your team collaborate on the same project, there may be overlapping writing content among you. However, it is vital to uphold the originality of your own writing. Copying writings from your teammates or any external sources is strictly prohibited.
Occasionally, you may need certain content from your teammates to explain your parts. In such cases, you need to clearly indicate the parts that you have personally contributed and those that your teammates have contributed to your writing.
Each of you will deliver a 15-minute presentation covering the following aspects:
The motivation behind your project as well as its main objectives, along with the specific methods employed, experimental design, and analysis plan employed in your project.
Identification of ongoing and anticipated challenges, as well as strategies for overcoming those challenges.
Detailed explanations of the contributions made by each team member, highlighting what has been accomplished and what is planned for the future.
Progress reports will be presented in three phases. To ensure diversity across each phase, only one member per team is allowed to present during each phase.
Attendance at the progress report sessions is mandatory. Failure to attend more than two sessions will result in a penalty (1 point deduction per session beyond the initial two) unless appropriate accommodations have been arranged.
The final draft of your individual writing will be reviewed by three peers. In turn, you will also be responsible for reviewing three papers written by individuals outside of your team.
The objectives of the peer reviews are as follows:
To facilitate your learning of proper research paper review techniques.
To assist authors in enhancing the quality of their research by incorporating your feedback.
It is essential that you write comprehensive and helpful reviews that provide valuable insights and suggestions to the authors.
During the majority of the semester, you will be engaged in a team project. The objectives of this project are as follows:
To develop the skills necessary for conducting effective collaborative research.
To produce high-quality research outcomes through the synergy among team members.
As a team, you are expected to:
Maintain regular and frequent meetings with your teammates on a weekly basis.
Prepare and deliver oral and poster presentations by the end of the semester.
Collaboratively write a research paper by combining individual writings of team members.
Team projects will be ranked by peers, and these rankings will be taken into account in the evaluation of the projects.
All members of each team will receive the same scores for the research paper, oral & poster presentations, and ranking. However, the overall project score will be weighted based on individual contributions.
Instructors will rigorously utilize large language model (LLM)-based systems, such as ChatGPT, to review and evaluate your individual writings.
You are highly encouraged to leverage LLM-based systems to revise and review the content you have already written, enhancing its quality and coherence.
You are not allowed to rely on LLM-based systems to generate any content from scratch. For example, you cannot use an LLM to create an introduction by providing only a title or to write additional paragraphs that may plausibly follow a given paragraph.
Engaging LLM-based systems to generate fabricated data, experimental results, or analyses is strictly prohibited. Any violation of this policy will result in a referral to the Honor Council for further investigation.
While you are permitted to utilize LLM-based systems to discover relevant resources (e.g., references), it is your responsibility to ensure the accuracy and reliability of those resources.
This course is conducted in person, and no virtual links or digital recordings will be provided.
It is recommended that you bring your laptop as there will be many in-class exercises.
Homework 1: Exploration
Form a group of 2-3 members for your team project.
Go to [People → Team Projects*] in Canvas and add your team members to the same group.
Create an Overleaf project with the following naming convention: CS371-Lastname1_Lastname2_Lastname3
(replace the last names with yours in alphabetical order).
Provide a concise and descriptive title that accurately reflects your project.
Update the author information to include all team members.
Share the Overleaf project with your teammates and instructors, granting them "Can Edit" permissions (refer to the instructor information here).
Submit the URL of your shared Overleaf project to Canvas.
Go to [People -> Progress Reports*] in Canvas and select a date for your progress report presentation.
Ensure that each phase of the progress report has only one team member assigned to it.
Team Assignment (0.6 points): Is the team formed?
Overleaf: Is the team Overleaf project shared? (0.2 points)
Progress Report: Is the date for the progress report presentation chosen? (0.2 points)
Fall 2023
Select a specific task and determine the research objectives of your project.
Choose a particular task for your team and discuss the types of data you wish to concentrate on (e.g., open relation extraction on discussion forums).
Outline the primary research objectives (which may be more than one) of your project:
Devise an innovative methodology that outperforms previous approaches on specific or relevant datasets.
Construct a novel dataset that facilitates task adaptation to new domains or languages.
Conduct a comprehensive analysis of state-of-the-art models, focusing on comparative aspects to gain insights into their strengths and weaknesses.
Develop a practical application employing existing resources to advance translational research.
Make slides summarizing the above outline for a short presentation (one per group) and submit them to [Assignments → Exercises → 2.1. Task Section] in Canvas.
Today's team does not need to be the one for your final project; however, use this opportunity to explore the interests of your colleagues.
Describe the motivation behind your selected task.
If the selected task has been popular in recent years, highlight its prevalence and reference recent seminal works related to the study:
TASK has recently gained lots of interests because REASONS (CITATIONS).
You may want to make separate citations per reason if appropriate e.g., REASON 1 (CITATIONS), REASON 2 (CITATIONS)
If the selected task is new or has not been widely recognized in recent times, explain its significance and cite relevant works that provide evidence of its importance:
TASK is important because REASONS (CITATIONS), yet it has been underexplored.
What are the advantages and disadvantages of opting for novel tasks compared to popular ones? Should you ever choose tasks that are neither novel nor popular?
Identify the challenges present in this task that you aim to enhance, and provide citations to previous works that have addressed these challenges:
Despite the great advancement, CHALLENGE is challenging because REASONS (CITATIONS).
Concisely describe the major approaches that have been developed to tackle these challenges and cite the corresponding works:
Several stuides have proposed APPROACH to overcome this issue (CITATIONS).
It should serve as an overarching description of previous works that have employed similar approaches.
Explain the limitations associated with those approaches.:
However, it is still limited by REASONS.
If your work addresses multiple challenges, you may consider dedicating a paragraph to each challenge, where each paragraph follows all the steps outlined in this section.
Date | Topic | Assignment |
---|---|---|
Examine the below, and form a team comprising up to 3 members using [People → 2.1. Task Selection*] in Canvas.
The novelty of your research will be assessed based on these objectives. Ensure that your objectives are designed to address the challenges outlined in the .
The scope of references here should encompass a broader range than the ones cited in the section.
Finding the most relevant works at this stage is strongly advised, especially considering the limited timeframe to complete the section, which should be used to understand the work rather than to explore additional sources.
08/23
08/28
08/30
(continue)
09/04
Labor Day
09/06
09/11
(continue)
09/13
09/18
(continue)
09/20
09/25
09/27
(continue)
10/02
10/04
10/09
Fall Break
10/11
10/16
(continue)
10/18
10/23
10/25
10/30
(continue)
11/01
11/06
11/08
11/13
(continue)
11/15
11/20
(continue)
11/22
Thanksgiving Recess
11/27
Oral Presentations
11/29
Oral Presentations
12/04
Poster Presentations
Team Project
Provide the overview of your project.
Elaborate on your approach and reference the Approach section:
In this paper, we present APPROACH to improve CHALLENGE (Section #).
You may need to describe multiple approaches if your work serves multiple objectives.
Explain the hypothesis underlying your approach by specifying which methods are expected to address particular issues:
We hypothesize that METHOD can resolve ISSUE because REASONS.
Our approach uses METHOD to resolve ISSUE because REASONS.
What distinguishes "approach" from "method"?
How do we differentiate between "challenge" and "issue"?
Specify the types of data and/or models used to evaluate your approach, explain the reasons for using them, and reference the Experiments section for detailed information on this aspect:
Our approach is evaluated on DATA with MODELS (Section #).
You may not want to mention the exact names of the datasets or models, but instead refer to their types to emphasize the generalizability of your work.
Provide a high-level description of the primary findings from the analysis that offer supporting evidence for your hypothesis, and reference the Analysis section where a comprehensive breakdown of the results can be found:
Our experiments show that FINDINGS (Section #).
At this stage, you do not know the outcomes of the experiments. Thus, write your expected findings here.
List specific contributions of your work:
This work makes THREE main contributions as follows:
We create a new dataset for TASK.
We introduce a new model for TASK.
MODEL achieves the state-state-of-the-art performance on TASK.
A real system is built for TASK.
Indicate the primary contribution of your work (if any):
To the best of our knowledge, this is the first work that does APPROACH for TASK.
This work will facilitate the development of ...
Learn how to structure the Introduction section.
Find your team members.
Select 3-5 papers related to your task.
For each paper, excerpt writings in the introduction section for the following categories:
Broad Impacts
Intellectual Merit
Approach
Findings
Contributions
Identify parts of the writings that do not fit into any of the categories above and propose appropriate categories for them.
Make slides summarizing the above analysis for a short presentation (one per group) and submit them to [Assignments → Exercises → 2.4. Exercise] in Canvas.
Homework 2: Introduction
Write the Introduction section in your individual overleaf project.
Recommended length: 50 - 100 lines.
Submit the PDF version of your draft, including the Introduction section.
Broad Impacts (1 point): Is the significance of the task well-justified?
Intellectual Merit (1.5 points): Are the challenges of the task effectively outlined?
Approach (1 point): Is the main approach coherently stated?
Findings (0.5 points): Are the (expected) findings clearly indicated?
Contributions (0.5 points): Are the contributions of the work concisely summarized?
References (0.5 points): Are the relevant works and sections appropriately referenced?
The scores for findings and contributions are lower because they cannot be completely figured out at this time of the writing, not because they are less critical.
Conduct a survey of previous work relevant to your research.
It is common to survey the literature for the following three categories:
These are general categories; feel free to make your own categorization as needed.
Conduct a comprehensive survey of the most recent works (e.g., last 2-3 years) that show state-of-the-art results in your task. The survey should not be limited by a certain domain or dataset and should encompass a broad range of approaches and methodologies:
Begin by examining the latest research papers and proceed by exploring their Related Work and Experiment sections for additional insights.
Useful sites: Papers with Code, ACL Anthology.
If you are the first to address this task, explore recent works related to similar tasks.
Many previous studies have shown remarkable results on YOUR TASK.
Only a few works have tackled YOUR TASK.
Although YOUR TASK has been underexplored, several works have been done on similar tasks.
Conduct comparative studies among previous works to identify their strengths and weaknesses. Describe each work briefly in 1-2 sentences, highlighting its unique contributions and differences from other approaches:
CITATION was the first to adapt APPROACH to TASK.
CITATION presented APPROACH/MODEL that showed the state-of-the-art results on DATA.
Use /citet
instead of /cite
in LaTex, which allows you to indicate the authors without parentheses.
Describe the limitations of the previous works.
Despite the great work, these models have CHALLENGES.
Explain how your work is distinguished from theirs and can potentially overcome such challenges:
Our work is distinguished because REASONS that handle ISSUES better.
Avoid citing a preprint version (e.g., arXiv) of a paper if it has been published in a peer-reviewed venue; instead, cite the published venue to ensure credibility, as papers on arXiv are not peer-reviewed and may have limited credentials.
If you intend to apply existing methods or techniques from other tasks to your own, conduct a survey of significant works that have employed such methodologies across various tasks:
YOUR METHOD has been sucessfully adapted to TASKS.
If you are the first to introduce this methodology, find papers using similar methods.
Provide a brief 1-2 line description of each work and elucidate how these methods have contributed to the improvement of their respective tasks.
CITATION used METHOD and signficantly improved ASPECTS of TASK.
Explain the reasons why these methods are likely to enhance specific aspects of your task:
Given the great success of METHOD, we beileve it can enhance ASPECTS of YOUR TASK.
If you plan to adapt existing tasks or methods to a new domain or language, survey renowned works in that particular domain/language, irrespective of the tasks they address:
Many recent studies have focused on DATA.
Provide a concise 1-2 line description for each work and highlight the key challenges or contributions they encountered while dealing with data from the specified domain/language:
CITATION presented METHOD to tackle TASK on DATA and showed promising results.
CITATION tackeled TASK on DATA and found CHALLENGES.
Clarify the importance of applying these findings to your task in the new domain/language, emphasizing the potential benefits and insights that can be gained from tackling such a cross-domain or cross-language challenge:
Given the growing interest, many people will benefit if there is a robust model for YOUR TASK on DOMAIN/LANGUAGE.
Conduct an extensive and thorough survey related to your task.
Assemble your team members.
Survey recent papers relevant to your task, categorizing them based on criteria such as task, methodology, data, etc.
For each paper, provide a concise overview of the main approach, highlighting its strengths and weaknesses.
Explain how your work is distinguished from the existing works, emphasizing the advantages and unique contributions of your approach.
Make slides summarizing the above analysis for a short presentation (one per group) and submit them to [Assignments → Exercises → 3.2. Exercise] in Canvas.
A practical way of finding relevant papers is to start with one recent paper, collect the state-of-the-art papers cited in the paper (usually found in the Experiment section), branch out to those cited papers, and repeat this process recursively.
Write the Related Work section in your individual overleaf project.
Recommended length: 50 - 100 lines.
Submit the PDF version of your current draft up to the Related Work section.
Comprehensiveness: To what extent does the survey cover relevant literature? (3 points)
Appropriateness: How relevant and significant are the cited works to your task? (1 point)
Categorization: How well are the related works organized into appropriate categories? (1 point)
Description: How well is each work explained for its methodology and contribution? (2 points)
Limitation: How well are the limitations of existing works discussed? (1 point)
Distinction: How effectively is your work differentiated from the previous works? (2 points)
This chapter guides you to write the approach section.
Typically, it is better to write the approach section as abstract as possible so your methods become generalizable for many tasks. For example, even if you use BERT as an encocder but your approach can take any transformer as an encoder, it is better to denote that your method uses a transformer as an encoder instead of BERT.
What are the ways to handle arbitrarily large input using a contextualized encoder?
One popular method is called the "Sliding Window", which splits the input into multiple blocks of text, generates embeddings for each block separately, and merges them at the end.
Modify the baseline method such that a block has overlapped tokens with its surrounding blocks (both front and back). Once all blocks are encoded, each overlapped token should have two embeddings. Create an average embedding of those two embeddings and make it the final embedding for the overlapped token.
Would the following method be equivalent to the above method?
This chapter discusses how to develop new algorithms and write them in pseudocode.
Your task is to design an algorithm that takes a post with the title and its comments with replies from a discussion forum (e.g., Reddit) and converts them into a multi-turn one-to-one dialogue.
A post with the title (https://www.reddit.com/r/college/comments/v7h9rs):
How do you focus when you’re depressed?
I have so many assignments due, with exams coming up too. Life's just keeps hitting me recently and I'm finding it really hard to sit down and take information in. Writing is hard, listening and paying attention is hard. Even if I manage to listen or read none of the information stays in my head. Any help is very appreciated!!
Comments and replies:
Get up early everyday and use the library to study if you have one, idk if you're like me but as soon as I get home I'm kinda done for the day so it helps to stay somewhere where you can't really relax.
Thank you, I’ll give this a try tomorrow
What helps me is embracing when I'm feeling down and allowing myself to take a deserved break. Sometimes I confuse my depressive episodes with burnout and it's important to know your limits. The biggest pro tip to not be overwhelmed with so much to do all at once is doing something every day. Dedicating simply 30 minutes to an hour a day of intense studying goes a long way over time vs cramming at the end. If you're able to do more than 1 hour then great! But know that you don't have to do 6-7 intense studying hours a day to be successful. Be intentional with your time and work smarter vs harder. Your future self will thank you.
This is so nice to hear, and very helpful, thank you!
Hardest part is starting to study, once I have like 15 minutes into my study session that’s my only focus and just forget everything else.
Give a comparison overview of your algorithms with key features:
We introduce two algorithms for the reddit-to-dialogue generation: the baseline algorithm considers every sentence in the post an utterance of Speaker 1 and each comment an utterance of Speaker 2 (Section 3.1), whereas the advanced algorithm finds an appropriate span of sentences from the post to form an utterance for Speaker 1 and an appropriate span of any comment to form an utterance for Speaker 2 (Section 3.2).
Indicate the objective of your algorithm(s):
The main objective is to generate a multi-turn dialogue using a post, its comments, and replies that flows naturally in context.
Describe what the input and output data should be (possible with a figure) that are commonly applied to all algorithms:
All algorithms assume that the number of sentences in the input post is less than or equal to the number of comments. The generated dialogues involve two speakers where utterances of Speakers 1 and 2 are extracted from the post and comments, respectively.
The title or each sentence in the post is considered an utterance of Speaker 1 (S1
).
S1
: How do you focus when you’re depressed?
S2
: What helps me is embracing when I'm feeling down and allowing myself to take ...
S1
: I have so many assignments due, with exams coming up too.
S2
: Get up early everyday and use the library to study if you have one, ...
S1
: Life's just keeps hitting me recently and I'm finding it really hard to sit down and take information in.
S2
: Hardest part is starting to study, once I have like 15 minutes into my study session that’s my only focus and just forget everything else.
Illustrate the baseline algorithm in pseudocode. Create helper methods if they help the readability and/or generalizability of your algorithm.
Give a brief overview of the algorithm by explaining what each line of the code does.
Describe helper methods (if any) in detail.
Define the input:
Is the input correctly described according to the objective?
Initialize the output and auxiliary data structures:
Describe the loop:
Return the output:
How do you estimate such likelihoods?
Any span of consecutive sentences is considered an utterance of S1
.
S1
: How do you focus when you’re depressed? I have so many assignments due, with exams coming up too.
S2
: What helps me is embracing when I'm feeling down and allowing myself to take a deserved break.
S1
: Life's just keeps hitting me recently and I'm finding it really hard to sit down and take information in.
S2
: Get up early everyday and use the library to study if you have one, idk if you're like me but as soon as I get home I'm kinda done for the day so it helps to stay somewhere where you can't really relax.
S1
: Writing is hard, listening and paying attention is hard.
S2
: Hardest part is starting to study, once I have like 15 minutes into my study session that’s my only focus and just forget everything else.
S1
: Even if I manage to listen or read none of the information stays in my head. Any help is very appreciated!!
S2
: Sometimes I confuse my depressive episodes with burnout and it's important to know your limits. The biggest pro tip to not be overwhelmed with so much to do all at once is doing something every day.
Given the input text where is the 'th token in ​, a contextualized encoder (e.g., BERT) takes and generates an embedding for every token using as well as its context. The challenge is that this encoder can take only up the -number of tokens such that it cannot handle any input where .
Let W = W_1 \cup \cdots \cup W_k \where if ; otherwise, such that . Then, the encoder takes each ​ and generates for every token in . Finally, the embedding matrix is created by sequentially stacking all embeddings in .
In a sequence-to-sequence model (aka, an encoder-decoder model), a decoder takes an embedding matrix and predicts what token should come next. It is often the case that this embedding matrix is also bounded by a certain size, which becomes an issue when the size of the matrix becomes larger than (for the case above, where ). One common method to handle this issue is to use an attention matrix for dimensionality reduction as follows:
The embedding matrix is first transposed to then multiplied by an attention matrix such that . Finally, the transpose of , that is gets fed into the decoder.
An attention matrix is multiplied by the embedding matrix such that . Finally, gets fed into the decoder.
For each utterance of S1
, find a comment that is the most relevant and make it the response to from Speaker 2 (S2
).
Let be an input post where is the 'th sentence in , and be a set of 's comments such that where is the 'th comment in and is the 'th sentence in .
Let be the list of utterances representing the output dialogue (L1
) and be a set of segments created from (L2
).
The algorithm visits every sentence (L3
) and appends it to (L4
). It then finds the most-relevant segment (L5
) and adds to (L6
). gets trimmed with (L7
).
Finally, it returns as the output (L8
).
Describe the method:
The method removes and returns the first sentence in .
Describe the method:
The method makes each comment a segment s.t. , where C'j = c_{j1}\,^\frown ...^\frown c_{j\ell} (: text concatenation).
Describe the method:
The method takes comprising all previous utterances and , then estimates the likelihood of being the next utterance.
Describe the method:
the method removes from such that .
For each utterance of S1
, find a span of any consecutive sentences in comments that is the most relevant and make it the response to from S2
.
Write the Approach section in your individual overleaf project.
Recommended length: 150 - 300 lines (including algorithm boxes, figures, and tables).
Submit the PDF version of your current draft up to the Approach section.
Overview: is the overall architecture of your approach clearly demonstrated (from input to output)? (2 points)
Substances: are there at least two methods (e.g., baseline and advanced) described and *(conceptually) compared? (2 points)
Clarity: are all methods clearly explained? (2 points)
Soundness: are the descriptions of the methods sounding? (2 points)
Supplements: are enough supplements (e.g., figures, tables, algorithms) provided? (2 points)
Justification: are the choices of methods well-justified? (1 point)
Generalizability: is the approach described using specific or generalizable methods? (1 point)
Novelty: is the approach considered to be novel? (1 point)
When you create a dataset, the followings need to be clearly described:
Data collection (e.g., sources of the data).
Preprocessing if performed (e.g., scripts that you write, existing tools used).
Annotation scheme and guidelines if conducted with justification.
People involved in this process (e.g., annotators, survey subjects).
Quality of the created data (e.g., inter-annotator agreement).
Statistics and analysis of the original, preprocessed, annotated data.
Here are a few papers presenting new datasets:
, Li et al., EMNLP 2020 (see Section 3).
, Yang and Choi, SIGDIAL, 2019 (see Section 3).
This section describes models used for comparative study.
Describe existing models or frameworks commonly adopted by your models (if any):
All our models adopt MODEL (CITATION) as the encoder.
List all models used for your experiments. Give a brief description of each model by referencing specific sections explaining the core methods used by the model:
The following three models are experimented:
BASELINE: DESCRIPTION (Section #)
ADVANCED: BASELINE + METHOD (Section #)
BEST: ADVANCED + METHOD (Section #)
It is important to design models in a way that clearly shows key differences in methods.
Because a neural model produces a different result every time trained, you need to train it 3 ~ 5 times and report its average score with the standard deviation (Section 5.3).
Why would a neural model produce a different result every time it is trained?
Thus, indicate how many times each model is trained and what is used as the evaluation metric(s):
Every model is trained 3 times and its average F1-score and the standard deviation is used as the evaluation metric.
If you experiment on datasets used in previous work, you must use the same evaluation metrics for fair comparisons. Even if you present a new metric, you still need to evaluate both the old and new metrics to show the advantage of your new metric.
If you use a non-standard metric that has not been used in previous work because:
The task is new,
The new aspect introduced for this task has never been tested before,
You find a better way of evaluating this, which has not been used in the previous work
explain why you cannot apply standard metrics to evaluate this task and describe the new metric:
Since TASK has not been evaluated on ASPECT(S) in prevoius work, we introduce new metrics ...
If your new evaluation metric is novel that requires bigger attention, explain it in the approach section so it would be considered one of the main contributions.
Describe hyper-parameters used to build the models (e.g., epoch, learning rate, hidden layer, optimizer, batch size):
MODEL is trained for # epochs using the learning rate of FLOAT, ...
Explain anything special that you do for training:
Early stop is adopted to control the number of epochs if the score on the development set does not improve over two epochs.
Describe computing devices used for the experiments:
Our experiments use NVIDIA Titan RTX GPUs, which takes 10/20/30 hours for training the BASELINE/ADVANCED/BEST MODELS, respectively.
It is important to describe the experimental settings for replication, although they are often put in the appendix due to the page limit for the actual paper submission.
If you observe enhanced training efficiency (e.g., your new loss function requires a fewer number of epochs to train), create a figure (e.g., x-axis: epochs, y-axis: accuracy) describing the training processes of the baseline and the enhanced models.
Our ENHANCED MODEL reaches the same accuracy (or higher) than the BASELINE model after only a third of epochs.
If you experience unusual phenomena during training (e.g., results on the development set are unstable), describe the phenomena and analyze why they are happening:
This chapter guides you how to write the analysis section
This section aims to provide an in-depth look at what your models can and cannot do so that researchers can make informed decisions in the following step and the best practice in applications.
This section describes datasets used for the experiments.
If there are datasets for your task that have been widely used by previous work, you need to test your models on them for fair comparisons. For each dataset, briefly describe the dataset:
For our experiments, DATASET is used (CITATION) to evaluate our MODEL.
If it is not widely used but appropriate to evaluate your approach, explain why it is selected:
For our MODEL, the DATASET is used (CITATION) because REASON(S).
If it is a new dataset that you create, briefly mention it and reference the section describing how the dataset is created (e.g., Data Creation):
All our models are evaluated on the EXISTING DATASET (CITATION) as well as YOUR DATASET (Section #).
It is always better to experiment with multiple datasets to validate the generalizability of your approach.
Your work is not comparable to previous work unless it uses the same data split. Indicate which previous work you follow to split the training, development, and evaluation sets:
The same split as CITATION is used for our experiments.
During the development (including hyper-parameter tuning), your model should be trained on the training set and tested on the development set (aka. validation set).
While you are training, you should frequently check the performance of your model (usually once every epoch) and save the model that gives the highest performance on the development set.
Once the training is done, your best model (on the development set) is tested on the evaluation set, and the results are reported in the paper.
If a new dataset is used, you need to create your own split. If the dataset has:
< 1K instances, use -fold cross-validation for evaluation ().
[1K, 10K] instances, use the 75/10/15 split for the training/development/evaluation sets.
> 10K instances, use the 80/10/10 split.
It is important to create training, development, and evaluation sets that follow similar distributions (in terms of labels, etc.) as the entire data.
Cross-validation is typically used for development, and for evaluation. However, when the data is not sufficiently large, the evaluation set becomes too small, which can cause many variants in the model performance tested on the set. Thus, cross-validation is used to average out the variants.
Once you split the data, create a table describing the distribution and statistics of each set. This table should include all necessary statistics to help researchers understand your experimental results (e.g., how many entities per sentence for named entity recognition, how many questions per document for question answering).
If it is unclear how the data is split from the previous work, contact the authors. If the authors do not respond, create your own split and describe the datasets.
Write the Experiments section in your individual overleaf project.
Recommended length: 100 - 200 lines (including tables and figures).
Submit the PDF version of your current draft up to the Experiments section.
Data Section: are the sources and choices of the datasets reasonably explained? (1 point)
Data Split: are the statistics of training/development/evaluation or cross-validation sets distinctly illustrated? (2 points)
Model Descriptions: are the models designed to soundingly distinguish differences in methods? (2 points)
Evaluation Metrics: are the evaluation metrics clearly explained? (2 points)
Experimental Settings: are the settings described in a way that readers can replicate the experiments? (1 point)
Model Development: is the model development progress depicted? (1 point)
Result Tables: are the experimental results evidently summarized in tables? (2 points)
Result Interpretations: are the key findings from the results convincingly interpreted? (2 points)
This section gives a detailed analysis in performance.
Once you have shown the overall performance of your models in the Experiments section, you need to give a more detailed analysis.
You should consider any quantitative analysis not involving manual annotation in this section that can emphasize the significance of your novel approach. Qualitative analysis is provided in the Error Analysis section.
If your task involves multi-class classifiction, it is good to show how your model performs for each label. Label distributions play an important role in this analysis because most machine learning algorithms tend not to perform well for labels with small distributions. Once you present the table, explain why your model does not perform well on certain labels.
Although the above example shows results from one model, it is better to show results from multiple models in comparisons.
Label analysis can be presented by a confusion matrix if certain pairs of labels are getting confused more often than the others. Once you present the matrix, explain why certain labels are confused more than the others.
If your data can be categorized into meaningful groups (e.g., challenging input, complex output), then provide the model performance on each category and explain why your models perform well (or not well) on certain groups.
If the model speed is one of your contributions, group data with respect to different sizes and explain for which groups your model shows strength. The model speed should be adequately profiled by making sure it is not interrupted by other processes.
You should measure the speed multiple times, remove outliers, and present the average speeds to ensure the robustness.
If the model speed can be measured theoratically by counting involved operations, show the plots and explain the overal trend.
This section provides a qualitative error analysis.
It is important to understand what errors are produced by your model, how often those errors occur on which occasions, and what causes such errors. Error analysis should be done both quantitatively and qualitatively.
Sample instances that your model makes errors for. Focus on data portions that your performance analysis has questions about.
Categorize errors by analyzing them (manually) and pick signature examples.
Provide distributions of the categorized errors in a figure (see below).
Explain why the model makes certian errors with examples.
Write the Analysis section in your individual overleaf project.
Recommended length: 100 - 200 lines (including tables and figures).
Submit the PDF version of your current draft up to the Analysis section.
Performance Analysis: Is the model performance analyzed in-depth (e.g., by labels and categories)? Is the efficiency of the model illustrated (if applicable)? (5 points)
Error Analysis: Are the errors meaningfully categorized, interpreted, and illustrated with examples? (3 points)
Discussions: Are the limitations and future directions clearly described? (2 points)
This section discusses limitations and the future direction of this research.
Once you complete the analysis, you know what your approach can or cannot do for the task. The limitations can be thought of in multiple dimensions:
Technicality: what are the technical challenges for enhancing the performance in terms of accuracy or scalability?
Robustness: how reliable and generalizable are your results?
Applicability: what hinders your approach from being adapted by practical applications?
As the owner of this work, you are responsible for suggesting future research directions to the community. Explain what researchers can do to overcome the limitations.
This section guides you how to write the conclusion and the abstract sections.
It is a common practice to have the main title (with catchy words) describing the overall objective and the secondary title (with specific terms) indicating the key methods.
The abstract is the face of the paper that will get readers interested in your work. Make sure that the writing in the abstract does not seem redundant to the one in the introduction although their contents may overlap.
The abstract is a standalone section such that it should be comprehensive on its own. Also, the introduction should be not a continuation of the abstract; in other words, the introduction should not assume that the reader already read the abstract.
Describe the importance and/or progress of your task:
YOUR TASK has been heavily explored due to its IMPORTANCE.
APPROACH has achieved remarkable results on YOUR TASK.
Indicate the main challenges you address in this work:
However, CHALLENGES still have not been resolved.
Introduce the main gist of your work
This paper presents YOUR APPROACH that uses METHODS.
Describe how your methods can overcome the challenges:
METHOD 1 and METHOD 2 are used to overcome CHALLENGE 1.
METHOD 3 are used to overcome CHALLENGE 2.
Group methods addressing the same challenge and give more specifics if needed.
Illustrate the experimental design:
Our models are experimented on DATASETS.
Summarize the key findings:
Our best MODEL achieves the state-of-the-art accuracy of ##.#% and shows strengths in ASPECTS.
Emphasize the main contribution of the paper:
To the best of our knowledge, it is the first work to ...
Our resource is useful for ...
Write the Conclusion section and the Abstract in your individual overleaf project.
Recommended length: 20 - 40 lines for the conclusion, and 20 - 35 lines for the abstract.
Submit the final draft of your individual writing in PDF.
Summary: Does the conclusion well-summarize the contributions of key methods? (2 points)
Future Work: Are the next steps of this research clearly motivated and described? (2 points)
Abstract: Is the abstract written in a compelling and comprehensive manner? (3 points)
The summary in the conclusion should be distinguished from the abstract and or the overview in the introduction in the sense that it should be conclusive.
First, indicate the overall contribution of your system:
We present YOUR SYSTEM that improves ... YOUR TASK.
Then, conclude the key contributions of specific methods:
Our METHOD shows the state-of-the-art accuracy on ASPECTS.
Our METHOD is found to be more effective in ASPECTS.
If the impact of your work reaches the point that it can make a significant contribution to the field, describe its broader impacts:
Our model allows FIELD to OVERCOME EXISTING CHALLENGES.
Finally, mention how you plan to release relevant resources (if possible):
All our resources including the dataset, models, and source codes are available through our open source project in URL.
The future work in the conclusion should be more explicit than the future directions of discussions in the analysis in the sense that it should specify your next steps of this research.
First, indicate the key challenges that you would address as the next steps:
Despite great peformance, our model still lack in CHALLENGES.
Then, illustrate your plan to handle such challenges:
We plan to tackle CHALLENGES using METHODS.
Check the paper assignments in the comment section of [Grades → Peer Reviews].
Complete reviews for the three assigned papers using this form.
Make sure to fill out the form three times, one for each paper.
Summary: Are the key aspects proficiently pointed out and described (1 point)?
Strengths: Are the major contributions objectively assessed (1.5 points)?
Weaknesses: Are the main issues soundingly addressed (1.5 points)?
Feedback: Are the comments and suggestions helpful (1 point)?
Two types of presentations are often given at conferences, oral and poster.
The oral presentation is provided in a lecture setting where you present slides to a large audience for 10 - 15 minutes followed by questions. Although your interaction with the audience is limited, it gives you an opportunity to appeal your work to the research community and sometimes leads them to contact you later for more discussions.
The cover page must include the followings:
Title (in a large font)
Venue (e.g., CS385: Research Practicum - Emory University)
Location (e.g., Atlanta, GA, USA) and date
Authors and their affiliations (e.g., Jinho D. Choi - Department of Computer Science, Emory University)
Follow the instructions provided by the LaTex team.
The following editors are recommended to write in LaTex:
Overleaf: an online editor that is widely used in collaborative writing.
TeXstudio: an offline editor that supports Windows, Mac OS, and most Linux distributions.
Throughout the guidelines, this template is used to show examples, which is the official template for ACL conferences.
This chapter guides you how to write research papers in LaTex.
Most reputable journals and conference proceedings recommend LaTex for scientific writing. This chapter helps you keep consistent formatting in your writings, which has the following benefits:
Readers can easily follow your content since they do not need to learn a new formatting style.
You will spend less time writing since you already know what formatting you want to use to present the content.
Your writings will look professional since the guidelines suggested here have been used to publish numerous papers.
Our template uses acl.sty
to format the paper, indicated at the top of acl_latex.tex
:
If you replace
final
toreview
, it turns into the anonymous mode.
The standard packages include the followings:
The following packages add useful fonts:
The following packages are recommended:
Labels must not include any space. Use the following prefixes to name labels:
Section: sec:
Subsection: ssec:
Subsubsection: sssec:
Tables (including sub-tables): tab:
Figures (including sub-figures): fig:
Algorithms: alg:
Here are a few examples:
The label of the introduction section → sec:introduction
The label of a subsection describing a decoding strategy → ssec:decoding-strategy
The label of the table showing data statistics → tab:data-stats
acl_latex.tex
: contains the main content.
acl_natbib.bst
: defines the bibliography format.
acl.sty
: defines the paper format.
custom.bib
: contains all references.
Except for acl_latex.tex
, all the other tex files are saved under the tex
folder, making it easier to manage extended contents. There are 9 files under the tex
folder:
abstract.tex
introduction.tex
related-work.tex
approach.tex
experiments.tex
analysis.tex
conclusion.tex
acknowledgements.tex
appndeix.tex
These files should be added in acl_latex.tex
:
Additionally, the img
folder contains all image files included as figures in the paper.
Any preprint must be checked whether or not it has been published to a peer-reviewed venue. If it has, use the reference from the peer-reviewed venue instead of the preprint source such as arXiv.
Keep the following conventions to add entries in the bib file:
- For key
, make sure there is no duplicate.
- For title
, surround the text with curly braces; otherwise, the title will be lowercased in print:
- For booktitle
or journal
, do not use acronyms but the full venue names. For instance, the following is good:
whereas the following is bad:
Also, do not append the acronym in parentheses at the end of the venue name. For instance, remove (ACL)
from the following::
Use the series
field rather to indicate the acronym.
- For series
, use the format acronym'year
(e.g., ACL'20
), where acronym
is the acronym of the venue and year
is the last two digits of the published year.
- For pages
, put two dashes between the first and the last pages (e.g., 1--10
).
Use \citet
when the reference is used in context:
Use \cite
when the reference is used outside of the context:
Use \citealt
when the reference is used inside of parentheses:
Write each sentence in a separate line which makes it easier to comment out selective sentences. For instance, write as follows:
instead of writing as follows:
which allows the following to comment out the second sentence:
Put \noindent
on any paragraph that
Starts at the top of the page,
Follows tables, figures, or algorithm boxes.
Use the following template to include tables in the column:
Use the following options for all tables:
If the table exceeds the column width, put the tabular
inside a \resizebox
instead:
Make sure values in the header row are always center-aligned regardless of the configuration. You can specify this by using \multicolumn
:
Make sure the label starts with the prefix tab:
.
Use the following template to include tables that expand to the full page.
If the table exceeds the page width, use \textwidth
instead of \columnwidth
for \resizebox
.
Use the following template to create sub-tables.
The subtable
environment requires the subcaption
package.
Put \vspace{0.5em}
at the end of every sub-table except for the very last one.
Our includes 4 files on the top level:
Our uses acl_natbib.bst
to format the bibliography, indicated in acl.sty
:
Our uses custom.bib
, is indicated at the bottom of acl_latex.tex
, for adding references to be included in the paper:
- For url
, add the link to the original source of the paper (e.g., ).
See the other explanations for the .
See to create sub-tables that expand to the full page.
It is better to avoid the pronoun "we" in your writing (the possessive pronoun "our" is ok). If you want to give an overview of your paper or approach, use "this paper" or indicate a specific section instead:
We introduce ... -> This paper introduces ...
-> This section introduces ...
We present our transformer model in Section 3.
-> Section 3 presents our transformer model.
Use passive voice to describe your methodology or findings:
We use a transformer model to ...
-> A transformer model is used to ...
We collect 1,000 posts from ...
-> 1,000 posts are collected from ...
You may want to use the pronoun "we" to distinguish discussions or notes that are helpful but do not necessarily contribute to the general knowledge:
Note that we also experimented with ... that did not show a better performance.
Use the present tense to describe your work:
Three models were developed for ...
-> Three models are developed for ...
Use the past tense to describe previous work:
Chen and Choi, 2016 introduced ...
In our preliminary study, an accuracy of 92% was achieved for ...
Before including figures, make sure the followings:
Export all images to PDF format. Converting image files (e.g., PNG) to PDF does not render well; you must export the vectorized images to PDF.
Crop white margins around the image. If you use Mac OS or Linux, you can simply type the following command in a terminal, which will crop image.pdf
and save it to image-crop.pdf
:
Use the following template to include figures in column:
Use the following options for all figures:
The width can be configured proportionally. The following example sets the width to 0.9 * \columnwidth
:
If the figure seems too large, use scale
instead of width
as the option for \includegraphics
:
Make sure the label starts with the prefix fig:
.
Use the following template to include tables that expand to the full page.
Use the \textwidth
option instead of \columnwidth
for \resizebox
.
See the other explanations for the Figures in Column.
Use the following template to create sub-figures.
The subfigure
environment requires the subcaption
package.
Put \vspace{0.5em}
at the end of every sub-figure except for the very last one.
See Figures in Page to create sub-tables that expand to the full page.
This chapter provides a schedule and rubric for progress reports.
Please refer to the syllabus for detailed information about progress reports.
Submit your presentation slides in PDF by the selected date to Canvas.
Motivation (0.5 points): It assesses the clarity and effectiveness of the project's motivation. It examines how well you present the reasons and justifications for undertaking the research.
Objectives (0.5 points): It evaluates the scope and specificity of the project's objectives. It focuses on how well you define the intended outcomes and goals of the research.
Specific Methods (1 point): It examines your explanation and demonstration of the particular methods used in the project. It assesses your ability to articulate the techniques, tools, or approaches employed to address the research objectives.
Experimental Design (1 point): It evaluates the quality and soundness of the experimental design used in the project. It assesses how well you plan and structure the experiments to develop models and achieve the research objectives.
Analysis Plan (0.5 points): It assesses your proposed or implemented analysis plan for the experimental results. It evaluates your ability to outline the techniques and procedures used to analyze and interpret the research findings.
Challenges (1 point): It examines your identification and discussion of challenges encountered during the project. It assesses your capacity to identify potential obstacles and provide strategies or plans to overcome them.
Member Contributions (0.5 points): It evaluates the individual contributions of team members to the progress report. It assesses the extent to which each team member actively participated and contributed to the project.
TBA
TBA
TBA
TBA
TBA
TBA
Present your team project to the class, highlighting its key aspects and demonstrating its value and significance.
Connect with your teammates.
Make promotion slides for a 5 minutes presentation, including the following:
An overview of your project
The motivation behind your project and its broad impact
Key contributions and the novelty of your project
Proposed methods, such as algorithms, models, and data
Anticipated challenges and expected results
Weekly plans that outline individual responsibilities
Submit your presentation slides in PDF to [Assignments → Exercises → Team Promotion] in Canvas (one per team).
Unlocking Market Mysteries: Pushing GPT to its Limit on Stock Market Prediction - Adedoyin Aromolaran, Harry He, Harry Jeon (, , , )
The Political Climate: Speech Identification and the Discourse of Environment - Louis Mullarkey, Latifa Tan (, , , )
Discerning Distress: A Framework for Crisis Identification in Digital Dialogue using LLMs - Darin Kishore, Ellie Paek (, , , )
Back to the Basics: Evaluating Information Retrieval and Generative Models for Chest X-Ray Synthesis - Alex Belov, Karam Khanna (, , , )
Evaluating OpenAI’s Approach to Diversifying DALL-E 3’s Image Generation - Gianluis Hernandez, Fred Hinojosa (, , , )
I picked project 1 because the emoji part of the research gave an interesting result which disapprove some part of the hypothesis. Both parts of the research would be much better if they extend to a bigger scale.
I am fascinated by the authors' dedication to designing the chat box. The huge amount of work put into the chat box is crazy. I also firmly believed in the use of this project. It not only has research usage, but also can be use in business area. I believe this is very important in computer science research since a lot of computer sicence research can be realized in to business. Based on my personal experience with Siri, the results of this research are proven.
I selected this group because their overall research methodologies were sound. They had an advantage because they had already brought in existing research, but even though they were not able to achieve their overarching goal of predicting different diseases using these leads, they followed each step research step thoroughly.
Compared to other projects, their projects are more completed.
Although I am not a biology major, the work is simply incredible for the gan model. As a project manager who manages data, I am fascinated by the ways they show their data. I learnt a lot from them on ways to clean data show data and select data from their research experience.
The area of research sounds very beneficial. The oral presentation seemed that they are very well prepared, and the purpose of the project was explained very well to me during the poster presentation.
Very complete project, with large dataset pulled together to support. Good to see critical analysis of the evaluation step even
Has real life applications and remarkable results
I am choosing this project secondly because a I find it interesting. The findings are not what I would expect and also highlight a chance for improvement in MLM. From their presentation and poster, it sounds like the authors found an area in MLM that has largely been ignored/not properly developed and conducted experiments that identified that problem, proposed two solutions, and then found the resulting best solution. They also incorporated multiple languages into their research which is impressive. While some other projects like could likely have a bigger impact on the research community due to it's involvement in chronic disease and health, this work, I think, was a better overall project because of their research and proposed solutions. I think this work will end up holding more weight within the research community than the other papers because these three actually solved a real problem in MLM despite the time and resource constraints.
I think this project is a well constructed project with interesting result. This research has lots of meanings in terms of the "must do" of basically all NLP researches in the future. Also, the general structure of the research is clear whether in the presentation or on the poster.
Really enjoyed listening to both the slides and poster presentation for this project as the relevance of the contributions was very clear to me—current gender bias metrics are not robust, and it's important to correct these biases for masked language modeling as the applications are widespread (e.g., Google search, YouTube recommendations, autocomplete, etc.). It was also interesting that the work was tested across four Indo-European languages (English, Spanish, Portuguese, German), and I would be curious to see how the results turn out for Asian languages such as Mandarin or Korean—they may be significantly different due to cultural variations and the corpus that these masked language models are trained on.
Very complete project with tons of implicit extensibility that the authors are very aware of and solid basis for the method being applied.
Was easy to understand and novel
I think this project is good because the clearly organized experiment and model design. Although they did not get the desired result that will reflect a sigmoid curve, the future improvement of this project is also clear. In addition, AI + HPC as a research direction is very potential.
I selected this project because I thought their design and methods were very complex and interesting. I think takings risks should be rewarded and because I felt their topic and research was quite advanced and successful, I think they had some of the highest quality.
I picked this project because it provides a novel direction for the existing evaluation matrix. It can be used to improve speech to text into another level!
I believe this group was very sound because they also followed research methodologies soundly. They were able to create their own algorithms and work on a legitimate NLP task that others had not done. Their steps are clear to follow, and they are able to achieve results and move forward in an area that is untouched.
Explanation makes me understand well about their project
For this project, I think the model that created was very informative. They compared the Amazon with another product, and it did generate useful results. The paper is very novel as well.
I could clearly see that they have worked really hard to come this far. Their poster was the best one in the class!
I thought that this project was of high quality because the findings were conveyed in a clear, concise manner. I could easily understand why this topic is important and why their specific contributions with developing a new metric (other than the word error rate) is pivotal, because the big-picture was well-highlighted through both the presentation slides and poster. I also thought that the algorithm described was very innovative and I wonder what the next steps are from here for this group—one option might be working on multiparty speaker diarization, since their data is currently only focused on having two parties right now.
The purpose of the project was explained very well to me. I felt this is an interesting project.
I selected this project because the novelty of creating the TDER evaluation metric, along with implementing the Needleman-Wunsch algorithm and their own 3-d matrix was very unique. I also believe that it was the closest project to being fully completed especially since they achieved results and performances above the other metrics. Furthermore, I believe that it was also presented very concisely when compared to other presentations which also contributes to its rankings.
I selected this project because the model and algorithm this work used was really cool and different. I give special credit to him for doing as much, if not more, work than everyone else for just one person. Both his powerpoint and poster presentation had extremely high quality and delivered.
First of all, the author did the whole paper himself. This is very impressing. His model about assumption was impression. I was fascinated by his result.
I chose this group because their quality is good, given this is a single-person group.
The authors did great work. Their presentation was clean and clear and their poster was very well-made. Furthermore their work has the promise to give major contributions to the field of NLP's and language models. It also highlights interesting contrasts between the performance of the different forms of BERT and I'd love to see their data compared with high school seniors scores on the GER. Simply put, their project has the credibility of novelty and aiding the NLP field incredibly while also being interesting.
They have shown the most progress throughout the semester, and they had a clear, concise oral presentation.
I chose this group due to their research values. They created a dataset for sentence completion tasks, and they evaluated each popular pre-trained model on their dataset, not only giving the quality of their dataset but also evaluating those models' ability in sentence completion tasks.
I selected this project because I believe that their methodology is very sound and that with a larger data set, their methods can produce novel results. I also believe that their work could advance the NLP field (I am no expert though so take this with a grain of salt) with a substantial result because it could hint at the weaknesses of transformer models at performing tasks and thus perhaps hint at what exactly the transformer models capture in their predictions. I also believe that during the poster presentations, they did the best job explaining how their project worked.
This section presents experimental results.
Create a table displaying experimental results from your models on each dataset and evaluation metric. The table should also include results from previous work directly comparable to yours.
If the table is too large (e.g., taking more than 1/3 of the page), it may overwhelm the readers. In this case, shrink it by including only the critical results and put the rest in the appendix.
Here are a few tips for creating the result table:
Expand it to the full page if it consists of many columns.
Use acronyms for the header titles if too long, and explain them in the caption.
If space allows, include both the average scores and standard deviations. The standard deviation is usually notated by the plus-minus sign (e.g., ).
Highlight the key results by making them bold.
Sometimes, it makes more sense to use multiple tables to present your results (e.g., working on multiple tasks), in which case, use a consistent scheme across the tables so they can be easily compared.
Once the result table is presented, you need to give an interpretation of the results. First, summarize the overall observations:
Each model shows an incremental improvement over its predecessor.
MODEL 2 shows a noticeable improvement over MODEL 1, indicating the effectiveness of our METHOD.
The ADVANCED MODEL shows a significant improvement of #.#% from the BASELINE MODEL.
Then, describe any key findings:
It is interesting that MODEL 2 shows better performance over MODEL 1 on DATASET 1 but the results are opposite on DATASET 2.
Give an interpretation for each key finding (and indicate a specific subsection in the Analysis section where further analysis is provided):
It is likely because METHOD works well for ASPECTS in DATASET 1, but not necessairly for ASPECTS in DATASET 2 (Section #.#).
In general, high-level interpretations are provided in the Experiments section whereas more detailed analyses are provided in the Analysis section. These two sections, however, can be merged into one if the space is limited.
Finally, explain any additional results that are not included in the table but help readers interpret this work better:
It it worth mentioning that we also experimented with METHOD 1, which showed a similar result as METHOD 2.
The interpretation should not be simply reading the table. The main goal of this interpretation is to provide insights that are not so obvious to the readers by reading the table, but you learn from the period of this study.
This chapter provides guidance for writing the related work section by conducting a comprehensive literature review and differentiating your work from existing research.
Writing an extensive related work section is important because:
It demonstrates your comprehension of the relevant field.
It provides supporting evidence for your hypothesis.
Reviewers appreciate relevant citations of their work.