Research Practicum in Artificial Intelligence
Jinho D. Choi
  • Overview
    • Syllabus
    • Schedule
    • Discussions
  • Speed Dating
    • Profiles
  • Faculty Interests
    • AI Faculty
  • Research Areas
    • AI Conferences
  • Task Selection
  • Introduction
    • Motivation
    • Overview
    • Exercise
  • Related Work
    • Literature Review
    • Exercise
  • Approach
    • Algorithm Development
    • Model Design
    • Data Creation
  • Research Challenges
  • Experiments
    • Datasets
    • Models
    • Results
    • 5.4. Homework
  • Analysis
    • Performance Analysis
    • Error Analysis
    • Discussions
    • 6.4. Homework
  • Conclusion & Abstract
    • Conclusion
    • Title & Abstract
  • Peer Review
  • Presentations
  • Team Projects
    • Fall 2023
    • Fall 2022
  • Assignments
    • HW1: Speed Dating
    • HW2: Research Areas
    • HW3: Team Promotion
    • HW4: Introduction
    • HW5: Related Work
    • HW6: Approach
    • HW7: Experiments
    • HW8: Analysis
    • HW9: Conclusion & Abstract
    • HW10: Peer Review
    • Team Project
  • Supplementary
    • LaTex Guidelines
      • Getting Started
      • File Structure
      • Packages
      • References
      • Paragraphs
      • Labels
      • Tables
      • Figures
      • Lists
    • Writing Tips
    • Progress Reports
    • Team Promotion
Powered by GitBook
On this page
  • By Labels
  • By Categories
  • Speed Analysis
Export as PDF
  1. Analysis

Performance Analysis

This section gives a detailed analysis in performance.

PreviousAnalysisNextError Analysis

Last updated 8 months ago

Once you have shown the overall performance of your models in the section, you need to give a more detailed analysis.

You should consider any quantitative analysis not involving manual annotation in this section that can emphasize the significance of your novel approach. Qualitative analysis is provided in the section.

By Labels

If your task involves multi-class classifiction, it is good to show how your model performs for each label. Label distributions play an important role in this analysis because most machine learning algorithms tend not to perform well for labels with small distributions. Once you present the table, explain why your model does not perform well on certain labels.

Although the above example shows results from one model, it is better to show results from multiple models in comparisons.

Label analysis can be presented by a confusion matrix if certain pairs of labels are getting confused more often than the others. Once you present the matrix, explain why certain labels are confused more than the others.

By Categories

If your data can be categorized into meaningful groups (e.g., challenging input, complex output), then provide the model performance on each category and explain why your models perform well (or not well) on certain groups.

Speed Analysis

If the model speed is one of your contributions, group data with respect to different sizes and explain for which groups your model shows strength. The model speed should be adequately profiled by making sure it is not interrupted by other processes.

You should measure the speed multiple times, remove outliers, and present the average speeds to ensure the robustness.

If the model speed can be measured theoratically by counting involved operations, show the plots and explain the overal trend.

Experiments
Error Analysis
Excerpted from .
Excerpted from .
Excerpted from .
Excerpted from .
Excerpted from .
Vaidya et al., LAW, 2011
Li et al., EMNLP, 2020
He and Choi, IWPT, 2021
Choi and Palmer, ACL, 2011
Choi and McCallum, ACL, 2013