This section discusses limitations and the future direction of this research.
Once you complete the analysis, you know what your approach can or cannot do for the task. The limitations can be thought of in multiple dimensions:
Technicality: what are the technical challenges for enhancing the performance in terms of accuracy or scalability?
Robustness: how reliable and generalizable are your results?
Applicability: what hinders your approach from being adapted by practical applications?
As the owner of this work, you are responsible for suggesting future research directions to the community. Explain what researchers can do to overcome the limitations.
This chapter guides you how to write the analysis section
This section aims to provide an in-depth look at what your models can and cannot do so that researchers can make informed decisions in the following step and the best practice in applications.
Write the Analysis section in your individual overleaf project.
Recommended length: 100 - 200 lines (including tables and figures).
Submit the PDF version of your current draft up to the Analysis section.
Performance Analysis: Is the model performance analyzed in-depth (e.g., by labels and categories)? Is the efficiency of the model illustrated (if applicable)? (5 points)
Error Analysis: Are the errors meaningfully categorized, interpreted, and illustrated with examples? (3 points)
Discussions: Are the limitations and future directions clearly described? (2 points)
This section provides a qualitative error analysis.
It is important to understand what errors are produced by your model, how often those errors occur on which occasions, and what causes such errors. Error analysis should be done both quantitatively and qualitatively.
Sample instances that your model makes errors for. Focus on data portions that your performance analysis has questions about.
Categorize errors by analyzing them (manually) and pick signature examples.
Provide distributions of the categorized errors in a figure (see below).
Explain why the model makes certian errors with examples.
This section gives a detailed analysis in performance.
Once you have shown the overall performance of your models in the Experiments section, you need to give a more detailed analysis.
You should consider any quantitative analysis not involving manual annotation in this section that can emphasize the significance of your novel approach. Qualitative analysis is provided in the Error Analysis section.
If your task involves multi-class classifiction, it is good to show how your model performs for each label. Label distributions play an important role in this analysis because most machine learning algorithms tend not to perform well for labels with small distributions. Once you present the table, explain why your model does not perform well on certain labels.
Although the above example shows results from one model, it is better to show results from multiple models in comparisons.
Label analysis can be presented by a confusion matrix if certain pairs of labels are getting confused more often than the others. Once you present the matrix, explain why certain labels are confused more than the others.
If your data can be categorized into meaningful groups (e.g., challenging input, complex output), then provide the model performance on each category and explain why your models perform well (or not well) on certain groups.
If the model speed is one of your contributions, group data with respect to different sizes and explain for which groups your model shows strength. The model speed should be adequately profiled by making sure it is not interrupted by other processes.
You should measure the speed multiple times, remove outliers, and present the average speeds to ensure the robustness.
If the model speed can be measured theoratically by counting involved operations, show the plots and explain the overal trend.
