6.2. Team Evaluation
Criteria
Effective
The degree to which the chatbot is able to achieve its intended goals and produce desirable outcomes for users.
Whether the advice provided helps the user feel more confident in their conversational abilities.
How well it provides mental health support to individuals with varying levels of stress.
Whether the user found the previous interaction helpful.
Whether the chatbot was able to successfully complete her goal with the help of the chatbot.
The user's rating of how much the chatbot helped them in their career path.
How users feel about its ability to prepare them for real interviews.
Satisfactory
The degree to which the chatbot meets users' expectations and provides a satisfactory experience.
Whether the insights, questions, and communication are well-balanced and whether the users feel comfortable with the ideal proportion.
How comfortable the user felt during the conversation.
Whether it understood the user's needs and provided personalization based on their style and preferences.
Whether the user felt a bit better after the interaction.
The pleasantness of the system persona and whether it meets users' expectations.
Correctness
The degree to which the chatbot provides accurate and correct information or recommendations to users.
How accurately it identifies the presence of stress and changes response style accordingly.
Whether the bot's suggestions were relevant to the user's personal taste, style, and preferences, and whether the user would wear everything suggested by the bot.
The accuracy of its performance metrics, such as the accuracy of style recommendations.
Whether it is successful in making accurate recommendations for the user.
The quality and specificity of the feedback users received from the system, and whether the feedback provided accurate and correct information.
Interpretability
The degree to which the chatbot's responses and actions can be understood and interpreted by users.
Whether users can understand what the chatbot is saying.
Whether users experience instances of missed messages, misunderstandings, or abrupt exits, and whether the chatbot's responses can be interpreted clearly by users.
Its ability to understand the context of users' inquiries and provide relevant responses that can be understood and interpreted by users.
Whether the chatbot can assess the user's level of knowledge and adjust its responses accordingly, whether it uses language that the user can understand, and whether the user feels engaged with the chatbot's insights.
Coherent
The degree to which the chatbot's responses and actions are logically consistent and connected with each other, and with the context of the conversation.
Whether it can recognize the topics mentioned by the user and provide relevant information that is logically connected with those topics.
Whether its responses are self-explanatory and related to each other in a logical and coherent manner.
Its level of consistency with actual interviews, i.e., whether its responses are logically consistent with the expectations and norms of real-life interviews.
Whether its suggestions are relevant and helpful for the given topic, and whether they are logically consistent with the context of the conversation.
Natural
The degree to which the chatbot's responses and actions resemble those of a human being, and are perceived as natural, fluent, and realistic by users.
How closely its responses resemble those of a human being, i.e., whether they are naturalistic in terms of language, syntax, grammar, tone, and other linguistic aspects.
Ethical
The degree to which the chatbot's responses and actions adhere to ethical principles and standards, and avoid causing harm or offense to users or other stakeholders.
Whether it contains any content or responses that are offensive or inappropriate, such as hate speech, discrimination, or harassment.
Empathetic
The degree to understand and respond to the emotions and feelings of users in a compassionate and sensitive manner, and to provide emotional support or encouragement when appropriate.
Whether it demonstrates an understanding of the user's emotions and concerns, and responds in a way that is supportive and validating.
Using appropriate language and tone, acknowledging the user's perspective, and offering words of encouragement or empathy.
Informative
The degree to provide accurate and useful information to users in response to their queries or requests for assistance.
Its ability to provide factual information about fitness, such as exercise techniques, workout routines, or nutrition advice.
Assessing the accuracy and relevance of the information provided, as well as the chatbot's ability to understand and respond appropriately to the user's specific needs and goals.
Engaging
The degree to capture and hold the user's attention, as well as to create a positive and enjoyable user experience.
The user's level of interest and involvement during the conversation.
Assessing the chatbot's ability to generate interesting and relevant topics of conversation, to respond in a timely and personalized manner, and to use engaging language and visual elements to create a more immersive and interactive experience.
Personalization
The degree to tailor its responses and recommendations to the individual user's preferences, needs, and past interactions.
Evaluate the chatbot's ability to personalize advice based on the user's preferences and past interactions.
Whether the chatbot can effectively use data about the user, such as their history, feedback, and stated preferences, to provide personalized responses and recommendations. The following example illustrates this concept:
Last updated
