• Jinho Choi: jdchoi77

• Talyn Fan: talynfan

• Benjamin Ascoli: bossben

• Sichang Tu:

• Environment: New

• Location: LOCAL_PATH/conversational-ai/venv

• Base interpreter: Python 3.11

Only the following times and dates are available for the specific services:

• Haircut:

  • Monday 10 AM, 1 PM, 2 PM

  • Tuesday: 2 PM

• Hair coloring:

  • Wednesday: 10 AM, 11 AM, 1 PM

  • Thursday: 10 AM, 11 AM

• Perms

  • Friday: 10 AM, 11 AM, 1 PM, 2 PM

  • Saturday: 10 AM, 2 PM

Thus, your system should not schedule an appointment for any other slots:

Task 1

1. Create a python file quiz2.py under the quiz package and copy the code.

2. Update the transitions to design a dialogue flow for the above dialogue system.

Task 2

• Create a PDF file quiz2.pdf that describes the limitations of your system.

Submission

1. Commit and push quiz2.py to your GitHub repository.

2. Submit quiz2.pdf to Canvas.

Resource

• Source code:

Proposal

The proposal writing must be created in LaTex using this template.

• LaTex Guidelines

The proposal should be 6 - 8 pages, including figures and tables (excluding references). Your proposal must include the following:

1. Write an abstract summarizing your proposal.

2. What is the team vision of your dialogue system? What makes your system interesting and important? Is a dialogue system the best practice to pursue your vision?

3. Who is your primary target audience? If your target audience is narrow, what is your strategy to engage the general audience in your system?

4. Describe a high-level dialogue flow of your system. Additionally, provide sample dialogues that you expect your system to conduct.

5. Describe your scientific approach in detail. What techniques and/or datasets will you use, and how do you plan to integrate them into your system?

6. How will you evaluate your system? Be specific about what aspects you plan to assess and how you will conduct such evaluation in terms of metrics and demographics.

7. What is novel about your system? Survey similar existing systems and compare them to show the uniqueness of your system.

8. Provide a weekly timeline specifying which members will work on which parts.

Submissions

• Submit your presentation slides and the report in PDF to Canvas.

Provide your ideas in detail, about 100 words (or more) per question.

Task 2

Group a team of 3-4 members for your project.

Submission

1. Submit quiz1.pdf to Canvas.

2. Go to [People → Team Projects] in Canvas and assign your team members to the same group.

Submission

Submit quiz6.pdf summarizing your evaluation results, including:

• Your evaluation categories and their descriptions.

• Quantitative analysis of your chatbot using the metric.

• Qualitative analysis of your chatbot using assessment reasoning.

• Based on the analyses, potential improvements can (or should) be made.

Report

The proposal writing must be created in LaTex using this template.

• LaTex Guidelines

The report should be 8 - 10 pages, including figures and tables (excluding references). Your proposal must include the following:

1. Abstract: Summarize the report, including the motivation, approach, and results.

2. Vision: Explain what this project aims to accomplish and why that is beneficial to society.

3. Challenges: Describe the main challenges of your project. Describe how your vision, approach, evaluation, etc. may have changed due to the challenges.

4. Dialogue Flow: Describe the overall dialogue flow of your system. Additionally, provide sample dialogues that your system can conduct.

5. Methodology: Illustrate specific methods used in your chatbot (other than the ones built-in STDM), including external APIs, datasets, macros, etc.

6. Evaluation: Describe the demographics and statistics of the users who participated in assessing your chatbot. Explain in detail how you evaluate your chatbot.

7. Novelty: Point out the main novelty of your chatbot against competitors. Survey on related systems similar to yours.

8. Contributions: Depict the contribution of each team member throughout the course of the project and estimate the percentages of individual contributions so that they sum up to 100%.

Submissions

• Submit your presentation slides and the report in PDF to Canvas.

• Efficient Estimation of Word Representations in Vector Space, Mikolov et al., ICLR, 2013. <- Word2Vec

• GloVe: Global Vectors for Word Representation, Pennington et al., EMNLP, 2014.

• Deep Contextualized Word Representations, Ppeters et al., NAACL, 2018. <- ELMo

Transformers

• Attention is All You Need, Vaswani et al., NIPS, 2017. <- Transformer

• Generating Wikipedia by Summarizing Long Sequences, Liu et al., ICLR, 2018.

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin et al., NAACL, 2018.

Tokenization

• Neural Machine Translation of Rare Words with Subword Units, Sennrich et al., ACL, 2016. <- Byte-Pair Encoding (BPE)

• Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation, Wu et al., arXiv, 2016. <- WordPiece

• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, Kudo and Richardson, EMNLP, 2018.

GPT (Generative Pre-trained Transformer)

• Improving Language Understanding by Generative Pre-Training, Radford et al., OpenAI, 2018. <- GPT-1

• Language Models are Unsupervised Multitask Learners, Radford et al., OpenAI, 2019. <- GPT-2

• Language Models are Few-Shot Learners, Brown et al., NeurIPS, 2020. <- GPT-3

• Create a python file quiz5.py under the quiz package and copy the code.

• Update the code to design a dialogue flow for the assigned dialogue system.

• Create a PDF file quiz5.pdf that describes the approach (e.g., prompt engineering) and how the large language model improved over the limitations you described in Quiz 2.

Task 2

Answer the following questions in quiz5.py:

1. What are the limitations of the Bag-of-Words representation?

2. Describe the Chain Rule and Markov Assumption and how they are used to estimate the probability of a word sequence.

3. Explain how the Word2Vec approach uses feed-forward neural networks to generate word embeddings. What are the advantages of the Word2Vec representation over the Bag-of-Words representation?

4. Explain what patterns are learned in the multi-head attentions of a Transformer. What are the advantages of the Transformer embeddings over the Word2Vec embeddings?

• Emora STDM: A Versatile Framework for Innovative Dialogue System Development, James D. Finch and Jinho D. Choi, Proceedings of the Annual Meeting of the Special Interest Group on Discourse and Dialogue: System Demonstrations (SIGDIAL:DEMO), 2020.

• Emora STDM GitHub Repository

• Whether the advice provided helps the user feel more confident in their conversational abilities.

• How well it provides mental health support to individuals with varying levels of stress.

• Whether the user found the previous interaction helpful.

• Whether the chatbot was able to successfully complete her goal with the help of the chatbot.

• The user's rating of how much the chatbot helped them in their career path.

• How users feel about its ability to prepare them for real interviews.

Satisfactory

The degree to which the chatbot meets users' expectations and provides a satisfactory experience.

• Whether the insights, questions, and communication are well-balanced and whether the users feel comfortable with the ideal proportion.

• How comfortable the user felt during the conversation.

• Whether it understood the user's needs and provided personalization based on their style and preferences.

• Whether the user felt a bit better after the interaction.

• The pleasantness of the system persona and whether it meets users' expectations.

Correctness

The degree to which the chatbot provides accurate and correct information or recommendations to users.

• How accurately it identifies the presence of stress and changes response style accordingly.

• Whether the bot's suggestions were relevant to the user's personal taste, style, and preferences, and whether the user would wear everything suggested by the bot.

• The accuracy of its performance metrics, such as the accuracy of style recommendations.

• Whether it is successful in making accurate recommendations for the user.

• The quality and specificity of the feedback users received from the system, and whether the feedback provided accurate and correct information.

Interpretability

The degree to which the chatbot's responses and actions can be understood and interpreted by users.

• Whether users can understand what the chatbot is saying.

• Whether users experience instances of missed messages, misunderstandings, or abrupt exits, and whether the chatbot's responses can be interpreted clearly by users.

• Its ability to understand the context of users' inquiries and provide relevant responses that can be understood and interpreted by users.

• Whether the chatbot can assess the user's level of knowledge and adjust its responses accordingly, whether it uses language that the user can understand, and whether the user feels engaged with the chatbot's insights.

Coherent

The degree to which the chatbot's responses and actions are logically consistent and connected with each other, and with the context of the conversation.

• Whether it can recognize the topics mentioned by the user and provide relevant information that is logically connected with those topics.

• Whether its responses are self-explanatory and related to each other in a logical and coherent manner.

• Its level of consistency with actual interviews, i.e., whether its responses are logically consistent with the expectations and norms of real-life interviews.

• Whether its suggestions are relevant and helpful for the given topic, and whether they are logically consistent with the context of the conversation.

Natural

The degree to which the chatbot's responses and actions resemble those of a human being, and are perceived as natural, fluent, and realistic by users.

• How closely its responses resemble those of a human being, i.e., whether they are naturalistic in terms of language, syntax, grammar, tone, and other linguistic aspects.

Ethical

The degree to which the chatbot's responses and actions adhere to ethical principles and standards, and avoid causing harm or offense to users or other stakeholders.

• Whether it contains any content or responses that are offensive or inappropriate, such as hate speech, discrimination, or harassment.

Empathetic

The degree to understand and respond to the emotions and feelings of users in a compassionate and sensitive manner, and to provide emotional support or encouragement when appropriate.

• Whether it demonstrates an understanding of the user's emotions and concerns, and responds in a way that is supportive and validating.

• Using appropriate language and tone, acknowledging the user's perspective, and offering words of encouragement or empathy.

Informative

The degree to provide accurate and useful information to users in response to their queries or requests for assistance.

• Its ability to provide factual information about fitness, such as exercise techniques, workout routines, or nutrition advice.

• Assessing the accuracy and relevance of the information provided, as well as the chatbot's ability to understand and respond appropriately to the user's specific needs and goals.

Engaging

The degree to capture and hold the user's attention, as well as to create a positive and enjoyable user experience.

• The user's level of interest and involvement during the conversation.

• Assessing the chatbot's ability to generate interesting and relevant topics of conversation, to respond in a timely and personalized manner, and to use engaging language and visual elements to create a more immersive and interactive experience.

Personalization

The degree to tailor its responses and recommendations to the individual user's preferences, needs, and past interactions.

• Evaluate the chatbot's ability to personalize advice based on the user's preferences and past interactions.

• Whether the chatbot can effectively use data about the user, such as their history, feedback, and stated preferences, to provide personalized responses and recommendations. The following example illustrates this concept:

Your chatbot should give an appropriate response to every user response. For this assignment, you must use all of the following:

• An ontology covering common movie genres and a branch of movies that you target,

• At least one macro,

• At least one regular expression (can be used inside a macro).

Task 1

1. Create a Python file quiz3.py under the quiz package.

2. Create a JSON file ontology_quiz3.json under the resources directory.

3. Update them to design a dialogue flow for the chatbot.

Task 2

Create a PDF file quiz3.pdf that describes the following:

• Sample dialogues that your chatbot can conduct.

• Explanations of how the ontology, macro(s), and regular expression(s) are used for contextual understanding in your chatbot.

Submission

1. Commit and push quiz3.py to your GitHub repository.

2. Submit quiz3.pdf to Canvas.

Let us create a JSON file, ontology_animal.json, containing an ontology of animals:

• #2: the key ontology is paired with a dictionary as a value.

• #3: the key animal represents the category, and its subcategories are indicated in the list.

• #4-6: each subcategory, mammal, reptile, and amphibian, has its own subcategory.

• #7: the ontology hierarchy: animal -> mammal -> dog.

Given the ontology, the above transitions can be rewritten as follow:

• #4: matches the key "mammal" as well as its subcategories: "dog", "ape", and "rat".

• #5: matches the key "reptile" as well as its subcategories: "snake" and "lizard".

• #6: matches the key "amphibian" as well as its subcategories: "frog" and "salamander".

Expression

It is possible that a category is mentioned in a non-canonical way; the above conditions do not match "puppy" because it is not introduced as a category in the ontology. In this case, we can specify the aliases as "expressions":

• #10: the key expressions is paired with a dictionary as a value.

• #4: allows matching "canine" and "puppy" for the dog category.

Once you load the updated JSON file, it now understands "puppy" as an expression of "dog":

Variable

Values matched by the ontology can also be stored in variables:

• #4,7,10: the matched term gets stored in the variable FAVORITE_ANIMAL.

• #5,8,11: the system uses the value of FAVORITE_ANIMAL to generate the response.

Loading

The custom ontology must be loaded to the knowledge base of the dialogue flow before it runs:

• #1: loads the ontology in ontology_animal.json to the knowledge base of df.

Code Snippet

Notice that when the user input does not match any of the conditions, it prints the error message and loops back to the start state.

Custom States

It is possible to name any transition you create as a state:

• #6: names the state as time.

• #9: names the state as weather.

• #13: names the state as play_raining_tacos.

• #17: references to the play_raining_tacos state.

Exit State

State referencing can be abused to create transitions that never end (infinite loop). It is important to design an exit state so you can terminate the conversation without throwing errors:

• #6,10,14,17: loops back to the start state.

• #19-21: creates an exit state to terminate the dialogue.

The above transitions can handle the following inputs (on top of the cases of good and bad):

Partial Matching

Matching the entire input is often too restrictive. Let us use a list to allow partial matching:

• #4: the keyword 'good' is surrounded by the square brackets (good -> [good]), and matched to any word in the input.

• #7: the keyphrase 'could be better' is surrounded by the square brackets, and matched to any phrase in the input.

The above transitions yield the following dialogues:

Nesting

To allow partial matching of multiple expressions, a set can be put in a list:

• #4: the keywords 'good' and 'fantastic' are first put in a set for multiple matching and then in a list for partial matching.

• #7: the keyword 'bad' and the keyphrase 'could be better' are first put in a set for multiple matching, then in a list for partial matching.

Sequential Matching

Keywords or keyphrases delimited by commas in a list are matched sequentially:

• #10: first matches 'how', then matches 'you' sequentially so that "how" must be followed by "you" in the input for this condition to be true.

All terms in the list must be matched sequentially such that the conditions are false for the following cases:

Let us replace 'how' with '{how, and}' and 'you' with '{you, going}' in #10:

• #10: first matches ('how' or 'and'), then matches ('you' or 'going') sequentially.

Code Snippet

Currently, the run method returns True no matter what the input is.

Integration

Let us create transitions using this macro. A macro is represented by an alias preceded by the pound sign (#):

• #4: calls the macro #GET_NAME that is an alias of MacroGetName.

• #13: creates a dictionary defining aliases for macros.

• #14: creates an object of MacroGetName and saves it to the alias GET_NAME.

To call the macro, we need to add the alias dictionary macros to the dialogue flow:

• #3: adds all macros defined in macros to the dialogue flow df.

Parameters

The run method has three parameters:

• ngrams: is a set of strings representing every n-gram of the input matched by the Natex.

• vars: is the variable dictionary, maintained by a DialogueFlow object, where the keys and values are variable names and objects corresponding to their values.

• args: is a list of strings representing arguments specified in the macro call.

Let us modify the run method to see what ngrams and vars give:

• #2: prints the original string of the matched input span before preprocessing.

• #3: prints the input span, preprocessed by STDM and matched by the Natex.

• #4: prints a set of n-grams.

When you interact with the the dialogue flow by running it (df.run()), it prints the followings:

The raw_text method returns the original input:

The text method returns the preprocessed input used to match the Natex:

The ngrams gives a set of all possible n-grams in text():

Finally, the vars gives a dictionary consisting of both system-level and user-custom variables (no user-custom variables are saved at the moment):

Implementation

Let us update the run method that matches the title, first name, and last name in the input and saves them to the variables $TITLE, $FIRSTNAME, and $LASTNAME, respectively:

• #2: creates a regular expression to match the title, first name and last name.

• #3: searches for the span to match.

• #4: returns False if no match is found.

• #6-18 -> exercise.

• #20-22: saves the recognized title, first name, and last name to the corresponding variables.

• #24: returns True as the regular expression matches the input span.

Given the updated macro, the above transitions can be modified as follow:

• #5: uses the variables $FIRSTNAME and $LASTNAME retrieved by the macro to generate the output.

The followings show outputs:

Matching

Natex supports several ways of matching the input with key terms.

Term

The condition is true if the input exactly matches the term. A term is represented as a string and can have more than one token:

• #4: matches the input with 'could be better'.

• #7: error is a reserved term indicating the default condition of this conditional branching, similar to the wildcard condition (_) in a match statement.

Set

The condition is true if the input exactly matches any term in the set. A set is represented by curly brackets ({}):

• #7: matches the input with either 'good' or 'not bad'.

Unordered List

The condition is true if some terms in the input match all terms in the unordered list, regardless of the order. An unordered list is represented by angle brackets (<>):

• #10: matches the input with both 'very' and 'good' in any order.

Ordered List

The condition is true if some terms in the input match all terms in the ordered list, a.k.a. sequence, in the same order. An ordered list is represented by square brackets ([]):

• #13: matches the input with both 'so' and 'good' in that order.

Rigid Sequence

The condition is true if all terms in the input exactly match all terms in the rigid sequence in the same order. A rigid sequence is represented by square brackets ([ ]), where the left bracket is followed by an exclamation mark (!):

#16: matches the input with both 'hello' and 'world' in that order.