CSCI 5980, NLP with Deep Learning

Fall 2022, Tuesday and Thursday, 9:45am to 11:00am, Bruininks Hall 119

Course Information

Natural Language Processing (NLP) is a core component of artificial intelligence (AI). Deep learning approaches have shown dramatic improvements on various NLP tasks for the last years. The course will provide students with a comprehensive introduction to neural network models for NLP, as well as practical skills to implement them. There will be a semester-long class project where your team will develop a deep learning model on NLP tasks and conduct an in-depth evaluation on the failure cases.

8980 vs 5980: Both lectures use the similar lecture content but they have different focuses; 5980 focuses on more "processing" parts of NLP, particularly with deep learning methods. Students will gain an instruction to cutting-edge techniques in deep learning for NLP. On the other hand, 8980 (Introduction to NLP Research) covers broad aspects of NLP research as an interdisciplinary problem, including data annotation, error analysis, and applications to different fields.

All class material will be posted on the class page. We will use Canvas for homework submissions and grading, and Slack for discussion and QA so other students who may have the same question can get some help from your questions.

Instructor: Dongyeop Kang (a.k.a DK)
Class meets: Tuesday and Thursday, 9:45am to 11:00am, Bruininks Hall 119
Office hours: Friday, 3:30pm to 4:30pm in Shepherd 259
Class page: dykang.github.io/classes/csci5980/Fall2022
Slack: csci5980-01f22.slack.com/
Canvas: canvas.umn.edu/courses/333230

Grading and Late Policy

Grading

30% Homeworks (total three homeworks)
50% Final Project
10% Summary of guest lectures
10% (potential bonus) Class Participation

Active participation in class discussions and project presentations

Late policy for deliverables

Each student will be granted 3 late days to use for homeworks over the duration of the semester. After all free late days are used up, penalty is 25% for each additional late day. However, projects submitted late after all late days have been used will receive no credit.

Schedule

We will cover basic and advanced deep learning models for NLP, with some practical sessions.
Pleaes pay attention to due dates (every due is by 11:59PM) and project presentations. 🍬 is an optional reading.

Date	Topic	Readings
W1: Sep 6	Class overview [slides]
W1: Sep 8	Intro to NLP [slides]
W2: Sep 13	Text Classification [slides]	Determining the sentiment of opinions Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank Text classifier with NLTK and Scikit-Learn
W2: Sep 15	Neural Nets and Backpropagation [slides]	Yes you should understand backprop Natural Language Processing (Almost) from Scratch Learning Representations by Backpropagating Errors (seminal Rumelhart et al. backpropagation paper) Derivatives, Backpropagation, and Vectorization
W3: Sep 20	Practice: Building a Neural Net using Toolkits [slides]
W3: Sep 22	Projects details: Custom and Default; Tips [slides] HW1 out [link]
W4: Sep 27	No class (3M Symposium)
W4: Sep 29	Distributional semantics and word vectors [slides]	From Frequency to Meaning: Vector Space Models of Semantics Efficient Estimation of Word Representations in Vector Space Linguistic Regularities in Continuous Space Word Representations GloVe: Global Vectors for Word Representation Retrofitting Word Vectors to Semantic Lexicons WordNet
W5: Oct 4	Contextualized Word Embeddings (1) [slides]	Deep contextualized word representations BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding A Primer in BERTology: What we know about how BERT works
W5: Oct 6	Contextualized Word Embeddings (2) [slides]	Deep contextualized word representations BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding A Primer in BERTology: What we know about how BERT works
W6: Oct 11	Project Proposal A	Team Semantish (Faizel Khan, Seva Arkhipenka): Choosing the right experts for BioMedical QA tasks Team HSC01 (Sahil Raina, Harshavardhan Battula, Cody Wang): Self Supervision in Graph Neural Networks for Text Classification Team Novella (Kyler Sood, Anush Kumar, Riley O'Neill): Author Imitation Learning
W6: Oct 13	Project Proposal B Project proposal report due HW1 due	Team Sprite (Chen Jiang, Tianhao Zhang, Zhecheng Sheng): Controllable Long-Dialogue Generation Team Polyglots (Asal Shavandi, Chahyon Ku, London Lowmanstone, Josh Spitzer-Resnick): MultiCoNER II: Multilingual Complex Named Entity Recognition Team NLP Amateur (Chen Hu, Jiaqi Liu, Keyang Xuan): Cross-lingual Transfer Learning for Irony Detection
W7: Oct 18	Language models, Fancy RNNs, Seq2Seq [slides]	A Neural Probabilistic Language Model Long Short-Term Memory On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
W7: Oct 20	Deep dive on Transformers [slides]	Attention is All you Need Scaling Laws for Neural Language Models Tutorial on Illustrated Transformer
W8: Oct 25	Pretrainig Paradigm [slides]	Language Models are Unsupervised Multitask Learners Language Models are Few-Shot Learners BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Neural machine translation of rare words with subword units
W8: Oct 27	Prompting [slides] HW2 out [link]	Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing Calibrate Before Use: Improving Few-Shot Performance of Language Models Prefix-Tuning: Optimizing Continuous Prompts for Generation
W9: Nov 1	Advanced Search Algoritms [slides]	The Curious Case of Neural Text Degeneration On the Properties of Neural Machine Translation: Encoder–Decoder Approaches Mutual Information and Diverse Decoding Improve Neural Machine Translation An Actor-Critic Algorithm for Sequence Prediction
W9: Nov 3	CNNs, Recursive Nets, Trees [slides]	Constituency and Dependency Parsing, Jurafsky and Martin Chapter 13 and 15-15.4 Convolutional Neural Networks for Sentence Classification Parsing with Compositional Vector Grammars Constituency Parsing with a Self-Attentive Encoder
W10: Nov 8	Graph Neural Nets [slides]	Semi-Supervised Classification with Graph Convolutional Networks Graph Attention Networks Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling Do Transformers Really Perform Bad for Graph Representation? Pure Transformers are Powerful Graph Learners
W10: Nov 10	Deep Generative Models (VAE, GAN, Diffusion) [slides]	Auto-Encoding Variational Bayes Generating Sentences from a Continuous Space Generative Adversarial Networks SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient Denoising Diffusion Probabilistic Models Diffusion-LM Improves Controllable Text Generation
W11: Nov 15	Structured Prediction [slides]	An Introduction to Conditional Random Fields -- Sections 2-2.6.1, pages 272-302 (Sutton and McCallum 2011) Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data Bidirectional LSTM-CRF Models for Sequence Tagging
W11: Nov 17	Final Project Presentation A Project midterm report due HW2 due	Team Sprite (Chen Jiang, Tianhao Zhang, Zhecheng Sheng): Controllable Long-Dialogue Generation Team Polyglots (Asal Shavandi, Chahyon Ku, London Lowmanstone, Josh Spitzer-Resnick): MultiCoNER II: Multilingual Complex Named Entity Recognition Team NLP Amateur (Chen Hu, Jiaqi Liu, Keyang Xuan): Cross-lingual Transfer Learning for Irony Detection
W12: Nov 22	Final Project Presentation B HW3 out [link]	Team Semantish (Faizel Khan, Seva Arkhipenka): Choosing the right experts for BioMedical QA tasks Team HSC01 (Sahil Raina, Harshavardhan Battula, Cody Wang): Self Supervision in Graph Neural Networks for Text Classification Team Novella (Kyler Sood, Anush Kumar, Riley O'Neill): Author Imitation Learning
W12: Nov 24	No class (Thanksgiving)
W13: Nov 29	Guest lecture recording (NeurIPS) [video] 1-page summary on guest lecture
W13: Dec 1	Guest lecture recording (NeurIPS) [video] 1-page summary on guest lecture
W14: Dec 6	No class (EMNLP)
W14: Dec 8	No class (EMNLP) Final project report due HW3 due

Project Details (50%)

Please refer to the [project description slides] on Sep 22. A course project would be one of the following types:

New research results judged suitable for acceptance to a top NLP or ML conference like ACL/EMNLP/NeurIPS/ICLR,
Evaluation and critical analysis of existing work on a new dataset,
An in-depth literature survey, or
New open-source repository or dataset with a high impact on the community

Every group member should submit their report, link to code (or a zipped code), and presentation slides on Canvas before the deadline. For both proposal and final reports, please use official ACL style templates (Overleaf or links). Note that your report and slides would be publicly shared on this page.

Your project will be evaluated in the following criteria:

Proposal report (Due: Oct 13) and presetnation (Due: Oct 11 / 13) (10%)

Maximum 3 page report and 10-min presentation including 3-min QA in class
Upload your PDF on Canvas
Upload your slides here before your presentation
Expected content in the report:
- Title, members (2-3 students per group) and role assignment, motivation, literature survey, problem formulation, and broader impact

Midterm report (10%), Due: Nov 17

Maximum 8 page report
Upload your PDF on Canvas
Expected content to be presented:
- Specific feedback you like to get from audience
- Motivation
- Problem definition
- Novel contribution compared to prior work
- Proposed methods
- Preliminary results
- Plan until the end of semester

Final presentation (10%), Due: Nov 17 / 22

15-min presnetation and 5-min QA
Upload your slides here before your presentation
Expected content to be presented:
- Motivation, problem definition, and novel contribution compared to prior work
- Proposed methods with "motivational examples"
- Experimental setups and final results
- Error analysis
- Discussion on limitations, ethical issues, and future directions

Final report and code (20%), Due: Dec 8

Maximum 8 pages with unlimited reference and appendix
Rubrick for evaluation

Prerequisites

CSCI 5521 Introduction to Machine Learning or any other course that covers fundamental machine learning algorithms. CSCI 8980 Introduction to NLP Research or CSCI 5541 Natural Langauge Prorcessing

Furthermore, this course assumes:

Good coding ability, corresponding to at least a third or fourth-year undergraduate CS major. Assignments will be in Python.
Background in basic probability, linear algebra, and calculus.

Notes to students

Academic Integrity

Assignments and project reports for the class must represent individual effort unless group work is explicitly allowed. Verbal collaboration on your assignments or class projects with your classmates and instructor is acceptable. But, everything you turn in must be your own work, and you must note the names of anyone you collaborated with on each problem and cite resources that you used to learn about the problem. If you have any doubts about whether a particular action may be construed as cheating, ask the instructor for clarification before you do it. Cheating in this course will result in a grade of F for course and the University policies will be followed.

Students with Disabilities

If you have a disability for which you are or may be requesting an accommodation, you are encouraged to contact both your instructor and Disability Resources Center (DRC).

COVID-19

All students are expected to abide by campus policies regarding COVID-19 including masking and vaccination requirements. This is an in-person class with daily in-person activities, but we may consider a hybrid or online option. If you're feeling sick, stay at home and catch up with the course materials instead of coming to class!

Book

Textbook is not required but the following books are primarily referred:

Jurafsky and Martin, Speech and Language Processing, 3rd edition [online]
Jacob Eisenstein. Natural Language Processing

Resources

The course materials are inspired by the slides of Chris Manning at Stanford, David Bamman at UC Berkeley, and Graham Neubig at CMU.

Natural Language Processing with Deep Learning, Chris Manning, Stanford
Neural Networks for NLP, Graham Neubig, CMU