CSCI 5980, NLP with Deep Learning

Fall 2022, Tuesday and Thursday, 9:45am to 11:00am, Bruininks Hall 119


Course Information


Natural Language Processing (NLP) is a core component of artificial intelligence (AI). Deep learning approaches have shown dramatic improvements on various NLP tasks for the last years. The course will provide students with a comprehensive introduction to neural network models for NLP, as well as practical skills to implement them. There will be a semester-long class project where your team will develop a deep learning model on NLP tasks and conduct an in-depth evaluation on the failure cases.

8980 vs 5980: Both lectures use the similar lecture content but they have different focuses; 5980 focuses on more "processing" parts of NLP, particularly with deep learning methods. Students will gain an instruction to cutting-edge techniques in deep learning for NLP. On the other hand, 8980 (Introduction to NLP Research) covers broad aspects of NLP research as an interdisciplinary problem, including data annotation, error analysis, and applications to different fields.

All class material will be posted on the class page. We will use Canvas for homework submissions and grading, and Slack for discussion and QA so other students who may have the same question can get some help from your questions.

Instructor
Dongyeop Kang (a.k.a DK)
Class meets
Tuesday and Thursday, 9:45am to 11:00am, Bruininks Hall 119
Office hours
Friday, 3:30pm to 4:30pm in Shepherd 259

Class page
dykang.github.io/classes/csci5980/Fall2022
Slack
csci5980-01f22.slack.com/
Canvas
canvas.umn.edu/courses/333230

Grading and Late Policy


Grading

  • 30% Homeworks (total three homeworks)
  • 50% Final Project
  • 10% Summary of guest lectures
  • 10% (potential bonus) Class Participation
    • Active participation in class discussions and project presentations

Late policy for deliverables

Each student will be granted 3 late days to use for homeworks over the duration of the semester. After all free late days are used up, penalty is 25% for each additional late day. However, projects submitted late after all late days have been used will receive no credit.

Schedule


We will cover basic and advanced deep learning models for NLP, with some practical sessions.
Pleaes pay attention to due dates (every due is by 11:59PM) and project presentations. 🍬 is an optional reading.

Date Topic Readings
W1:
Sep 6
Class overview [slides]
W1:
Sep 8
Intro to NLP [slides]
W2:
Sep 13
Text Classification [slides]
W2:
Sep 15
Neural Nets and Backpropagation [slides]
W3:
Sep 20
Practice: Building a Neural Net using Toolkits [slides]
W3:
Sep 22
Projects details: Custom and Default; Tips [slides]
HW1 out [link]
W4:
Sep 27
No class (3M Symposium)
W4:
Sep 29
Distributional semantics and word vectors [slides]
W5:
Oct 4
Contextualized Word Embeddings (1) [slides]
W5:
Oct 6
Contextualized Word Embeddings (2) [slides]
W6:
Oct 11
Project Proposal A
  • Team Semantish (Faizel Khan, Seva Arkhipenka): Choosing the right experts for BioMedical QA tasks
  • Team HSC01 (Sahil Raina, Harshavardhan Battula, Cody Wang): Self Supervision in Graph Neural Networks for Text Classification
  • Team Novella (Kyler Sood, Anush Kumar, Riley O'Neill): Author Imitation Learning
W6:
Oct 13
Project Proposal B
Project proposal report due
HW1 due
  • Team Sprite (Chen Jiang, Tianhao Zhang, Zhecheng Sheng): Controllable Long-Dialogue Generation
  • Team Polyglots (Asal Shavandi, Chahyon Ku, London Lowmanstone, Josh Spitzer-Resnick): MultiCoNER II: Multilingual Complex Named Entity Recognition
  • Team NLP Amateur (Chen Hu, Jiaqi Liu, Keyang Xuan): Cross-lingual Transfer Learning for Irony Detection
W7:
Oct 18
Language models, Fancy RNNs, Seq2Seq [slides]
W7:
Oct 20
Deep dive on Transformers [slides]
W8:
Oct 25
Pretrainig Paradigm [slides]
W8:
Oct 27
Prompting [slides]
HW2 out [link]
W9:
Nov 1
Advanced Search Algoritms [slides]
W9:
Nov 3
CNNs, Recursive Nets, Trees [slides]
W10:
Nov 8
Graph Neural Nets [slides]
W10:
Nov 10
Deep Generative Models (VAE, GAN, Diffusion) [slides]
W11:
Nov 15
Structured Prediction [slides]
W11:
Nov 17
Final Project Presentation A
Project midterm report due
HW2 due
  • Team Sprite (Chen Jiang, Tianhao Zhang, Zhecheng Sheng): Controllable Long-Dialogue Generation
  • Team Polyglots (Asal Shavandi, Chahyon Ku, London Lowmanstone, Josh Spitzer-Resnick): MultiCoNER II: Multilingual Complex Named Entity Recognition
  • Team NLP Amateur (Chen Hu, Jiaqi Liu, Keyang Xuan): Cross-lingual Transfer Learning for Irony Detection
W12:
Nov 22
Final Project Presentation B
HW3 out [link]
  • Team Semantish (Faizel Khan, Seva Arkhipenka): Choosing the right experts for BioMedical QA tasks
  • Team HSC01 (Sahil Raina, Harshavardhan Battula, Cody Wang): Self Supervision in Graph Neural Networks for Text Classification
  • Team Novella (Kyler Sood, Anush Kumar, Riley O'Neill): Author Imitation Learning
W12:
Nov 24
No class (Thanksgiving)
W13:
Nov 29
Guest lecture recording (NeurIPS) [video]
1-page summary on guest lecture
W13:
Dec 1
Guest lecture recording (NeurIPS) [video]
1-page summary on guest lecture
W14:
Dec 6
No class (EMNLP)
W14:
Dec 8
No class (EMNLP)
Final project report due
HW3 due

Project Details (50%)


Please refer to the [project description slides] on Sep 22. A course project would be one of the following types:

  • New research results judged suitable for acceptance to a top NLP or ML conference like ACL/EMNLP/NeurIPS/ICLR,
  • Evaluation and critical analysis of existing work on a new dataset,
  • An in-depth literature survey, or
  • New open-source repository or dataset with a high impact on the community

Every group member should submit their report, link to code (or a zipped code), and presentation slides on Canvas before the deadline. For both proposal and final reports, please use official ACL style templates (Overleaf or links). Note that your report and slides would be publicly shared on this page.

Your project will be evaluated in the following criteria:

  • Proposal report (Due: Oct 13) and presetnation (Due: Oct 11 / 13) (10%)
    • Maximum 3 page report and 10-min presentation including 3-min QA in class
    • Upload your PDF on Canvas
    • Upload your slides here before your presentation
    • Expected content in the report:
      • Title, members (2-3 students per group) and role assignment, motivation, literature survey, problem formulation, and broader impact
  • Midterm report (10%), Due: Nov 17
    • Maximum 8 page report
    • Upload your PDF on Canvas
    • Expected content to be presented:
      • Specific feedback you like to get from audience
      • Motivation
      • Problem definition
      • Novel contribution compared to prior work
      • Proposed methods
      • Preliminary results
      • Plan until the end of semester
  • Final presentation (10%), Due: Nov 17 / 22
    • 15-min presnetation and 5-min QA
    • Upload your slides here before your presentation
    • Expected content to be presented:
      • Motivation, problem definition, and novel contribution compared to prior work
      • Proposed methods with "motivational examples"
      • Experimental setups and final results
      • Error analysis
      • Discussion on limitations, ethical issues, and future directions
  • Final report and code (20%), Due: Dec 8
    • Maximum 8 pages with unlimited reference and appendix
    • Rubrick for evaluation

Prerequisites


CSCI 5521 Introduction to Machine Learning or any other course that covers fundamental machine learning algorithms. CSCI 8980 Introduction to NLP Research or CSCI 5541 Natural Langauge Prorcessing

Furthermore, this course assumes:

  • Good coding ability, corresponding to at least a third or fourth-year undergraduate CS major. Assignments will be in Python.
  • Background in basic probability, linear algebra, and calculus.

Notes to students


Academic Integrity

Assignments and project reports for the class must represent individual effort unless group work is explicitly allowed. Verbal collaboration on your assignments or class projects with your classmates and instructor is acceptable. But, everything you turn in must be your own work, and you must note the names of anyone you collaborated with on each problem and cite resources that you used to learn about the problem. If you have any doubts about whether a particular action may be construed as cheating, ask the instructor for clarification before you do it. Cheating in this course will result in a grade of F for course and the University policies will be followed.


Students with Disabilities

If you have a disability for which you are or may be requesting an accommodation, you are encouraged to contact both your instructor and Disability Resources Center (DRC).


COVID-19

All students are expected to abide by campus policies regarding COVID-19 including masking and vaccination requirements. This is an in-person class with daily in-person activities, but we may consider a hybrid or online option. If you're feeling sick, stay at home and catch up with the course materials instead of coming to class!



Book

Textbook is not required but the following books are primarily referred:
  • Jurafsky and Martin, Speech and Language Processing, 3rd edition [online]
  • Jacob Eisenstein. Natural Language Processing

Resources

The course materials are inspired by the slides of Chris Manning at Stanford, David Bamman at UC Berkeley, and Graham Neubig at CMU.