Machine Learning (CS 446) Fall 2019

Course Staff

Professor:

Sanmi Koyejo

Office: 3314 SC

Office hours: Monday 2pm-3pm; or by appointment.

E-mail: sanmi

Teaching Assistants (alphabetical order):

Course Assistants (alphabetical order):

Course Description

The goal of Machine Learning is to build computer systems that can adapt and learn from their experience. This course will study the theory and application of learning methods that have proved valuable and successful in practical applications. We review the theory of machine learning in order to get a good understanding of the basic issues in this area, and present the main paradigms and techniques needed to obtain successful performance in application areas such as natural language and text understanding, speech recognition, computer vision, data mining, adaptive computer systems and others. The main body of the course will review several supervised and (semi/un)supervised learning approaches. These include methods for learning linear representations, Bayesian / Probabilistic methods, decision-tree methods, kernel based methods and neural networks, as well as clustering and dimensionality reduction techniques. We will also discuss how to model machine learning problems and discuss some open problems.

Topics to be covered include:

• Linear/Logistic Regression
• Variable Selection / Sparsity
• Optimization - Gradient Descent
• Support Vector Machines
• Convolutional/Recurrent Neural Networks
• Clustering
• Graphical Models
• Expectation Maximization
• Variational Inference
• Generative Adversarial Networks
• Multilabel Classification
• Structured Prediction

Prerequisites

Students are expected to have taken a class in linear algebra, probability and statistics and a basic class in theory of computation and algorithms. Students are expected to be familiar with the python programming language.

Course Materials

Required Text

None

Optional books with relevant material:

• Kevin Murphy, Machine Learning: A Probabilistic Perspective, MIT Press
• Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press
• Trevor Hastie Robert Tibshirani & Jerome Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer Science & Business Media
• C.M. Bishop, Pattern Recognition and Machine Learning, Springer
• Tom Mitchell, Machine Learning, McGraw Hill
• Ian Goodfellow, Yoshua Bengio & Aaron Courville, Deep Learning, The MIT Press

Lecture notes, course handouts, pointers to relevant papers, and other materials will be available as HTML and PDF documents on Relate and Piazza

Exams

Exam will be in class exam with 75 minutes. Please find information on our upcoming exams in the corresponding section of the class calendar.

Homework

We do not accept late homework. The lowest homework grade will be dropped.

There will be five homework assignments. The homeworks will consist of a combination of machine problems and written exercises to be submitted on the course relate website. All homeworks are due at Midnight CT on the specified day. Submitted solutions to the machine problems must be written in Python 3, using only standard libraries (numpy / scipy). We will sometimes block libraries as required. Solutions to the written problems must be submitted as typeset PDF’s (we suggest LaTeX, but anything typeset is fine). Each question must begin on a new page.

• Homework 0
• Homework 1
• Homework 2
• Homework 3
• Homework 4
• Homework 5

Project

All students will participate in a predictive modelling competition. Details will be provided in subsequent weeks. You are free to use any methods you like, however, you will be required to submit your source code for the final submission. The course project will be introduced in class, including further details on the use of computing resources.

The Project Grade will be based on predictive performance as compared to four baselines (i.e. 25%, 50%, 75%, 100% of the grade). Beyond this, additional performance gives you bragging rights! Your grade will be based on the final submission due Dec 12, Midnight

Computing:

• Microsoft Azure has provided cloud computing resources for the project. Details will be provided in class.
• Google Colab is a great free computing resource that uses a Jupyter notebook style environment.

View project details »

Additional Participation Credit (up to 5%)

This may be earned by participating in lecture & online discussions, scribing lectures and by reporting mistakes in the lecture notes. Note that it is possible to earn 105% grade.

Scribing Lecture: All students are encouraged (not required!) to scribe at most one lecture for participation credit. You will be awarded the credit only if your notes are of a reasonable standard (as determined by course staff). Otherwise, you will be given an opportunity to improve and correct your notes. Note that this credit is a subset of the extra credit, which can be earned in other ways.

We require one set of notes from the scribe team. The scribe team should take detailed notes during the lecture, then convert their notes into LaTeX format of 5-8 pages long. Scribe notes must be submitted within two weeks after the lecture (one week is ideal to get some feedback). Please submit your scribe notes on Relate Please sign up here (Google Sheets). Do not ask for permission. Login to Google Docs with your Illinois Credentials.

Scribe submissions

Grading Policies

View policies »

Online Discussion Platform

We will use Piazza for our online discussion platform. You are encouraged to use Piazza to ask the course staff and your classmates and course-related questions (we will refer all email queries to piazza). Please self-register at https://piazza.com/illinois/fall2019/cs446

Academic integrity

Feel free to discuss the assignment with each other in general terms, and to search the Web for general guidance (not for complete solutions). All solutions should be written up individually. If you make substantial use of some information from outside sources, make sure you acknowledge the sources in your solution. In particular, make sure you acknowledge all other students you worked with on the homework/projects. Failure to do this will result in a zero grade. We will follow the departmental honor code policy here: https://cs.illinois.edu/academics/honor-code

What to do in an emergency

See this handout

Python Help

• The Scipy Lectures
• Learn Python the hard way
• Python tutorial
• Facts and myths about Python names and values
• CSE workshop training material
• From Python to Numpy (An open-access book on numpy vectorization techniques, Nicolas P. Rougier, 2017)

Numpy Help

• Introduction to Python for Science
• Numpy/Scipy documentation
• More in this reddit thread
• An introduction to Numpy and SciPy
• 100 Numpy exercises
• The Numpy MedKit by Stéfan van der Walt

Linear Algebra

• Matrix Cookbook
• Immersive Linear Algebra
• Essence of Linear Algebra (YouTube, by 3Blue1Brown)
• Fast.ai Linear Algebra Course (Fast.ai, notes and video)
• Linear Algebra (YouTube, by MathTheBeautiful)

Statistics

• Statistics for Hackers by Jake VanderPlas