Sa, 12-5 Mafia party at 8pm at Manfred's place 425 0461 Bring friends - we need bodies !!! Tu, 12-8,11-4pm Poster presentations in E2-489 - up to 15 pages Th, 12-10,6pm Projects due at Manfred's office and electronically No incompletes - sorry
The default is that you need to implement your own algs. If you want to use a package then you need to make a case to the instructor for approval. SVM packages are preapproved!
Class: MW 5-6:45, E2-194
Office hours: M 1:00-2:00
(in E2 357) W 12:30-1:30
Prerequisite: some probability theory and linear algebra
TA: Maya Hristakeva
Office hours: Mo 7-8pm, F 10-11am - BE 358
Summary of lectures
1 Notes 1 Better scan & some more corrections
Intro to Machine Learning using curve fitting as an example
Overfitting, complexity control, regularization
Experimental setups w. training, validation and test sets
Relevant chapter from Bishop's book
On-line versus batch
Definition of regret
Halfing algorithm and its bound
Weighted Majority algorithm
Regret bound for WM via potential function
Hedge algorithm, Follow the Leader algorithm
Hedge algorithm paper
2 Notes 2
Analysis of Hedge algorithm via potential function
Maple file on how to tune Hedge .ms .pdf
Talk re. various Share Updates incl. one that induce longterm memory
Long term memory paper
Original "Tracking the best expert" paper
Talk re measuring on-lineness
Original Disk Spindown paper
Talk w. more details on Disk Spindown Problem
Disk Spindown Data
TCP Problem Slightly expanded
Bruno's Original TCP Report New!
tcpdataset1.txt New!
Homework 1 Due F 10-16-09 5pm under Manfred's door
3 Notes 3
Information theoretic motivation of relative entropy
Relevant chapters from Cover/Thomas book
Visualizations of relative entropies Maple file
- motivation of updates and analyses of Hedge alg
Online updates for linear regression
- and learning linear threshold functions
Motivation for the GD, EG and EGU
Implicit versus explicit
4 Notes 4
Logistic regression
Stochastic Gradient Descent
Newton updates
Linearly Least Squares
Newton type algs for logistic regression
5 Recall derivation of GD, EG, and EGU in the case of linear regression (Lecture 3)
Notes 5
EG versus GD
How to prove the regret bounds
The kernel trick and its limits
Leaving the span talk
Leaving the span paper
6 Notes 6
Optimization
- Lagrangians
- Duality
How applied to Support Vector Machines
More on kernels
7 Finish Support Vector Machines and kernels
Convexity and Jensen's inequality Notes 7
Bregman divergences, Generalized Pythagorean Thm, Matching Loss
Matching loss design Thanks Maya! Did another pass!
Maching loss of a piecewise linear tranfer function pdf in maple
Homework 2 Due We 10-28-09, beg. of class
8 Finish matching loss
Part I: Entropty regularized LPBoost
Part II: Optimization and Experiments
9 Notes 9
More details about AdaBoost original paper
Boosting algs lead to good classifiers and example filters
Paul Viola's work on face recognition
Damian Ead's presentation
Interactive method for building pharmaceutical compounds chapter from Jun Liao's thesis
Cross validation
10 ROC curve for evaluating a ranking
ROC curves of perfect and random classifier
Logistic Regression - norm and shrink stretch regularizations
Spam dataset created by D. Skulley
Homework 3 Due Mo Nov 9 in class
11 Notes 11
Loss bounds for expert model when experts predict
Conditional probabilities and Bayes Rule
How does Bayes rule fit into Expert Framework
Hw2 solutions
12 Notes 12 Rescanned
ML and MAP estimators
Naive Bayes and spam application
filtering spam w. SVMs
filtering spam w. Naive Bayes
Exponential families w. connection to Bregman divergences
13 Homework 4 2-page project description - due Mo Nov 16 in class
Talk re building caching strategies based on the shifting expert framework
Conference paper Master's thesis w. more details
Combine two priority lists for caching w. ARC Alg
Notes 13 Alternate method based on exponenential weights and Weighted Median algorithm (for the 2-list case) Rescanned
Master's Project by Corrie Scalisie's: Weighted Median with Share updates beats ARC
14 Veterans day
15 Discussion of homework 3 solutions and review
A good sample project by Ning Bao
A wild project - A Kinship Face Matching Game slides report
16 Dynamica Programming
EM Notes 16
Homework 5 2-page project progress report
17 Notes 17 Rescanned
Multiclass Logisting Regression paper on on-line multiclass log. regr. algs
Multiclass Boosting AdaBoost paper paper on multiclass boosting Voted Perceptron
Bug Machine
17 Notes 18
The optimal Hedge algorithm
The optimal Hedge alg talk paper
19 Notes 19
Trace of a matrix
Matrix derivative formulas
Online PCA talk
Online PCA paper
20 Why do relative entropy appear everywhere in nature?
The blessing and curse of the multiplicative updates
- Three mechanisms for avoiding the curse
- Motivating multiplicative updates as relative
entropy minimization problem
About PCR
---------------------------------------------------------