6

Pretty soon I will be finishing up Understanding Machine Learning by Shai Ben-David and Shai Shalev-Shwartz. I absolutely love the subject and want to learn more, the only issue is I'm having trouble finding a book that could come after this. Ultimately, my goal is to read papers in JMLR's COLT.

  1. Is there a book similar to Understanding Machine Learning that would progress my knowledge further and would go well after reading UML?
  2. Is there any other materials (not a book) that could allow me to learn more or prepare me for reading a journal like the one mentioned above?

(also taking courses in this is not really an option so this will be for self-study)

This question was also asked here on AI SE from suggestion of comments.

PMaynard
  • 163
  • 4
  • 2
    Although TCS SE is one of the best places to ask these questions, Artificial Intelligence SE also has some users interested in this topic and that may be able to answer this question. Of course, learning theory is a central in AI. –  Apr 17 '20 at 03:48
  • See e.g., this: https://cstheory.stackexchange.com/a/46574/13319 (the Kearns-Vazirani book is a bit old, but still very relevant). – Clement C. Apr 17 '20 at 16:51
  • These recent lecture notes on Statistical Learning Theory (organized as a single, organized file) by Bruce Hajek and Maxim Raginsky may also be helpful. – Clement C. Apr 17 '20 at 16:53
  • This question was also asked here: https://ai.stackexchange.com/q/20355/2444. –  Apr 17 '20 at 17:53
  • PMaynard, I didn't realize that you posted this question elsewhere. Our site policy prohibits simultaneous crossposting. – Lev Reyzin Apr 18 '20 at 00:21
  • @LevReyzin Prohibits? What kind of policy is that? It's sometimes discouraged, but prohibition? Do you know what that means? Of course, the OP is not prohibited to ask elsewhere and he should ask elsewhere if the question can be answered by different people with different perspectives. –  Apr 18 '20 at 02:44
  • @PMaynard Don't worry. There's no prohibition. No other Stack Exchange website really prohibits this, AFAIK. I am a moderator at Artificial Intelligence SE and there's no such thing as "prohibition" with respect to this topic. Sometimes it's discouraged, but I completely disagree with this discouragement. A question, in fact, can be on-topic on different sites and there's no decent reason not to be able to ask the question on multiple sites, especially if one is looking for different perspectives and answers. –  Apr 18 '20 at 02:46
  • @LevReyzin My understanding is that the crosspost on AI.SE followed the suggestion in the first comment (it wasn't crossposted prior to it). (Also, nbro, not sure what the explicit rules are, but on Math.SE, Mathoverflow (and here) it's at the very least frowned upon.) – Clement C. Apr 18 '20 at 05:05
  • @ClementC. AFAIK, there's no rule that prohibits cross-posting. For example, this answer says "as a general rule", according to whom? The author of the answer. That's it. There's only an unreasonable "discouragement". Also, although the question was asked first here, there I provided an answer before any other answer provided here, which suggests that cross-posting shouldn't be discouraged because it can be beneficial. –  Apr 18 '20 at 06:19
  • 1
    @nbro, as a moderator, you should know that different stackexchange sites have their own site-specific rules. You should probably also not go to other stackexchanges encouraging people to ignore their policies because you "disagree" with them.

    Our policy is spelled out here: https://cstheory.meta.stackexchange.com/questions/225/official-faq-for-theoretical-computer-science/231#231.

    I will not close this post because we already have upvoted answers and because PMaynard is a new user here.

     – Lev Reyzin Apr 19 '20 at 01:03
  • @LevReyzin You were talking about "prohibition" above. The word "prohibition" is very strong. I know what I said above. Read again all my comments above. I am exactly following that answer, actually, although I don't completely agree with it. Note that I suggested this user about AI SE, but it was only hours later that he posted on AI SE because he hadn't received any answer here. As a moderator, you should note that there isn't really a "prohibition" with respect to this topic. But let's not make this a fight now. The issue is over. –  Apr 19 '20 at 01:07
  • We should probably make our policy clearer. Hours are not what we had in mind as the right amount of time to wait. Also, we do routinely close cross-posted questions. – Lev Reyzin Apr 20 '20 at 15:33

3 Answers3

6

Another good introductory book is "Foundations of Machine Learning" by Mohri et al.: https://www.amazon.com/Foundations-Machine-Learning-Mehryar-Mohri/dp/0262039400/. It has a large overlap with the Shai and Shai book, but also quite a bit of content that they don't cover.

There are also good books and surveys on more advanced or specialized topics:

There are more, but I hope this is a good start.

Lev Reyzin
  • 11,968
  • 13
  • 63
  • 103
  • 1
    Thank you this seems like a great list! I'm thinking it would be best to additionally work on computational complexity, high dimensional probability, and convex optimization. This also seems to confirm my suspicion that there are no more 'general' books on the subject after this. Please let me know if you have anymore tips on learning this subject. Additionally it would be nice to know about how far I am from being able to read and understand journals like COLT. – PMaynard Apr 17 '20 at 18:19
  • For complexity, you should look up Arora/Barak. For various results in high dimensional probability, the "data science" book by Blum, Hopcroft, and Kannan is good: https://www.cs.cornell.edu/jeh/book.pdf. For convex optimization, books and surveys by Vempala are good.

    As for reading COLT/ALT/JMLR papers, some are easier than others, and you can learn some topics before others. There's no reason not to try reading the papers as soon as you want and to consult books and references as needed.

     – Lev Reyzin Apr 17 '20 at 18:23
  • Good idea to maybe start with some more approachable papers earlier on. Thanks again for the book recommendations (funny you should mention it Arora/Barak is on my nightstand right now). – PMaynard Apr 17 '20 at 18:29
  • 2
    For high-dimensional probability, the... "High-Dimensional Probability" book by Vershynin is really good. https://www.math.uci.edu/~rvershyn/papers/HDP-book/HDP-book.html @PMaynard – Clement C. Apr 17 '20 at 18:55
  • Foundations is excellent. I was about to comment with this and then immediately saw your answer. Its treatment is quite mathematically mature, in my opinion. – Max von Hippel Apr 21 '20 at 03:25
4

I have a list of references (incomplete) that may interest you: https://kiranvodrahalli.github.io/links/#resources-notes-textbooks-monographs-classes-etc

(second all the existing suggestions).

kiranv
  • 41
  • 1
2

I would recommend the book:

Borja Rodríguez Gàlvez
  • 21
  • 1