Looking for web scraper that works on YouTube channel video catalogs

Question

I'm trying to find a web scraper that will help me get detailed data from several YouTube channels I'm interested in researching.

I've tried the trial version of Easy Web Extractor, but the processed data count ends at 511. I've tried contacting their support, but they assure me that it should not be a trial limitation and have encouraged me to make changes that I've already tried.

I've also found a Web Scraper addon on the Chrome store, but I wanted to ask the fine community of Open Data stack exchange what your experience has been with data scraping software and what you would recommend.

I've generally had good luck with kimonolabs for creating APIs based on a scraping approach. — Erik Schwartz, Dec 04 '14 at 16:20
What kind of data do you need? The Linux tool youtube-dl can retrieve some data from channels, but I am not sure what data you want. — Nicolas Raoul, Dec 05 '14 at 06:03
Thanks for the replies y'all. I'm looking to crawl just 1 link deep and scrape details from a list of items in a channel's video catalog. — Beer_en_thu_si_asT, Dec 06 '14 at 15:42
It is no longer available. am I right.... How to download TubeKit? — Giacomo Crotti, Mar 14 '19 at 09:30
@GiacomoCrotti, I'm actually not sure. We no longer are going this route for our YouTube Data Gathering. — Beer_en_thu_si_asT, Apr 01 '19 at 16:42

Patrick Hoefler · Accepted Answer · 2014-12-18T14:26:15.063

3

TubeKit might be of interest to you:

TubeKit is a toolkit for creating YouTube crawlers. It allows one to build one's own crawler that can crawl YouTube based on a set of seed queries and collect up to 16 different attributes.

The tool is open source (licensed under CC BY-NC-SA^*) and has been developed for research purposes.

_{^* Creative Commons advise against using one of their licenses for software — but that's really beside the point in the context of this answer ;)}

edited Dec 18 '14 at 14:26

answered Dec 18 '14 at 14:16

Patrick Hoefler

5,790
4
31
47

It looks like TubeKit's core functionality was lost when Youtube implemented a new API last year. – Jul 30 '15 at 23:38
youtube-dl will work if this site doesn't, see here https://opendata.stackexchange.com/a/5733/1511 – philshem Mar 21 '20 at 16:40

score 3 · Answer 2 · edited Apr 13 '17 at 12:23

3

On Linux use youtube-dl command to download videos straight off of YouTube.

To install youtube-dl run sudo pip install youtube-dl

To download an entire channel:

youtube-dl -citw ytuser:<USER>

It is open source, and well-maintained, modifications in YouTube are usually taken care of very quickly.

Answer stolen from josten at https://askubuntu.com/questions/376268/how-to-download-entire-youtube-channel-using-commands-or-application

edited Apr 13 '17 at 12:23

Community

1

answered Jul 31 '15 at 08:40

Nicolas Raoul

8,426
5
28
61

similar answer here: https://askubuntu.com/a/856913/104223 – philshem Mar 21 '20 at 16:18

Looking for web scraper that works on YouTube channel video catalogs

2 Answers2