Get all links of a website

Question

Is there a way/tool to get all links of a website ? Just the links , not looking to create a local copy/download a website . Example - Links of all questions posted on Superuser . Platform Windows 7 , Ubuntu 14.04

I tried it . It only gives links of a single page . Not the whole website . — Renuka, Oct 26 '14 at 12:35
I realize you said you want 'links only' and not the whole site (ie: the one I marked as a possible dupe), but the #1 answer (HTTrack) for the duplicate has the ability to do what you ask, and I would guess at least one of the other suggestions would do it as well. If you have tried them, please let us know why they didn't work for you in your case. — Ƭᴇcʜιᴇ007, Oct 26 '14 at 16:40
Funnily enough I'm building an email-extractor, but it can already grab all links from a website. If you give me a few hours I'll make some amendments (so it displays the links) and post you the link to my repo on github and you can grab it from there — benscabbia, Oct 26 '14 at 16:57
@Ƭᴇcʜιᴇ007 . I installed HTTrack but couldn't find the "link only" option . Can you tell me where is it ? — Renuka, Oct 26 '14 at 18:29
@hagubear .Tried it but it didn't work fully . It didnt extract all the links . Also my PC/IP is banned from the sites on which I tried the software :( . — Renuka, Oct 26 '14 at 18:30
@Ƭᴇcʜιᴇ007 . It started downloading all the files . And hts-cache folder is empty for now . The guy doesn't say about any setting to make to not download/just crawl the site . — Renuka, Oct 26 '14 at 18:57

benscabbia · Answer 1 · 2014-10-27T13:24:03.907

1

Sorry for keeping you waiting. I have uploaded my program here.

The program is still in very-very early phase, so most features do not work, but it does, however, grab all links to other pages on the website.

It needs java to run and you should be able to double click the file and a UI should load up. Type in the SearchW box (in the GUI) the website address i.e. http://google.com, http://bbc.co.uk

Then you can copy and paste all the links as they are printed (I still need to implement an export feature but you'll be able to copy the links for the moment)

Let me know if you have any issues! And if you like it, I will, (once it's in a decent state) post a link to my repo where you'll be able to download the newer versions.

edited Oct 27 '14 at 13:24

answered Oct 27 '14 at 12:09

benscabbia

420

Hello . I tried it first on filehippo.com . it gave me this output which looks ok . The site isnt that big. http://pastebin.com/CfSe4RgD . Then I tried it on a bigger site . 9gag.com . It gave a output of only 61 lines http://pastebin.com/VE30u8DE. Filehippo was 208 lines . its impossible . 9gag has millions of posts . – Renuka Oct 27 '14 at 13:02
BTW Thank You . Keep updating if you can. I will keep testing it for you . :) – Renuka Oct 27 '14 at 13:12
1

@Renuka sorry I should have mentioned. The Parser only scan one level i.e. Grabs all links from the home page and then it checks each page for an email (I know it's not of interest to you). But if you want it to grab literally every single link on a website I will have to change a bit of code. The only problem is that if there is a link (say on homepage) which point out to another website, it will also start grabbing links from other websites (it doesn't know if the link is from the current website or a different website. But I'll see what I can do – benscabbia Oct 27 '14 at 13:13
1

@Renuka you're most welcome :). I won't be able to work on it for a few days but now I have someone who is waiting for updated versions I will do my best to keep developing it asap! – benscabbia Oct 27 '14 at 13:15
No problem :) .BTW I just double clicked the jar file and it gave me a UI . Didn't go to cmd to type that command you asked to . – Renuka Oct 27 '14 at 13:20
1

@Renuka I was hoping it would run directly (they weren't actually instructions but I have re-worded now). Can be a real pain to get it to run through console (would probably require system variables configuration etc) – benscabbia Oct 27 '14 at 13:25

Get all links of a website

1 Answers1