Comments on: How To Write A Simple Web Crawler In Ruby http://www.skorks.com/2009/07/how-to-write-a-web-crawler-in-ruby/ For the betterment of the software craft... Mon, 21 Nov 2011 13:57:06 +0000 hourly 1 http://wordpress.org/?v=3.1.2 By: Alan Skorkinhttp://www.skorks.com/2009/07/how-to-write-a-web-crawler-in-ruby/comment-page-1/#comment-3336 Alan Skorkin Wed, 18 Nov 2009 10:19:52 +0000 http://www.skorks.com/?p=894#comment-3336 Timely question, I've actually been learning quite a bit about search and crawling and indexing etc lately, I am planning to do a whole lot of posts about search, indexing, etc. I also want to do more with this crawler as there are a few things that I am now aware of that I wasn't before which would be worth discussing. So stay tuned, once my life settles a little and I get back into the groove, there will be a whole lot of interesting (hopefully) stuff coming up. Timely question, I’ve actually been learning quite a bit about search and crawling and indexing etc lately, I am planning to do a whole lot of posts about search, indexing, etc. I also want to do more with this crawler as there are a few things that I am now aware of that I wasn’t before which would be worth discussing. So stay tuned, once my life settles a little and I get back into the groove, there will be a whole lot of interesting (hopefully) stuff coming up.

]]> By: Georgios Mhttp://www.skorks.com/2009/07/how-to-write-a-web-crawler-in-ruby/comment-page-1/#comment-3329 Georgios M Thu, 12 Nov 2009 14:49:43 +0000 http://www.skorks.com/?p=894#comment-3329 Well done on this. :) Will you be making additions to this soon? (Indexing and Searching the index?) Thanks Well done on this. :) Will you be making additions to this soon? (Indexing and Searching the index?)

Thanks

]]> By: Alan Skorkinhttp://www.skorks.com/2009/07/how-to-write-a-web-crawler-in-ruby/comment-page-1/#comment-1863 Alan Skorkin Tue, 04 Aug 2009 08:14:31 +0000 http://www.skorks.com/?p=894#comment-1863 Yeah I tried that, but the amount of work needed to get that to work ends up being similar. Depending on what the relative url looks like as well as the absolute url of the host e.g. if relative url starts with a / then it is relative to the base of the host if it doesn't start with a slash then it is relative to it's current location then you still need to calculate how many ../ you need to do to prepend to your relative url in order for everything to be correct. These are just some of the considerations You're right though you can certainly do this using the uri library as well. Yeah I tried that, but the amount of work needed to get that to work ends up being similar.

Depending on what the relative url looks like as well as the absolute url of the host e.g.
if relative url starts with a / then it is relative to the base of the host
if it doesn’t start with a slash then it is relative to it’s current location
then you still need to calculate how many ../ you need to do to prepend to your relative url in order for everything to be correct. These are just some of the considerations

You’re right though you can certainly do this using the uri library as well.

]]> By: Eric Hodelhttp://www.skorks.com/2009/07/how-to-write-a-web-crawler-in-ruby/comment-page-1/#comment-1857 Eric Hodel Mon, 03 Aug 2009 23:13:11 +0000 http://www.skorks.com/?p=894#comment-1857 Why didn't you use the built-in URI library? require 'uri' uri = URI.parse 'http://example.com/foo/bar/baz' uri + '../../other/place' Why didn’t you use the built-in URI library?

require ‘uri’
uri = URI.parse ‘http://example.com/foo/bar/baz‘
uri + ‘../../other/place’

]]> By: pramodhttp://www.skorks.com/2009/07/how-to-write-a-web-crawler-in-ruby/comment-page-1/#comment-1852 pramod Mon, 03 Aug 2009 07:57:38 +0000 http://www.skorks.com/?p=894#comment-1852 Hi This is really good one. can u just give any references of book for learning the scraping or crawling on Ruby. i am new to this and very excited abt this . Hi
This is really good one. can u just give any references of book for learning the scraping or crawling on Ruby. i am new to this and very excited abt this .

]]> By: Vasudev Ramhttp://www.skorks.com/2009/07/how-to-write-a-web-crawler-in-ruby/comment-page-1/#comment-1836 Vasudev Ram Sat, 01 Aug 2009 15:51:14 +0000 http://www.skorks.com/?p=894#comment-1836 >Regarding using the ‘if’ after, i was actually just trying it out to see how it feels :). Good reason :) >Regarding using the ‘if’ after, i was actually just trying it out to see how it feels :).

Good reason :)

]]> By: Alan Skorkinhttp://www.skorks.com/2009/07/how-to-write-a-web-crawler-in-ruby/comment-page-1/#comment-1819 Alan Skorkin Wed, 29 Jul 2009 13:51:26 +0000 http://www.skorks.com/?p=894#comment-1819 Thanks man, I really appreciate all the feedback and testing and thanks for the ideas as well, they are certainly something that can be included in future versions. I am trying to keep it pretty simple for the moment. Once I've built a nice simple vertical slice, there are all sorts of different ways to expand and make it more configurable. Thanks man, I really appreciate all the feedback and testing and thanks for the ideas as well, they are certainly something that can be included in future versions. I am trying to keep it pretty simple for the moment. Once I’ve built a nice simple vertical slice, there are all sorts of different ways to expand and make it more configurable.

]]> By: Pieterhttp://www.skorks.com/2009/07/how-to-write-a-web-crawler-in-ruby/comment-page-1/#comment-1818 Pieter Wed, 29 Jul 2009 13:46:51 +0000 http://www.skorks.com/?p=894#comment-1818 It works great. I look forward to see the progress (I have grabbed your RSS feed for that!). On the point above, the 'risk' of course is that as you do not know the number of links on that site or page, that you may never have a full index of that website, and will have to repeat with different page counts by trial and error until you have found all the links on a site (even if you crawl only at depth 2) The alternative could of course be to make the page count per url.txt link, so that it will always traverse the full url.txt link, but only '-p ####' per link. Another way could be to keep the page count down by adding a blacklist or whitelist into the mix so that links to adservers, google etc could be excluded and not crawled. Look forward to your progress in this, and will grab and test when you update again, best regards It works great. I look forward to see the progress (I have grabbed your RSS feed for that!). On the point above, the ‘risk’ of course is that as you do not know the number of links on that site or page, that you may never have a full index of that website, and will have to repeat with different page counts by trial and error until you have found all the links on a site (even if you crawl only at depth 2)

The alternative could of course be to make the page count per url.txt link, so that it will always traverse the full url.txt link, but only ‘-p ####’ per link.

Another way could be to keep the page count down by adding a blacklist or whitelist into the mix so that links to adservers, google etc could be excluded and not crawled.

Look forward to your progress in this, and will grab and test when you update again, best regards

]]> By: Alan Skorkinhttp://www.skorks.com/2009/07/how-to-write-a-web-crawler-in-ruby/comment-page-1/#comment-1817 Alan Skorkin Wed, 29 Jul 2009 13:34:37 +0000 http://www.skorks.com/?p=894#comment-1817 Ahh, ok well that explains it. The reason that the urls in urls.txt are included is that if you hook up an indexer to the crawler then it will index the pages in urls.txt as well as crawling the links and indexing those pages. So it actually makes sense to count them as well. It is possible to make this configurable in case people didn't want to index the urls that they supply. Ahh, ok well that explains it. The reason that the urls in urls.txt are included is that if you hook up an indexer to the crawler then it will index the pages in urls.txt as well as crawling the links and indexing those pages. So it actually makes sense to count them as well.

It is possible to make this configurable in case people didn’t want to index the urls that they supply.

]]> By: Pieterhttp://www.skorks.com/2009/07/how-to-write-a-web-crawler-in-ruby/comment-page-1/#comment-1816 Pieter Wed, 29 Jul 2009 13:28:53 +0000 http://www.skorks.com/?p=894#comment-1816 No, I used some real links I wanted to crawl this time. But, I tested it again and you are right it does work with or without. The difference however was that in case 1 (which did not work) the number of links in url.txt was greater than the pages=100 setting (i.e. 400 links) and all it did was traverse through 100 links in url.txt and stopped, without doing any further crawling beyond that level (obviously as it has reached the pages limit). Would it be better if the links in the url.txt file are excluded from the page count? No, I used some real links I wanted to crawl this time. But, I tested it again and you are right it does work with or without. The difference however was that in case 1 (which did not work) the number of links in url.txt was greater than the pages=100 setting (i.e. 400 links) and all it did was traverse through 100 links in url.txt and stopped, without doing any further crawling beyond that level (obviously as it has reached the pages limit). Would it be better if the links in the url.txt file are excluded from the page count?

]]>