Wednesday, May 25, 2011

Web Scraper in Ruby

Just written a simple Web Scraper in Ruby for sharing.

It uses MongoDB to store data, use Mechanize to crawl and parse, use Parallel to do multi-threading and then export the results into CSV file.

You can read the source code here: https://github.com/3dd13/web_scrapers_ruby

張貼者： Unknown 於 2:40 PM

2 comments:

  1. NewprosoftOctober 1, 2011 at 3:30 PM

    Hi all,

    Web scraping іѕ thе administer οf automating thіѕ administer, ѕο thаt аѕ a replacement fοr οf manually copying thе data frοm websites, thе Web scraper software wіƖƖ perform thе same task within a fraction οf thе time. Thanks...

    Web Extractor

    ReplyDelete
  2. Carly FiorinaMarch 29, 2012 at 9:15 AM

    Hello Friends,

    Web Scraping is the process of taking html or data from the web and organizing that data into an organized format. Data in the web is embedded in a plethora of languages with varying syntax. Each method to retrieve that data must be custom built to extract that data from a particular website. Web scraping programs will basically simulate a person viewing a web site with a browser.
    Data Extract

    ReplyDelete

Subscribe to: Post Comments (Atom)

Ru05 Coders