WebScrapeDirect

Fastest, cheapest, instant web scrape service.

Contact Data extraction Content aggregation Data feeds Shopping feeds Compare Feeds Comparison Shopping Platforms Price Compare Solutions Online Price Comparison Engine Hands Free Feeds Hands Free Shopping Feeds User run or mananged service Monitoring competitors Monitoring websites Web scraping Output formats (txt,csv,xls etc) About WebScrapeDirect

We offer online price compare solutions. A code framework that will aggregate the prices from many websites and deliver it to users in an elegant manner that is cost effective for you.

Acquire real time prices from different websites and display on a single web page.

There are four distinct technical challenges to an online price comparison website.

1: Gather the user information. The use needs to tell you want they want, this can be a simple description ( addida supernova ) or a complicated workflow ( eg car insurance ).

2: Product name recognition. We must transpose the user requirement into something meaningful to each of the sites to be compared from. For example if comparing for car insurance a certain Volkwagen might be known to one website as "VW golf 1.6 GTi" and to another website as "Volkwagen Golf GT 1600i", these different names need to be recognised as the same car.

3: Price data acquisition. Retrieving the actual price can be done either from data feed or by searching the target websites with robots. If the merchant offers its catalogue as a data feed then it can be uploaded on a nightly basis into a database, if they offer a permanently updating rss feed of changes to their catalogue then this must be continuosly monitored. The more interesting case is where the merchant cannot offer data feeds, in which case we need to run robots to scrape the target site for each user request to extract the price information. There is also the possibility of using a scraper to extract an entire catalogue for entry into our own database, this would generally be run each night. Sometimes the data is too big for any solution other than running real time robots upon a user request, car insurance is a good example, where the various rating factors make for a data set of many trillions of prices (think all different postcodes multiplied by all different cars multiplied by all different ages etc). When robots are required for each user request they must be run simultaneously so a "multi-bot" environment is required.

4: Elegant user display. The final part of a price comparison engine is the user display, where we have the problem of waiting time for the slowest robots to finish. We want the user to see something immediately, but we don't have all the prices immediately and must wait for the last robot to complete before we can rank the results in price order. To render in a progressive and elegant way so that it is clear to the user that they are watching an ongoing price comapre process we need an AJAX javascript front end to handle the price data retrieved by our robots.

At WebScrapeDirect we have solutions to all these problems, so Contact us now .