cs132.webSpider
Class Spider

java.lang.Object
  cs132.webSpider.Spider

public class Spider
extends java.lang.Object

This class provides implementation of a Spider or web crawler. The state of the web crawler consists of:

• A flag indicating whether the crawler is to perform a DFS or BFS search.
• A maximum number of web pages the crawler will actually visit.
• A count of the number of web pages crawled so far.
• A mapping from URL's to WebPages for all URL's and WebPages seen by the crawler.
• A queue/stack of web pages that remain to be visited.

Once the Spider is created and provided with one or more initial web pages to crawl, a crawl can be initiated by invoking the performCrawl() method.

Constructor Summary

Spider(boolean isDFS, int limit, java.lang.String root)
Create a new Web Spider

Method Summary

void	addPageToCrawl(WebPage w) Add a web page to the set of pages to be crawled
WebPage	addURLToCrawl(java.net.URL u) Add the WebPage corresponding to a URL to the set of pages to crawl.
void	crawl(WebPage w) Crawl one WebPage.
WebPage	getNextPageToCrawl() Determines the web page that should next be crawled and removes it from the queue, skipping over any web pages that have already been visited.
int	getNumberOfPagesCrawled() Return the count of the number of web pages crawled
java.util.Collection<java.net.URL>	getURLs() Returns a collection of the URLs of all known webpages (e.g., the URL's corresponding to the WebPages returned by the getWebPages method).
java.util.Collection<WebPage>	getWebPages() Returns a collection of all known web pages, including both visited and unvisited webpages
WebPage	lookupWebPage(java.net.URL u) Given a URL, returns the corresponding web page corresponding to the normalized URL
static void	main(java.lang.String[] args) Invoke the Spider, crawl 5 web pages in DFS order, and print the result
void	performCrawl() Crawls all WebPages, adding more Webpages to the crawl queue as links to them are found.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Spider

public Spider(boolean isDFS,
              int limit,
              java.lang.String root)

Create a new Web Spider

Parameters:

isDFS - -- should the spider perform a DFS crawl?

limit - -- maximum number of web pages to crawl

root - -- root that limits extent of crawl. Only URLs that start with this root will be crawled.

Method Detail

main

public static void main(java.lang.String[] args)
                 throws java.lang.Exception

Invoke the Spider, crawl 5 web pages in DFS order, and print the result

Parameters:

args - - first element should be web page to start crawl from, second element is the root of the web crawl (only URL's that start with that root will be crawled).

Throws:

java.lang.Exception

getNumberOfPagesCrawled

public int getNumberOfPagesCrawled()

Return the count of the number of web pages crawled

Returns:

number of web pages crawled

addPageToCrawl

public void addPageToCrawl(WebPage w)

Add a web page to the set of pages to be crawled

Parameters:

w - - web page to crawl

addURLToCrawl

@CheckForNull
public WebPage addURLToCrawl(java.net.URL u)

Add the WebPage corresponding to a URL to the set of pages to crawl. If Util.shouldCrawlURL(URL, String) returns false, this method should simple return null. Otherwise, the URL is first normalized, to remove anchors (such as http://foo.com/bar.html#goo); this is done by the method Util.normalize(URL). If the URL has not been added before, this method creates a new WebPage object corresponding to the URL and remembers the association linking the URL to that web page. Otherwise, it remembers the previously used WebPage. If the WebPage has not yet been visited, the WebPage is added to the collection of pages to crawl. The WebPage corresponding to the URL is returned.

Parameters:

u - - URL of page

Returns:

- WebPage corresponding to the URL, or null if the URL should not be crawled or recorded.

performCrawl

public void performCrawl()

Crawls all WebPages, adding more Webpages to the crawl queue as links to them are found. Returns when no more pages can be crawled.

getWebPages

public java.util.Collection<WebPage> getWebPages()

Returns a collection of all known web pages, including both visited and unvisited webpages

Returns:

a collection of all web pages that were either explicitly added to the crawl list or those pointed to by a link in another crawled page.

getURLs

public java.util.Collection<java.net.URL> getURLs()

Returns a collection of the URLs of all known webpages (e.g., the URL's corresponding to the WebPages returned by the getWebPages method).

Returns:

a collection of the URLs of all known webpages.

crawl

public void crawl(WebPage w)

Crawl one WebPage. This method checks the number of crawled web pages, and does nothing if we have already reached the limit on the number of crawled web pages. Otherwise, the method:

• Increments the number of crawled web pages.
• Marks the web page as visited.
• Extracts all Hrefs from the web page. For each:
  • Find the WebPage that corresponds to the HRef/URL
  • Adds the WebPage to the crawl list if it has not been seen previously.
  • Adds the WebPage referenced by the HRef the set of links from w

Parameters:

w - - the web page to crawl

getNextPageToCrawl

@Nullable
public WebPage getNextPageToCrawl()

Determines the web page that should next be crawled and removes it from the queue, skipping over any web pages that have already been visited. Return null if there are no more pages to crawl.

Returns:

Next web page to crawl, or null if there are no more pages to crawl

lookupWebPage

@CheckForNull
public WebPage lookupWebPage(java.net.URL u)

Given a URL, returns the corresponding web page corresponding to the normalized URL

Parameters:

u - URL to lookup

Returns:

Corresponding web page, or null if no request has been made to crawl that web page.