cs132.webSpider
Class Util

java.lang.Object
  cs132.webSpider.Util

public class Util
extends java.lang.Object

Method Summary

static java.net.URL	normalize(java.net.URL u) Given a URL, normalize it by removing any references or queries.
static boolean	shouldCrawlURL(java.net.URL u, java.lang.String root) Determine if a URL should be recorded and crawled.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

normalize

public static java.net.URL normalize(java.net.URL u)

Given a URL, normalize it by removing any references or queries. For example, http://foo.com/index.html#abc gets normalized to http://foo.com/index.html

Parameters:

u - -- URL to normalize.

Returns:

-- normalized URL

shouldCrawlURL

public static boolean shouldCrawlURL(java.net.URL u,
                                     java.lang.String root)

Determine if a URL should be recorded and crawled. A url should not be recorded or crawled if any of the following are true:

• The url doesn't start with the web root passed to this method. For example, by passing a root of http://www.cs.umd.edu/, you can limit the crawl to web pages on the CS department web server.
• If the URL contains a query component, it should not be crawled.
• If the URL doesn't have a protocol of http or file, it should not be crawled.
• Determine the mimetype of the file using the FileNameMap returned by the URLConnection.getFileNameMap() method. The web page should only be crawled if the FileNameMap can't guess an extension (i.e., the getContentTypeFor method returns null) or if the returned mimetype starts with text/html.

Parameters:

u - -- URL to check

root - -- root of URL's that will be checked

Returns:

-- true if the URL should be recorded and crawled.