CMSC 498w - Web Architecture and Programming
Programming Assignment #1

DUE:
uploaded to the submission site by 11:59p, Eastern Standard Time, TUESDAY March 7.

Late Policy

The project is due before the stroke of midnight on Tuesday (3/7). Submissions received within two days after that (by 11:59p EST on March 9) will receive a 10 points (of 100) markdown. Submissions will be accepted until MONDAY night (11:59p on March 13) at a loss of 25 points. Thereafter they will not be accepted.

NOTE:The TA is not able to give extensions or waive these rules in any way under any circumstances. Anyone with a problem must contact Prof. Hendler prior to the deadline in question.

Assignment

For this project you will be writing a web crawler (a/k/a spider, robot). To do this, you will need to write and read HTTP 1.0 headers and to do the right thing given what comes back (details below). Your crawler will be "polite" in that it will have the right headers for identifying itself and will respect the robots exclusion protocol (The good news is we will make this easy to do, see below). Details:

• You will be given a starting url to use as your code's only command line argument. It will download that starting page, and recursively follow all the links given in it. To simplify parsing, the links will simply be given as a list of (potentially relative) uris in the body of the HTTP response. All pages will be text/plain.
• Your crawler will output a list of visited uris and the response codes it received when it visited them and an appropriate "comment" (see output details below). If it receives an ACCEPT code (200s) it will crawl the URLs (note that is URL, not URI) in that document and the comment will be "OK". If it receives an ERROR code (400s) it will note the code and comment "NG". If it receives a redirection code (300s) it will note the code and the comment will be the new location (and, of course, it will appropriately handle the new URL.)
• For completeness, your robot should report 000 for any other response header it receives. On the site you will be developing and running your robots, there will not be any 500 or other such codes, so if you receive one it is probably in your code (or someone has crashed our server).
• If you encounter a URI with a different URI scheme than HTTP, then you should output a 999 code, this URI and its scheme would be the comment (and do not attempt to visit it.)
• Your crawler must do Loop and Dup elimination, it should not visit any URL more than once.
• If your crawler is not permitted to go to a URL (due to robot exclusion) that page should also have code 999 with the comment "Excluded".

You must use HTTP 1.0 (not 1.* or 1.1) in your headers.

Supporting Code

You will be allowed to use java.net.URL to parse urls and retrieve sections of them. However, you may not use its connect or openConnection methods. Instead, you will be given a function that takes as input a hostname, port, and text string (the web request) and will return the response it receives from the web server (details will be made available with the code). You will be responsible for formatting the request and parsing the response.

If you prefer not to use this code, you may use the java socket package directly. The key point is that you may not use a library that creates the HTTP requests for you or parses the responses. You must handle parsing the responses yourself. (Standard parsing tools like the regexp package are allowed of course.)

You will also be given the .jar file from " websphinx." All you may use out of this package is the RobotsExclusion class. Note that any other use of websphinx is strictly prohibited! Before downloading any uri you must use this class to check whether the site has a robots.txt that forbids you from doing so.

Polite Robots

Your robot should be polite. As well as not requesting pages that are forbidden to it, it must include appropriate "politeness" headers. These include:

• Your robot should identify itself: the HTTP requests you create must include a user-agent header field. Your user agent must be "MyRobotLastname" (where Lastname is, of course, your Last Name - i.e. my robot is named MyRobotHendler). Please note this is case sensitive, and you are using "camel" case.
• Your robot must identify its machine: You don't actually have to do anything for this to happen as long as the machine you run from has a DNS entry - if it doesn't (i.e. you are running from work with some strange security proxy or such) then you should make the final code run from a glue machine or discuss this with the instructor or TA (I don't expect anyone to be in this situation).
• Your robot must identify a contact: Use the HTTP from field to provide a contact email address (yours please!).

Other details

Pointers to the code libraries being made available, the root URI to use, and details of submission of the project will be made available on the Forum page for CMSC 498w (starting on Friday, 2/24).

What You Will Turn In

You will submit your code, as well as a text file containing all the uris visited and their response codes. The output file should be formatted as code, space, URI, space, comment (or redirect), CRLF. For example:

200 http://foo.example.com/ OK 
301 http://foo.example.com/bar http://foo.example.com/bar2
200 http://foo.example.com/bar2 OK
999 mailto:hendler@cs.umd.edu mailto
400 http://foo.example.com/baz NG
999 http://foo.example.com/private/ Excluded

You may visit the pages in any search order you prefer.

There is a vague chance that our server will not handle the complete load of this class. If we have problems, we will ask you to put a time delay into your robot, and explain how in class (and this will be noted in the forum). We do not expect this to be the case.