Question: Are there any automatic differentiation tools that we can use?
Answer: Well, you could try http://www.sc.rwth-aachen.de/Diplom/AndreVehreschild.html and http://www.cs.cornell.edu/home/verma/AD/ but the second link seems to be dead. I think it would be easier to program your particualr derivatives manually.
Question: What does factoring the matrix, etc. have to do with computing a search direction?
Answer: I should have phrased this more clearly. The question asks how is the BFGS matrix stored, and what operations are done in using it to find the search direction?
Question: Regarding question 3 of the homework, am I to write also the codes for truncated newton and finite difference newton methods? Or should I download existing packages like TNPACK? Thanks.
Answer: You should find options that make fminunc use these methods. Don't write your own or use other software. If you believe that fminunc cannot be made to use some of these methods, then choose 4 fminunc algorithms and compare with those.
Question: I have a question about Cholesky Factorization updating. The formula you gave me is to update with a rank one matrix. But BFGS has a rank 2 update. So, how can I implement this for BFGS update. Do I have to perform this thing twice? >
Answer: Yes, you need to do 2 updates per iteration.
Question: There seem to be some errors in the excerpt from Gill, Murray, and Wright.
Answer: Sad but true. In the algorithm at the bottom of p. 42, in the line that updates u_r, the right hand side should have u_r, not u_j.
The algorithm at the top of p.43 is not so easily fixed. Right now, I believe it to be unfixable, although I would be happy if someone proved me wrong. A correct algorithm is to reuse the algorithm on p.42, but to change the signs in the update of t_j and l-bar from plus to minus. If you do this, though, you need to insert a test that sets d-bar to epsilon if it ever becomes less than epsilon.
By the way -- as you implement the algorithm, don't use any more vectors or matrices than are absolutely necessary.