In last week's posting An Online Budget Questionnaire, JavaCC, and the Three Ways of Putting Together, I mentioned our public online Election questionnaire which polls opinion on how the Government could best cut spending to reduce the UK budget deficit. Coding similar questionnaires for other topics, I have found a useful tool for recording the URLs visited by a browser and hence diagnosing URL-redirection bugs: LiveHTTPHeaders.
I needed such a tool because for some jobs, I have to link our questionnaires to other survey sites. A respondent logs in to (say) www.market-research-questionnaire-hosting.com; answers a few questions; gets redirected to our questionnaire; answers it; and gets redirected back to market-research-questionnaire-hosting, whose questionnaire they finish. In due course, we merge our data with that collected by market-research-questionnaire-hosting. So that we can match respondents across the two datasets, we ask market-research-questionnaire-hosting to pass us their respondent ID as a URL parameter; and we also pass them an ID so that we can redirect the respondent back to the same questionnaire instance that they left.
So market-research-questionnaire-hosting's programmers must make their site redirect to us; and we to theirs. Sometimes, one of us gets it wrong. This happened on one job, and it was while Googling for URL-tracing tools that I found LiveHTTPHeaders. Half an hour after installing it, I'd found where the bug lay.
You can download LiveHTTPHeaders from livehttpheaders.mozdev.org/installation.html. It's free and open-source under the Gnu General Public License. As a project hosted by mozdev.org, it is intended for any Mozilla-based browser: I use it with Firefox. To install (under Windows XP), I just clicked the "Install version 0.16 of LiveHTTPHeaders now" link near the top of the above-mentioned installation page, clicked the "Install Now" button on the resulting pop-up, and restarted Firefox.
To trace URLs, go to Firefox's Tools menu and select the "Live HTTP headers" option that should now be there. This brings up a "Live HTTP headers" window. Click the "Capture" checkbox at the bottom. Then visit your URLs, and watch the headers appear in the window. Save to a text file with the "Save All" button at bottom left.
I have just traced the act of going to my home page, from there to my own copy of An Online Budget Questionnaire, JavaCC, and the Three Ways of Putting Together, and from that to the Dobbs version I'd linked from it. Here is LiveHTTPHeaders's output for the first two URLs, which are for my home page and a photo of me on it:
http://www.j-paine.org/ GET / HTTP/1.1 Host: www.j-paine.org User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB7.0 (.NET CLR 3.5.30729) Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-gb,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 115 Connection: keep-alive HTTP/1.1 200 OK Transfer-Encoding: chunked Date: Sat, 24 Apr 2010 18:50:29 GMT Server: Apache/1.3.37 (Unix) mod_ssl/2.8.28 OpenSSL/0.9.6l PHP/4.1.2 PHP/3.0.17 mod_perl/1.21 mod_fastcgi/2.4.2 Last-Modified: Fri, 23 Apr 2010 06:53:10 GMT Etag: "115a09d-91eb-4bd143d6" Accept-Ranges: bytes --------------: ----- Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: text/html ---------------------------------------------------------- http://www.j-paine.org/ish_bin_da_thumb.jpg GET /ish_bin_da_thumb.jpg HTTP/1.1 Host: www.j-paine.org User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB7.0 (.NET CLR 3.5.30729) Accept: image/png,image/*;q=0.8,*/*;q=0.5 Accept-Language: en-gb,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 115 Connection: keep-alive Referer: http://www.j-paine.org/ HTTP/1.1 200 OK Date: Sat, 24 Apr 2010 18:50:30 GMT Server: Apache/1.3.37 (Unix) mod_ssl/2.8.28 OpenSSL/0.9.6l PHP/4.1.2 PHP/3.0.17 mod_perl/1.21 mod_fastcgi/2.4.2 Last-Modified: Fri, 09 Jan 2009 10:25:13 GMT Etag: "1e40647-b8b-49672609" Accept-Ranges: bytes Content-Length: 2955 Keep-Alive: timeout=15, max=100 Connection: Keep-Alive Content-Type: image/jpeg ----------------------------------------------------------
And here are some of the URLs that my browser then visited. You can see that stylesheets and JavaScript are included, and that some URLs contain parameters and "#" internal references. I don't know who ocsp is; there turn out to be a surprising number of Googles, most not shown below; and the Dobbs site is quite labyrinthine:
http://www.j-paine.org/vd13_thumb.jpeg http://toolbarqueries.google.co.uk/search?features=Rank&sourceid=navclient-ff&client=navclient-auto-ff&iqrn=jG-&querytime=SWlC&orig=0XBvF&ch=8026b6864&q=info:http%3A%2F%2Fwww.j-paine.org%2F http://ocsp.thawte.com/ https://www.google.com/accounts/ServiceLogin?service=toolbar&nui=1&hl=en-GB&continue=http%3A%2F%2Ftoolbar.google.co.uk%2Fcommand%3Fclose_browser&zx=21607 http://www.j-paine.org/dobbs/jg_javacc_and_3_ways.html http://www.drdobbs.com/blog/archives/2010/04/an_online_budge.html https://login.techweb.com/cas/login?service=http%3A//www.drdobbs.com/blog/archives/2010/04/an_online_budge.html%3Bjsessionid%3DOOB5FJQUTBOG3QE1GHPSKHWATMY32JVN&gateway=true https://clients4.google.com/firefox/metrics/collect http://www.drdobbs.com/blog/archives/2010/04/an_online_budge.html;jsessionid=OOB5FJQUTBOG3QE1GHPSKHWATMY32JVN http://www.drdobbs.com/quickLinksContainers/topstories.jhtml?_=1272160199651 http://i.cmpnet.com/ddj/v2/css/thickbox.css http://i.cmpnet.com/ddj/v2/css/style.css http://i.cmpnet.com/ddj/v2/css/left-col-slim.css http://img.drdobbs.com/v2/css/vsHubQuickLinksDialog.css http://i.cmpnet.com/nc/mt/js/togglediv.js http://img.drdobbs.com/v2/js/vsHubQuickLinksDialog.js http://s7.addthis.com/js/250/addthis_widget.js#pub=xa-4af4a81901b903fa http://jlinks.industrybrains.com/jsct?sid=644&ct=DDJ_ROS&num=5&layt=388x350&fmt=simp http://admin.brightcove.com/js/experience_util.js http://adserver.adtechus.com/addyn/3.0/5242.1/1183600/0/6/ADTECH;alias=Dr.Dobbs_Blogs_Tile_125x125;key=;kvarticleid=;kvauthor=;loc=100;target=_blank;grp=305489237;misc=1272160190419 http://oasc05134.247realmedia.com/RealMedia/ads/adstream_jx.ads/TWEBddj/technology/1%7BTIME_DATE_STAMP%7D@Middle1
Thanks to one of the URLs I saw,
I now know that my
Dobbs headshot is
http://i.cmpnet.com/ddj/v2/images/headshots/Jocelyn_Ireson-Paine77x95.jpg
.