[ Jocelyn Ireson-Paine's Home Page | Publications | Dobbs Code Talk Index | Dobbs Blog Version ]

LiveHTTPHeaders for Tracing URLs

In last week's posting An Online Budget Questionnaire, JavaCC, and the Three Ways of Putting Together, I mentioned our public online Election questionnaire which polls opinion on how the Government could best cut spending to reduce the UK budget deficit. Coding similar questionnaires for other topics, I have found a useful tool for recording the URLs visited by a browser and hence diagnosing URL-redirection bugs: LiveHTTPHeaders.

I needed such a tool because for some jobs, I have to link our questionnaires to other survey sites. A respondent logs in to (say) www.market-research-questionnaire-hosting.com; answers a few questions; gets redirected to our questionnaire; answers it; and gets redirected back to market-research-questionnaire-hosting, whose questionnaire they finish. In due course, we merge our data with that collected by market-research-questionnaire-hosting. So that we can match respondents across the two datasets, we ask market-research-questionnaire-hosting to pass us their respondent ID as a URL parameter; and we also pass them an ID so that we can redirect the respondent back to the same questionnaire instance that they left.

So market-research-questionnaire-hosting's programmers must make their site redirect to us; and we to theirs. Sometimes, one of us gets it wrong. This happened on one job, and it was while Googling for URL-tracing tools that I found LiveHTTPHeaders. Half an hour after installing it, I'd found where the bug lay.

You can download LiveHTTPHeaders from livehttpheaders.mozdev.org/installation.html. It's free and open-source under the Gnu General Public License. As a project hosted by mozdev.org, it is intended for any Mozilla-based browser: I use it with Firefox. To install (under Windows XP), I just clicked the "Install version 0.16 of LiveHTTPHeaders now" link near the top of the above-mentioned installation page, clicked the "Install Now" button on the resulting pop-up, and restarted Firefox.

To trace URLs, go to Firefox's Tools menu and select the "Live HTTP headers" option that should now be there. This brings up a "Live HTTP headers" window. Click the "Capture" checkbox at the bottom. Then visit your URLs, and watch the headers appear in the window. Save to a text file with the "Save All" button at bottom left.

I have just traced the act of going to my home page, from there to my own copy of An Online Budget Questionnaire, JavaCC, and the Three Ways of Putting Together, and from that to the Dobbs version I'd linked from it. Here is LiveHTTPHeaders's output for the first two URLs, which are for my home page and a photo of me on it:

http://www.j-paine.org/

GET / HTTP/1.1
Host: www.j-paine.org
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.3) 
Gecko/20100401 Firefox/3.6.3 GTB7.0 (.NET CLR 3.5.30729)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Sat, 24 Apr 2010 18:50:29 GMT
Server: Apache/1.3.37 (Unix) mod_ssl/2.8.28 OpenSSL/0.9.6l PHP/4.1.2 
PHP/3.0.17 mod_perl/1.21 mod_fastcgi/2.4.2
Last-Modified: Fri, 23 Apr 2010 06:53:10 GMT
Etag: "115a09d-91eb-4bd143d6"
Accept-Ranges: bytes
--------------: -----
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html
----------------------------------------------------------
http://www.j-paine.org/ish_bin_da_thumb.jpg

GET /ish_bin_da_thumb.jpg HTTP/1.1
Host: www.j-paine.org
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.3) 
Gecko/20100401 Firefox/3.6.3 GTB7.0 (.NET CLR 3.5.30729)
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://www.j-paine.org/

HTTP/1.1 200 OK
Date: Sat, 24 Apr 2010 18:50:30 GMT
Server: Apache/1.3.37 (Unix) mod_ssl/2.8.28 OpenSSL/0.9.6l PHP/4.1.2 
PHP/3.0.17 mod_perl/1.21 mod_fastcgi/2.4.2
Last-Modified: Fri, 09 Jan 2009 10:25:13 GMT
Etag: "1e40647-b8b-49672609"
Accept-Ranges: bytes
Content-Length: 2955
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: image/jpeg
----------------------------------------------------------

And here are some of the URLs that my browser then visited. You can see that stylesheets and JavaScript are included, and that some URLs contain parameters and "#" internal references. I don't know who ocsp is; there turn out to be a surprising number of Googles, most not shown below; and the Dobbs site is quite labyrinthine:

http://www.j-paine.org/vd13_thumb.jpeg
http://toolbarqueries.google.co.uk/search?features=Rank&sourceid=navclient-ff&client=navclient-auto-ff&iqrn=jG-&querytime=SWlC&orig=0XBvF&ch=8026b6864&q=info:http%3A%2F%2Fwww.j-paine.org%2F
http://ocsp.thawte.com/
https://www.google.com/accounts/ServiceLogin?service=toolbar&nui=1&hl=en-GB&continue=http%3A%2F%2Ftoolbar.google.co.uk%2Fcommand%3Fclose_browser&zx=21607
http://www.j-paine.org/dobbs/jg_javacc_and_3_ways.html
http://www.drdobbs.com/blog/archives/2010/04/an_online_budge.html
https://login.techweb.com/cas/login?service=http%3A//www.drdobbs.com/blog/archives/2010/04/an_online_budge.html%3Bjsessionid%3DOOB5FJQUTBOG3QE1GHPSKHWATMY32JVN&gateway=true
https://clients4.google.com/firefox/metrics/collect
http://www.drdobbs.com/blog/archives/2010/04/an_online_budge.html;jsessionid=OOB5FJQUTBOG3QE1GHPSKHWATMY32JVN
http://www.drdobbs.com/quickLinksContainers/topstories.jhtml?_=1272160199651
http://i.cmpnet.com/ddj/v2/css/thickbox.css
http://i.cmpnet.com/ddj/v2/css/style.css
http://i.cmpnet.com/ddj/v2/css/left-col-slim.css
http://img.drdobbs.com/v2/css/vsHubQuickLinksDialog.css
http://i.cmpnet.com/nc/mt/js/togglediv.js
http://img.drdobbs.com/v2/js/vsHubQuickLinksDialog.js
http://s7.addthis.com/js/250/addthis_widget.js#pub=xa-4af4a81901b903fa
http://jlinks.industrybrains.com/jsct?sid=644&ct=DDJ_ROS&num=5&layt=388x350&fmt=simp
http://admin.brightcove.com/js/experience_util.js
http://adserver.adtechus.com/addyn/3.0/5242.1/1183600/0/6/ADTECH;alias=Dr.Dobbs_Blogs_Tile_125x125;key=;kvarticleid=;kvauthor=;loc=100;target=_blank;grp=305489237;misc=1272160190419
http://oasc05134.247realmedia.com/RealMedia/ads/adstream_jx.ads/TWEBddj/technology/1%7BTIME_DATE_STAMP%7D@Middle1

Thanks to one of the URLs I saw, I now know that my Dobbs headshot is http://i.cmpnet.com/ddj/v2/images/headshots/Jocelyn_Ireson-Paine77x95.jpg.