Author: John J. Lee <jjl@pobox.com>
Keywords: FAQ,cookie,HTTP,HTML,form,table,Python,web,client,client-side,testing,sniffer,https,script,embedded
Date: 2005-01

# Python web-client programming general FAQs

- Is there any example code?

  There's (still!) a bit of a shortage of example code for ClientCookie and ClientForm &co., because the stuff I've written tends to either require access to restricted-access sites, or is proprietary code (and the same goes for other people's code).

- HTTPS on Windows?

  Use this _socket.pyd, or use Python 2.3.

- I want to see what my web browser is doing, but standard network sniffers like ethereal or netcat (nc) don't work for HTTPS. How do I sniff HTTPS traffic?

  Three good options:

  - Mozilla plugin: livehttpheaders.
  - ieHTTPHeaders does the same for MSIE.
  - Use lynx `-trace`, and filter out the junk with a script.

  I'm told you can also use a proxy like proxomitron (never tried it myself). There's also a commercial MSIE plugin.

- Embedded script is messing up my web-scraping. What do I do?

  It is possible to embed script in HTML pages (sandwiched between `<SCRIPT>here</SCRIPT>` tags, and in `javascript:` URLs) - JavaScript / ECMAScript, VBScript, or even Python. These scripts can do all sorts of things, including causing cookies to be set in a browser, submitting or filling in parts of forms in response to user actions, changing link colours as the mouse moves over a link, etc.

  If you come across this in a page you want to automate, you have four options. Here they are, roughly in order of simplicity.

  - Simply figure out what the embedded script is doing and emulate it in your Python code: for example, by manually adding cookies to your `CookieJar` instance, calling methods on `HTMLForm`s, calling `urlopen`, etc.
  - Dump ClientCookie and ClientForm and automate a browser instead (eg. use MS Internet Explorer via its COM automation interfaces, using the Python for Windows extensions, XXX Mozilla automation & XPCOM / PyXPCOM, Konqueror & DCOP / KParts / PyKDE).
  - Use Java's httpunit from Jython, since it knows some JavaScript.
  - Get ambitious and automatically delegate the work to an appropriate interpreter (Mozilla's JavaScript interpreter, for instance). This approach is the one taken by DOMForm (the JavaScript support is "very alpha", though!).
- Misc links
  - Another Java thing: maxq, which provides a proxy to aid automatic generation of functional tests written in Jython using the standard library unittest module (PyUnit) and the "Jakarta Commons" HttpClient library.
  - A useful set Zope-oriented links on tools for testing web applications.
  - O'Reilly book: Spidering Hacks. Very Perl-oriented.
  - Useful Mozilla plugin which, amongst other things, can display HTML form information and HTML table structure(thanks to Erno Kuusela for this link).
  - IOpus Internet Macros Cheap and nasty macro recording for IE. It works, just barely. Commercial software.
  - Open source functional testing tools. A nice list.
  - A HOWTO on web scraping from Dave Kuhlman.
- Will any of this code make its way into the Python standard library?

  The request / response processing extensions to urllib2 from ClientCookie have been merged into urllib2 for Python 2.4. The cookie processing has been added, as module cookielib. Eventually, I'll submit patches to get the http-equiv, refresh, and robots.txt code in there too, and maybe `mechanize.UserAgent` too (but not `mechanize.Browser`). The rest, probably not.

John J. Lee, January 2005.