The WWW Utils Module (
This module provides utilities to deal with the internet. All application code should access the internet through this module, since this where support for proxies and caching is implemented.
This module provides three main approaches to handling access to internet resources :
urllib based access (
Opens a url specified by the
requests based access (
Gets a pre-configured
suds based SOAP access (
Creates and returns a suds/SOAP client instance bound to the provided
The backends provided by these modules have integrated caching mechanisms built-in to speed up access to internet based resources.
Redirect caching speeds up network accesses by saving
redirects, and not needing to get the correct URL on a second access. This
redirect cache is stored as a pickled object in the
folder. The effect of this caching is far more apparent when a replicator
cache is also used.
Redirect caching is only supported by the urllib based backend
tendril.utils.www.bare), and is likely going to be phased out
entirely in the future.
Full Response Caching
This is a more typical kind of caching, which uses a backend-dependent mechanism to maintain a cache of full responses received.
Consider replacing uses of urllib/urllib2 backend with
requests and simplify this module. Currently, the
cache provided with the
requests implementation here
is the major bottleneck and seems to cause a major