Saving mobile data usage with Ziproxy

So, for a month or so I’m stuck using Orange 3G mobile internet. It’s not fun, and there is no other choice where I’m staying. At £12 per gigabyte it’s pretty damn expensive. I figured there has to be a way to economise my internet usage (and that doesn’t mean not using it…) and I found my solution with a piece of software called Ziproxy.

Ziproxy is a proxy server that doesn’t cache your traffic instead it compresses it on the fly to save bandwidth. I installed it on one of my VPS boxes and connect via an SSH tunnel to ensure it stays secure. In just under a week it has saved me 292MB of bandwidth (or almost 1/3 of my £12 allowance)

Raw traffic into the proxy according to the traffic log is 564MB, compressed traffic moving over the mobile link is 272MB. Now this is with most of the settings cranked to the max. Images processed are converted to black and white (Colour is expensive, yo) and between 10-25% JPEG compression. It’s not pretty, but it’s saving me money. The basic architecture is as follows:

Laptop –> SSH Tunnel to VPS over 3G –> [Ziproxy](http://ziproxy.sourceforge.net/) listening on localhost only –> Traffic is forwarded to Squid listening on localhost only –> Internet

To set this up, you’ll need access to a server on the internet that you have root access to. I’ve used AllSimple a UK based host for nearly two years and they are probably the best webhost I’ve ever used. I’m using their £15+vat / year server to run my proxy (alongside irssi, and newsbeuter) and it works great. This is going to be a brief overview so it assumes you have some knowledge in setting up SSH connections and editing config files. I

Once you have the server running, and you can SSH into it. Install Ziproxy using your prefered method. I use Debian, so I just install the package using apt-get install [Ziproxy](http://ziproxy.sourceforge.net/). Now you’ll need to configure it to only listen locally so no-one can abuse your proxy and do naughty things (Like look at porn in the UK…..). This can be achieved by editing the config file found in /etc/[Ziproxy](http://ziproxy.sourceforge.net/)/Ziproxy.conf:

## Accepts conections only from that address. ## WARNING: Remember to restrict the access to Ziproxy ## if your machine is directly connected to the Internet. OnlyFrom = “127.0.0.1”

This makes only respond to requests from 127.0.0.1 which is what we’ll appear to be coming from when we’ve set up out SSH tunnel using Putty. Whilst we’re in the config file, you should consider making the rest of the required changes as follows. Make sure you understand what they do before you copy and paste…

You can set the useragent for all proxied clients. This is helpful when configuring and testing because it means you can be sure the connection is passing through the proxy correctly. Once everything is working, you could disable it but I haven’t bothered

## Replaces the User-Agent data sent by the client with a custom string,
## OR defines User-Agent with that string if that entry was not defined.
## If disabled, [Ziproxy](http://ziproxy.sourceforge.net/) will just forward the User-Agent sent by the client.
## Normally you will want to leave this option DISABLED (commented).
##
## It's useful if you, for some reason, want to identify all the clients as
## some specific browser/version/OS.
## Certain websites may appear broken if the client uses a different browser than
## the one specified here.
## Certain webservers may break completely when an unrecognized User-Agent is provided
## (for example: www.rzeczpospolita.pl).
##
## Undefined by default (leave User-Agent as defined by the client).
 RedefineUserAgent = "Mozilla/5.0 (compatible; UltraBrowser/8.1; CP/M; console40x24; z80)"

To save more bandwidth, enable the following

## Whether to try to optimize HTML, CSS and Javascript, thus reducing their size
## ProcessHTML: text/html
## ProcessCSS:  text/css
## ProcessJS:   application/[x-]javascript)
## Although such data may be Gzipped too, optimizing prior to Gzipping normally
## reduces the data size even further.
## The final size depends much on how unoptimal is the coding of such data;
## some sites already present HTML pre-optimized so, in such cases, there won't
## be much gain.
## Note: Due to the higher complexity of such optimization, there's some risk of a page
## being corrupted.
## ****** THESE OPTIONS ARE EXPERIMENTAL ******
##
 ProcessHTML = true
 ProcessCSS = true
  ProcessJS = true

Remove transparency, and de-animate GIF files. Then make images grayscale.

## If enabled, will discard PNG/GIF/JP2K transparency and de-animate
## GIF images if necessary for recompression, at the cost of some image
## distortion.
## Note: Images with useless transparency/alpha data (all pixels
##       being opaque) do not require this option. In such cases [Ziproxy](http://ziproxy.sourceforge.net/)
##       will detect that and remove the useless data automatically.
## Disabled by default.
 AllowLookChange = true

## If enabled, convert images to grayscale before recompressing.
## This provides extra compression, at the cost of losing color data.
## Note: Not all images sent will be in grayscale, only the ones
##       considered worth recompression that way.
## Disabled by default.
ConvertToGrayscale = true

There are more settings including levels of JPEG compression and numerous even more technical options which you can find in the documentation. (man Ziproxy)

To set putty up, go into Connection -> SSH –> Tunnels and set a local port up with a destination of localhost:3128 (the port we’re running Ziproxy on).

 

If you’ve set up Ziproxy correctly, and it’s running when you visit a site that tells you your IP address it should report that of your server rather than your actual IP address. If it doesn’t go through your config and check everything is running correctly. When I visit IP Chicken I see that the header image is being heavily compressed (Yay!) and the user agent is what I set in my Ziproxyconfiguration file. Success!!

ipchicken
Screenshot showing images being converted to greyscale

 

Additionally, I am running squid on my VPS as an upstream proxy to reduce bandwidth to the internet. This isn’t necessary at all, but is easy to add by setting the NextProxy settings in the Ziproxy configuration file. In my set up, Squid is listening on port 3129 (and only on localhost, again this is important to avoid being abused as an open proxy!) so I forward traffic to it using these settings.

  ## Forward everything to another proxy server.
  ## Modifications/compression is still applied.
  ## Default: none (disabled)
   NextProxy="127.0.0.1"

  ## TCP port to be used by NextProxy.
  ## Default: 8080
  NextPort=3129

I think it might be better to run Squid before Ziproxy but I haven’t tested it. The logic being that squid will then cache the compressed files from Ziproxy reducing the server load from not having to recompress regularly accessed files. Maybe a local squid on my laptop to cache the compressed files on my side of the 3G link would make more sense (and save even more bandwidth!). I need to do further testing, but I’m rather happy with saving 292MB of traffic (equivalent to £3.51!)

I sometimes find that pages simply do not load. This is not a common occurence, when it happens I either see if the page is available over HTTPS (These are not proxied), or drop the SSH connection and try again once I’ve reconnected. This usually fixes the issue. Only once have I had to stop and restart both proxies.

This solution also works for android devices. I already use Irssi Connectbot on my tablet (It supports SSH tunneling), coupled with Proxy Droid I can get compressed internet automatically on my tablet/phone whenever I want to. This could potentially be used to avoid censorship as well as just reducing data consumption. As long as you could make an outgoing SSH connection, then remainder of your session is hidden from the local network. Running SSH on port 443 or other well known ports can sometimes get around networks that block port 21.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s