(Guide) apt-cacher for caching of downloaded packages. Save time on reinstalls

[staticn0de]

Guide finished, please post any issues

Hi all,

I'm back again. If anyone here is like me and you like to mess around with Kali to the point of catastrophic failure, you have probably spent a fair amount of time reinstalling and downloading from the (on occasion) over burdened mirror servers. Even if they are not running slow, some of these packages are a decent size and we don't all have super fast / limitless connections.

So, I was looking for options to save on the download. I considered mirroring both the main Kali repo and the Kali-security repo. This is about 360GB total and requires you to keep it up to date by polling the server roughly every four hours. If that is not a downside in itself, remember you are mirroring hundreds of GB in packages you may never even know about, let alone consider installing.

I've mentioned apt-cacher in a previous post. This great app lets you store any packages you download locally. For example, I download oclhastcat from the kali repo which is about 500Mb give or take. As it is installed, the .deb file which was downloaded is stored on my apt-cacher host. All I do is point apt at it. The best part is, I can store these cache files on the machine that I regularly reinstall, I just make a new partition after shrinking '/'or use a usb / hard drive.

You need a static IP for this to work. Set one on the computer running apt-cacher now. (If the computer running apt-cacher is also the client (the machine which is regularly downloading or reinstalling), it still needs a static IP) I'll be using the static IP 192.168.0.50 as my apt-cacher server and client (as my client is running the apt-cacher server)

[staticn0de]

Lets install apt-cacher

apt-get install apt-cacher

While the install is running you will be prompted for a daemon mode for apt-cacher. Select 'daemon'

When the install is finished, restart apache2 with the following command

/etc/init.d/apache2

Navigate to http://192.168.0.50:3142 and you will receive a plain white page (no error messages). This indicates that apt-cacher is running.

I had issues until I set my allowed hosts. Adjust the following setting to allow all clients to fetch packages through the apt-cacher server

nano +178 /etc/apt-cacher/apt-cacher.conf

adjust the line to read what is below by removing the #

allowed_hosts = *


Restart apt-cacher

/etc/init.d/apt-cacher restart

Now to configure Apt-get to run through apt-cacher by using a proxy. An example of my file is included below. Change the IP to match your apt-cacher host.

My file

root@kali:~#cat /etc/apt/apt.conf.d/01proxy
Acquire::http::Proxy "http://192.168.0.50:3142";
root@kali:~#

That is all the configuration we need if the host is the same as the client. If you have a dedicated machine to act as the server (the one apt-cacher is installed on) and want clients to make use of it, you can alter either of the following files on the client machine to point them at the apt-cacher server.

The 01proxy file
On the client machine, create file at /etc/apt/apt.conf.d/01proxy and populate it the same as the file shown above (that is, with the ip address of the machine running apt-cacher) and then run apt-get update.

The sources file
I don't like this method as much as it takes more effort. As with before, it's all about pointing to the apt-cacher server.

Open the sources file with nano /etc/apt/sources.list and then change your entries using the same method as below.

Assuming an entry looks like this

deb http://http.kali.org/kali kali main non-free contrib

Becomes what is shown below. The IP is the apt-cacher server

deb http://192.168.0.50:3142/http.kali.org/kali kali main non-free contrib

On the clients, it is time to update the cache. Run the following

apt-get update

if you receive errors about 'unable to access cache' you didn't run '/etc/init.d/apt-cacher restart' when you finished making configuration changes to proxies or sources.list

Go ahead and install something, I picked something large-ish to start with so I used

apt-get install linux-headers-$(uname -r)

Now, check the cache folder on the apt-cacher server. For me (as I haven't moved it yet this is located at the default location shown below.

cd /var/cache/apt-cacher/packages
ls -lha

Look at that! all those .deb files are now in our cache folder! Lets see if it actually saves time.


apt-get remove linux-headers-$(uname -r) && apt-get autoremove

Now they are deleted, reinstall them.

apt-get install linux-headers-$(uname -r)

Did you notice how they packages did not need to download? I know that is only one file. Imagine that is the ~500mb of updates normally facing a reinstall of Kali. Big time and download saver, right?

[staticn0de]

Now, the important part. I can almost hear some people thinking that 'yay, I can download to my /var directory which is promptly wiped when I reinstall anyway' (if the apt-cacher server and client are the same machine). Don't fret, we have a couple of options. We can use any of the following. Read the descriptions to pick what is for you.

Making a dedicated partition to hold apt-cacher files

Be warned, this can make you lose a lot of data if you do it wrong / have bad karma. Don't post saying you wiped your drive as you have been warned.

This would be the cleanest option. This way, if we reinstall we always have our backup partition holding our install files. Regardless or what we choose, its all about having a mount point and setting apt-cacher at it.

1. Boot into a live mode and launch gparted from the System Tools menu
2. I'll be resizing my root partition as it is the largest. Right click your root partition and select Resize / Move. for me, this is /dev/sda1 and has the boot flag (as I don't have a separate boot partition.
3. Drag the right hand arrow at the top of the screen to the left. You will see the Free space following (MiB) increasing as you drag left. This will be the available space for your apt-cacher files. I made mine 5Gb but you should adjust depending on the amount of packages you install.
4. Gparted will now display the free space, right click and select new. I left all values on this screen default except the file system which I changed to Ext4 and clicked Add
5. At the top of the screen, click the green arrow to confirm the changes. Shortly after you will need the operations completed message.
6. Shutdown and boot back into normal mode.
7. In normal mode, use 'fdisk -l' and you will see your new partition. For me, it is /dev/sda3 with an Id of 83 and System type Linux. We need to mount this so we can store files on it.
8. Make a directory for it using the following 'mkdir /mnt/deb-cache' and mount it using 'mount /dev/sda3 /mnt/deb-cache'. Make sure you substitute /sda3 and /deb-cache for whatever you are using.
9.Type in 'df -h' and you will see your partition mounted on /mnt/deb-cache with roughly your partition size available.
10. Make this partition available at boot time by adding it to fstab. use 'nano /etc/fstab' and then enter the following. Substitute the drive for your drive.

/dev/sda3 /mnt/deb-cache ext4 defaults 0 0

11. Save and close fstab. Use the following to simulate boot time and confirm your drive will mount at boot. substitute for your values.

umount /mnt/deb-cache && mount -a

12. No errors is a good sign. Now to stop apt-cacher, move everything to our new drive, change the package storage location, start apt-cacher and test it!
13. Use '/etc/init.d/apt-cacher stop' to stop apt-cacher
14. Apt-cacher stores it's files in /var/cache/apt-cacher by default. Use the command below to copy these files to /mnt/deb-cache. This will store all the files in the root of deb-cache. For example, the path to packages will be /mnt/deb-cache/packages/ . If you want files stored in an apt-cacher folder within /mnt/deb-cache, delete the training '.' from the cp command

cp -a /var/cache/apt-cacher/. /mnt/deb-cache

15. When finished, delete the older apt-cacher folder with 'rm -rf /var/cache/apt-cacher/'
16. Change the location apt-cacher looks at with 'nano /etc/apt-cacher/apt-cacher.conf' and then uncomment and ammend the line cache_dir to that below

cache_dir = /mnt/deb-cache

17. Start apt-cacher with '/etc/init.d/apt-cacher start'
18. Run an 'apt-get update' and then install something. I chose remmina 'apt-get install remmina'
19. If you used the same as me, check /mnt/deb-cache/packages and you will see that the .debs for remmina have joined our linux-headers files! Fantastic.


Using an external hard drive or usb

While this will work, you will need to have them connected when you are installing packages. Either way, its the same process in the end. Get the device name from fdisk -l, make the deb-cache folder and mount it automatically using fstab. Make sure you update the /etc/apt-cacher/apt-cacher.conf file cache_dir.

So I've reinstalled, how do I take advantage of my stored debs?

This is the easy part. Install Kali and open terminal, install apt-cacher using the instructions above (making sure you set up the proxy, set static IP, set the cache_dir and the allowed_hosts, create the deb-cache folder and mount your /dev/sda3 partition back on it, update fstab and remove the apt-cacher directory within /var/cache. After all that, start apt-cacher. If you get an error, run /etc/init.d/apt-cacher stop && /etc/init.d/apt-cacher start

Run apt-get update

Start installing! it won't need to download anything that is already stored on our backup partition!