Local / offline repository mirrors of Ubuntu & CentOS using rsync

( … because apt-mirror & debmirror are some perl/python/wget wrapper crap that does not do its one job right)

Ubuntu

We are going to isolate to only get the architecture AMD64. Using the exclude & include files you will be able to adjust for your own preferences.

First you want to find a mirror close to you. Have a look here

https://launchpad.net/ubuntu/+archivemirrors

You will have to find one with rsync to get the next bit working.

Please have 1.5 TB ready for local storage before continuing beyond this point.

Without further ado, here are the commands I have added to my /etc/crontab. Also, I am not explaining every single flag and option. You are not stupid and know how to do man and use the www as an adult. Remember to use --dry-run after the below rsync commands when you are setting them up!

rsync -vaSHP --exclude-from="/opt/mirror/exclude_ubuntu_dists" --include-from="/opt/mirror/include_ubuntu_dists" --exclude=* --delete --delete-after --delete-excluded rsync://de.archive.ubuntu.com/ubuntu/dists/ /opt/mirror/ubuntu/dists/ --bwlimit=10240 --progress --verbose --verbose

rsync -vaSHP --exclude-from="/opt/mirror/exclude_ubuntu_pool" --include-from="/opt/mirror/include_ubuntu_pool" --exclude=* --delete --delete-after --delete-excluded rsync://de.archive.ubuntu.com/ubuntu/pool/ /opt/mirror/ubuntu/pool/ --bwlimit=10240 --progress --verbose --verbose

>> ./exclude_ubuntu_dists

*-backports/
*-proposed/
*.iso
binary-i386/
debian-installer/
installer-amd64/
installer-i386/
source/
*i386*
*.orig.tar.gz

>> ./exclude_ubuntu_pool

binary-i386/
debian-installer/
installer-amd64/
installer-i386/
source/
*.iso
*.orig.tar.gz
*i386*
*-arm-*
*-aarch64-*
*-arm64-*
*-armel-*
*-armhf-*
*-mips-*
*-mipsel-*
*-powerpc-*
*-ppc64le-*
*-s390x-*
*-source-*
*-powerpc64le-*
*-ppc64el-*
*-ppc64-*

Yes, some of the excludes might be redundant, live a little. I have chosen to not include backports and proposed as I have no use for them. Your millage may vary. Add them if you want.

>> ./include_ubuntu_dists

bionic*/***
xenial*/***
focal*/***

>> ./include_ubuntu_pool

main*/***
universe*/***

I have chosen to exclude multiverse and restricted. Add them if you need them.

CentOS

I did not trash reposync in my headline. Of the three of them, reposync is not the worst. However rsync FTW! Again, first find a mirror near you. This link could be helpful. rsync sources only please.

https://www.centos.org/download/mirrors/

rsync -vaSHP --include="centosplus*/***" --include="configmanagement*/***" --include="extras*/***" --include="fasttrack*/***" --include="opstools*/***" --include="os*/***" --include="sclo*/***" --include="storage*/***" --include="updates*/***" --exclude=* --delete --delete-after --delete-excluded rsync://ftp.pasteur.fr/mirrors/CentOS/7/ /opt/mirror/centos/7/ --bwlimit=10240 --progress --verbose --verbose

rsync -vaSHP --exclude="aarch64*/***" --exclude="ppc64le*/***" --include="BaseOS*/***" --include="centosplus*/***" --include="extras*/***" --include="fasttrack*/***" --include="opstools*/***" --include="storage*/***" --exclude=* --delete --delete-after --delete-excluded rsync://ftp.pasteur.fr/mirrors/CentOS/8/ /opt/mirror/centos/8/ --bwlimit=10240 --progress --verbose --verbose

rsync -vaSHP --exclude="aarch64*/***" --exclude="ppc64le*/***" --exclude="source/" --exclude="debug/" --exclude="*.iso" --include="BaseOS*/***" --include="plus*/***" --include="extras*/***" --include="AppStream*/***" --exclude=* --delete --delete-after --delete-excluded rsync://mirror.in2p3.fr/pub/linux/rocky/8/ /opt/mirror/rocky/8/ --bwlimit=10240 --progress --verbose --verbose --time-limit=20"

CentOS “plain” is often not enough. You will want the EPEL as well. Here you go.

https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/7

https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL/8

rsync -vaSHP --delete --delete-after --delete-excluded rsync://mirror.in2p3.fr/pub/epel/7/x86_64/ /opt/mirror/centos/epel/7/x86_64/ --bwlimit=10240 --progress --verbose --verbose

rsync -vaSHP --delete --delete-after --delete-excluded rsync://mirror.in2p3.fr/pub/epel/8/Everything/x86_64/ /opt/mirror/centos/epel/8/x86_64/ --bwlimit=10240 --progress --verbose --verbose

Final note about the initial sync. It can be a handful – even for rsync 🙂 I have solved this by adding --time-limit=10 and have the rsync command run every 20 minutes. Something like this.

>> /etc/crontab

10,30,50  *  *  *  *  root  rsync ... --time-limit=10 

After the initial sync you should be OK with syncing just once or twice a day. Something like this.

30  5,17  *  *  *  root rsync ...