How to mirror CPAN

There are several ways to mirror CPAN depending upon what you want to achieve.

How do I create a private or offline mirror?

minicpan from CPAN::Mini is the best tool for this. Also look at CPAN::Mini::Inject which allows you to add your own modules into your private mirror.

Requirements for a full / public mirror

  • Good internet connectivity
  • Around 1GB of storage space for just the current modules.
  • Around 11GB of storage space for the full mirror.

It's highly recommended that you also subscribe to the announcements-only cpan-mirrors mailing list by emailing cpan-mirrors-subscribe at perl.org.

Tools

CPAN::Mini provides you with a minimal mirror of CPAN (the latest version of all modules). This makes working offline easy, it is the best tool of you are running a private mirror.

New: rrr-client allows instant mirroring, and should be used on official public mirrors where possible. See instant mirroring instructions.

rsync is the best tool if you need to mirror the whole of CPAN or if you are providing a public mirror. Rsync Instructions.

Only use FTP if these other methods are absolutely impossible. Never mirror with HTTP - you will end up with a million duplicate files in tens of gigabytes.

Which CPAN Mirror should I use?

Near future: Tier 1 mirrors are instantly updated and can be accessed as rsync://cpan-rsync.perl.org/CPAN/.

Current: Find your nearest rsync enabled site on http://www.cpan.org/SITES.html, or use mirrors.json especially if you are building a tool which lets the user select a mirror. You can also sync from rsync://cpan.rsync.perl.org/CPAN/, though you currently might get better performance from a "local" mirror.

Using rsync

Please limit to once or twice a day. For more frequent updates please see Instant mirroring.

On Unix systems

/usr/bin/rsync -av --delete cpan-rsync.perl.org::CPAN /project/CPAN/

Using 'crontab' you can make rsync run once a day, for example
4 40 * * * sleep $(expr $RANDOM \% 7200); /usr/bin/rsync -av --delete cpan-rsync.perl.org::CPAN /project/CPAN/
The "sleep $(...);" statement makes the command delay up to 2 hours before running rsync; the advantage of this is that you (and everybody else) won't access the mirror at the same time!

On Windows systems

C:\Program Files\Rsync\rsync -av --delete cpan-rsync.perl.org::CPAN /project/CPAN/

Using the 'AT' tool, you can schedule rsync to run daily, for example:
AT 20:00 /every:M,T,W,Th,F,S,Su "C:\Program Files\Rsync\rsync -av --delete cpan-rsync.perl.org::CPAN /project/CPAN/"

How do I create a public mirror?

  • Consider Instant mirroring, required if you wish to be a tier 1 mirror, or..
  • rsync once a day
  • Provide (in order of preference) rsync, HTTP and/or FTP public access
  • To be added to http://www.cpan.org/SITES.html and mirrors.json please complete the template confirming the public accessible URLs to your mirror: rsync, ftp, http and email it to cpan@perl.org.

Instant mirroring

Please join the trial. This will become the preferred method of mirroring for public sites, it is required for all Tier 1 mirrors. Currently being tested.