Create a local PyPI

Postado por em
Republicado de aboutsimon

Create a local PyPI mirror

Python

I assume you’re using Linux
Tested with Ubuntu 10.04.4 LTS

Some servers I administrate don’t have an internet connection, because of security reasons or they just don’t need to have one. The lack of an internet connection is pretty bad, when it comes to installing packages from PyPI. To enable my offline servers to get packages from PyPI I need a local mirror. First I thought I use wget and it’s mirroring feature, but I found out, there already is a PEP 381 for mirroring PyPI infrastructure and there already is a client for applying PEP 381.

Because I’ve a dedicated server for hosting mirrors, I choose this server to host my local PyPI mirror, too. It’s pretty easy, just install a virtualenv (optional but cleaner) and install pep381client.

virtualenv --no-site-packages pypi_mirror
source pypi_mirror/bin/activate
pip install pep381client
mkdir pypi_mirror/data/
pep381run pypi_mirror/data/

Now pep381run will start mirroring PyPI. After you completed the first run you can execute pep381run regularly via cron, to sync your local mirror with PyPI. A full mirrored PyPI will consume about 30GB of storage.

For a daily sync create /etc/cron.d/pypi_mirror:

45 5 * * * /path/pep381client/pep381run -q /path/pypi/data

If you like to get an email of all changes done in a sync you could use:

MAILTO=you@address.tld
45 5 * * * /path/pep381client/pep381run /path/pypi/data

For hosting your local PyPI mirror, you could use Nginx or Apache. I choose Nginx and my config looks like this:

server {
    listen 10.1.2.3:80;
    server_name pypi.domain.local;
    access_log /var/log/nginx/pypi.domain.local.access.log;
    location / {
        root /var/local/pypi_mirror/data/web;
        autoindex on;

        allow 10.1.2.0/24;
        allow 127.0.0.1;
        deny all;
    }
}

To use your new mirror you can install packages by telling pip from where to download the packages with the --index-url option.

pip install --index-url=http://pypi.domain.local/simple somepackage

If you like to install always from your own mirror you can create a pip config in your home directory.

~/.pip/pip.conf

[global]
index-url = http://pypi.domain.local/simple

The mirror can only be used for installing packages not for searching, because the search feature in pip is API based, which this method does not support. If you need more features than just a raw mirror for installing packages have a look at chishop or crate.io.

Comentários

Postar um comentário