Skip to content

Private pypi

As this documentation and the associated code repositories can assess, EcoAct fully embraces Open Source.

Nevertheless, for critical private domain model knowledge, it turns useful to have a private library hub in order to share the domain model between your different applications.

Note

There are numerous cloud services that will offer (well not offer, but you see what we mean 😁) you this library hosting functionality. You can (and should once mature enough) stick with them. But some of the poetry knowledge presented here might still prove useful to you! 😊

We will use pypiserver to do this hosting.

Setting up the private pypi stack

There is no sense in setting up locally pypiserver, we thus just present the production setup.

All relevant information can be found in the docker-compose.pypi.yml file (here).

To start the stack in production, you can make use of the Makefile provided in the repo (make help) to list available commands.

To use it you will need to create two folder in the folder containing docker-compose.pypi.yml

  • pypi_data: this will host the .tar.gz and the .whl files containing your library code.
  • pypi_auth: this will contain the .htpasswd file(s) used for authentication

you will need to create a .pypi.env file and setting

  • pypi_url entry to the DNS address you created in order to reach the pypi web interface.

To use htpasswd needed for pypi authentication, install 

sudo apt-get install apache2-utils
then run in the pypy_auth folder

 htpasswd -sc .htpasswd <USER>

You will be prompted for a <PASSWORD>. Note that poetry upload will not work if you use any non bash interpretable characters in your password.

You can then safely launch 

make pypi-launch

your pypi will be accessible at pypi_url.

Useful resource on the topic (we already mentioned that we like this website, didn't we? 😊).

Uploading a package with poetry

We assume that you have basic knowledge of poetry. If you do not, check out - This Arjan video - The official documentation (for dependancies this reading should not be superfluous)

In the code repository containing your pyproject.toml file, run 

poetry config repositories.<your_package_name> `pypi_url`
poetry config http-basic.<your_package_name> '<USER>' `<PASSWORD>`
This will point your package upload mechanism towards pypi_url. Then once you are satisfied with your code modifications, run 
# automatically bump minor version
poetry version patch
# build new package version
poetry build
# upload <your_package_name> to `pypi_url`
poetry publish --repository <your_package_name>

Installing a package in a docker image.

It turns out to be not so trivial to pass an environment variable to a dockerfile (in order not to commit our '' and <PASSWORD>...).

The best way we found to do this relies on docker-compose build.

in your application docker compose service stack, add the following section 

build:
  context: .
  dockerfile: ./<DOCKERFILE>
  args:
    PIP_EXTRA_INDEX_URL: ${PIP_EXTRA_INDEX_URL}
env_file:
  - ./.pypi.env

You will have to add to your .pypi.env the following variable:

  • PIP_EXTRA_INDEX_URL: takes the form https://<USER>:<PASSWORD>@<pypi_dns>. For example, if you host your private pypi at https://myawesomepypi.com, and you chose user password as user/password (very bad choice 😁), then put PIP_EXTRA_INDEX_URL=https://user:password@myawesomepypi.com

Then in your <DOCKERFILE>:

  • add after your FROM instruction : ARG PIP_EXTRA_INDEX_URL. To learn more on ARG and ENV, go here
  • add the following option to the line doing the pip install: 
--extra-index-url PIP_EXTRA_INDEX_URL

Finally, you can update your requirements.txt file, adding to it <your_package_name> (and do not forget to allow for automatic minor updates, putting ==0.* for instance if your package is in major 0).

You should then build the docker image in the following way

docker compose build