For one of my clients I had to convert a couple of pip-managed repositories to Poetry-managed ones. When I got to the last repo, there was one catch: it required Conda as well. This is quite straightforward, with the help of some packages and a couple of config changes.
# Pip v.s. Poetry v.s. Conda
Lets quickly recap the three tools we will discuss today.
# pip
Pip is probably one of the most known package managers for Python. It enables developers to fetch and install Python packages from either the internet or a private repository. Pip has been the standard for a lot of years and will probably being used for many years to come.
# Poetry
In the past few years Poetry quickly gained traction. Build upon pip, it’s a package manager on steroids. With the help of pyproject.toml
files you can define your packages (either individually or in groups), set the configuration for groups of packages, define settings for your favourite linters and tools and define the environment to build your packages. It can (and will so by default) also create your virtual environments.
# Conda
Then there’s also Conda. Conda has a lot of similarities with Poetry. One of the benefits of Conda (and this is also the reason my client uses it) is that repositories like conda-forge contain pre-compiled packages; something that’s not always the case with PyPi (the public package repository that both pip and Poetry use by default).
# Combining Poetry & Conda
When picking up this story, it sounded pretty straightforward. However, one of the issues is that both Poetry and Conda are package managers, as well as environment managers that can conflict with each other quite easily. So how do we set this up?
Since Poetry is quite straightforward to configure to the needs of Conda, we’ll use Conda as the base environment. Then we modify Poetry to use this environment as well.
Lets start with setting up a base environment.yml
to define our Conda setup:
|
|
These packages are just some examples, however there are a few important things to note here:
-
We define a Python version, this should be kept the same as what we will define in our
pyproject.toml
. -
poetry
will be installed as a dependency as well. This ensures that the poetry command (which we will use later) is used in the context of the Conda environment. -
We define platforms which will be used for our conda lock file, which will be generated with the
conda-lock
tool. You can remove the platforms you don’t need.
With this out of the way we can configure a lock file. The lock file will be used to create reproducible environments across installations. We create the lock files with a temporary environment:
|
|
So what’s happening here? We create an environment, using the conda-forge channel, and setup Python 3.11 (the same that we’ve defined in the environment.yml
) and conda-lock. We will then activate the environment, run the conda-lock
command which will pick-up the dependencies from the environment.yml
, resolve them and put their exact versions and hashes into a file called conda-linux-64.lock
. Using this file, we can create reprodusable environments, which is great when working in teams (everyone will get the exact same version of packages when setting up the environment).
After that’s done we deactivate the environment and remove it from the system.
Now it’s time to get the real environment up and running!
|
|
First, we’ll generate the environment from the lock file. We’ll give it a name, specify the conda-forge channel and make use of the lock file we generated earlier to install the required packages. Please note that I’m using the conda-osx-64.lock
file. If you’re on Windows or Linux, you should use one those flavours instead. If you’re just using a single platform, you can remove the others from the environment.yml
file.
Now it’s time to activate the environment and configure Poetry:
|
|
By running poetry config --local virtualenvs.create false
we will get a poetry.toml
that contains some local config that will prevent the creation of virtual environments for just this project. If you’re working in a team it’s important to commit this to your repository as well. When running the init command, make sure the Python version matches with what you’ve used to setup the Conda environment.
We can now use Poetry to install PyPi (or private repository, when configured) packages.
|
|
Since we’ve configured Poetry to not create an environment for this project, it will install the packages in the current environment - the Conda environment in this case.
With the setup above, we’ve added both conda(-forge) packages and PyPi packages. They’re all installed in the same environment, so the interpeter is able to find the packages, regardless of which tool installed them. Now we can create a webapp (an API for example) with Flask (as installed with Poetry) that uses gmsh (that’s installed with Conda).
And that’s it! Now you’ve got a Conda and Poetry hybrid environment. I would recommend to stick with a single solution as much as possible, but sometimes you’re required to combine the two.