Search indexer is failing to build due to a broken Python dependency

MusicBrainz Development
1

I didn’t know if it was best to leave this as a GitHub issue in musicbrainz-docker or a ticket in Jira. I’m building the server using Docker Compose on an Ubuntu VPS, and there’s an error during the build process that is coming from the backports.functools_lru_cache package. That package is importing a newer version of setuptools_scm (6.0.1)which is not compatible with Python 2.7, and the fix is to downgrade it to 5.0.2. Until the maintainer of that package pulls in my PR and fixes this, the indexer will fail to build.

I have a workaround in the meantime. In the SIR Dockerfile, I added a sed command which replaces the backports.functools_lru_cache package with one that I forked, in which I updated the setuptools_scm dependency. Depending on how long it takes the maintainer of that package to fix the problem, it may be worthwhile for musicbrainz-docker to do something like this as well.

RUN git clone --depth=1 --branch "v${SIR_VERSION}" https://github.com/metabrainz/sir.git /code \
    && cd /code \
    && sed -i 's/backports.functools_lru_cache==1.0.1/git+git:\/\/github.com\/FOSSforlife\/backports.functools_lru_cache@patch-1#egg=backports.functools_lru_cache/g' requirements.txt \
    && pip install -r requirements.txt \
    && rm -f /code/config.ini \
    && touch /etc/consul-template.conf

In addition, if you’d like to leave a thumbs up on my PR to backports.functools_lru_cache, that might encourage them to resolve this issue more quickly.

1 Like
2

That’s odd. sir should pull in backports.functools_lru_cache version 1.0.1 - that version doesn’t have a dependency on setuptools_scm according to its setup.py. Running pip install -r requirements.txt in a python2 virtualenv works just fine and pip list afterwards doesn’t list setuptools_scm as being installed. Running docker build . in the sir repository also works. Do you have more in-depth steps to reproduce the problem?

3

I was just able to replicate it by doing a clean install of musicbrainz-docker on a separate machine
(WSL Ubuntu) and doing docker-compose build indexer.

#6 3.958       File "/tmp/easy_install-JMOagm/hgtools-8.2.1/temp/easy_install-0ELRCu/setuptools_scm-6.0.1/setup.py", line 29, in scm_config
#6 3.958         "Programming Language :: Python :: 2.7",
#6 3.958       File "/tmp/easy_install-JMOagm/hgtools-8.2.1/temp/easy_install-0ELRCu/setuptools_scm-6.0.1/src/setuptools_scm/__init__.py", line 8, in <module>
#6 3.958       File "/tmp/easy_install-JMOagm/hgtools-8.2.1/temp/easy_install-0ELRCu/setuptools_scm-6.0.1/src/setuptools_scm/config.py", line 6, in <module>
#6 3.958       File "/tmp/easy_install-JMOagm/hgtools-8.2.1/temp/easy_install-0ELRCu/setuptools_scm-6.0.1/src/setuptools_scm/utils.py", line 41
#6 3.958         print(*k)
#6 3.958               ^
#6 3.958     SyntaxError: invalid syntax
#6 3.958
#6 3.958     ----------------------------------------
#6 3.963 Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-0aUGA6/backports.functools-lru-cache/
#6 4.108 You are using pip version 19.0.1, however version 20.3.4 is available.
#6 4.108 You should consider upgrading via the 'pip install --upgrade pip' command.
------
ERROR: Service 'indexer' failed to build

Interestingly enough, I can’t replicate it anymore on my local python installation. Maybe it has something to do with the cache? I used this installation to test installing the forked package I made, so maybe it’s somehow using that now.

4

It also just installed fine on Python 2 running on my Manjaro machine :confused: So right now I’m only able to reproduce it in Docker (although yesterday I was getting the error in Python on Ubuntu, but not anymore).

5

No, it has nothing to do with any cache. The images in musicbrainz-docker use setuptools 40. Only in setuptools 42 did setuptools learn about the python_requires. hgtools 8 specifies python_requires correctly (https://github.com/jaraco/hgtools/blob/fa05f06b89ae9751662a7b0183820d96813fbd20/setup.cfg#L22), but setuptools ignores that and incorrectly tries to install an hgtools version that can’t work. The following patch to the Dockerfile fixes that:

diff --git a/build/sir/Dockerfile b/build/sir/Dockerfile
index de1a9b9..7dc4b54 100644
--- a/build/sir/Dockerfile
+++ b/build/sir/Dockerfile
@@ -40,6 +40,7 @@ LABEL org.metabrainz.sir.version="${SIR_VERSION}"
 # hadolint ignore=DL3003
 RUN git clone --depth=1 --branch "v${SIR_VERSION}" https://github.com/metabrainz/sir.git /code \
     && cd /code \
+    && pip install -U setuptools>=42 \
     && pip install -r requirements.txt \
     && rm -f /code/config.ini \
     && touch /etc/consul-template.conf

Alternatively, https://github.com/metabrainz/musicbrainz-docker/pull/187 looks like it fixes this as well :slight_smile:

2 Likes
6

That branch worked for me! :partying_face: Thanks so much for the help.

7

For the record, that branch has been merged and the fix is part of the latest release v-2021-04-05.

2 Likes