0 votes
The machine I'm working on has no internet connection and I have downloaded and installed all the packages mentioned below on the system by executing this command (for example pyzmq package):

pip install pyzmq-17.1.2-cp27-cp27mu-manylinux1_x86_64.whl -f ./ --no-index

 

However, code-env is unable to read this and throws this error:

 

*********************************************************
*********************************************************
*
* create empty env
*
* command : /var/opt/dss/bin/python /opt/dss/dataiku-dss-5.1.0/scripts/virtualenv/virtualenv.py -p python2.7 /var/opt/dss/code-envs/python/test2
*
*********************************************************
*********************************************************
New python executable in /var/opt/dss/code-envs/python/test2/bin/python2.7
Also creating executable in /var/opt/dss/code-envs/python/test2/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /bin/python2.7
*********************************************************
*********************************************************
*
* list packages
*
* command : /var/opt/dss/code-envs/python/test2/bin/python /var/opt/dss/code-envs/python/test2/bin/pip freeze -l
*
*********************************************************
*********************************************************
Installing from Pip requirements:
pandas==0.23.4
python-dateutil==2.7.4
pytz==2018.6
requests==2.20.1
decorator==4.3.0
ipykernel==4.8.2
ipython_genutils==0.2.0
jupyter_client==5.2.3
jupyter_core==4.4.0
pexpect==4.6.0
pickleshare==0.7.5
ptyprocess==0.6.0
pyzmq==17.1.2
simplegeneric==0.8.1
tornado==5.1.1
traitlets==4.3.2
*********************************************************
*********************************************************
*
* install packages
*
* command : /var/opt/dss/code-envs/python/test2/bin/python /var/opt/dss/code-envs/python/test2/bin/pip install --upgrade -r /var/opt/dss/tmp/pip-requirements-install/req357965962070314402.txt
*
*********************************************************
*********************************************************
Collecting pandas==0.23.4 (from -r /var/opt/dss/tmp/pip-requirements-install/req357965962070314402.txt (line 2))
  Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fea5df80dd0>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/pandas/
  Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fea5df80f50>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/pandas/
  Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fea5df23050>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/pandas/
  Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fea5df23350>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/pandas/
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fea5df234d0>: Failed to establish a new connection: [Errno 101] Network is unreachable',)': /simple/pandas/
  Could not find a version that satisfies the requirement pandas==0.23.4 (from -r /var/opt/dss/tmp/pip-requirements-install/req357965962070314402.txt (line 2)) (from versions: )
No matching distribution found for pandas==0.23.4 (from -r /var/opt/dss/tmp/pip-requirements-install/req357965962070314402.txt (line 2))
by

1 Answer

+1 vote
Hi,

Code envs will use "pip install -r requirements.txt" with a requirements.txt made of the packages you requested.

By design, code envs will never try to reuse packages that are already installed at the system level (what you did with your wheel install), since the goal of code envs in DSS is to provide isolated environments.

You can add custom options to the pip command line by going into the code env settings, unchecking "use global settings", and adding command line options. From the doc of pip, you should be able to install from your wheel by adding the following options:

--no-index
--find-links=/path/to/your/wheels

(see https://pip.pypa.io/en/stable/user_guide/#id14)
by
Got it, I had to download and place each dependency, and it worked. But it throws me another error post the pip phase:

Collecting ipykernel==4.8.2 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8))
Collecting ipython_genutils==0.2.0 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 9))
Collecting jupyter_client==5.2.3 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 10))
Collecting jupyter_core==4.4.0 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 11))
Collecting pexpect==4.6.0 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 12))
Collecting pickleshare==0.7.5 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 13))
Collecting ptyprocess==0.6.0 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 14))
Collecting pyzmq==17.1.2 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 15))
Collecting simplegeneric==0.8.1 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 16))
Collecting tornado==5.1.1 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 17))
Collecting traitlets==4.3.2 (from -r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 18))
Requirement already satisfied, skipping upgrade: numpy>=1.9.0 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from pandas==0.23.4->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 2)) (1.14.2)
Requirement already satisfied, skipping upgrade: six>=1.5 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from python-dateutil==2.7.4->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 3)) (1.11.0)
Requirement already satisfied, skipping upgrade: idna<2.8,>=2.5 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from requests==2.20.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 5)) (2.7)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from requests==2.20.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 5)) (3.0.4)
Requirement already satisfied, skipping upgrade: urllib3<1.25,>=1.21.1 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from requests==2.20.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 5)) (1.24.1)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from requests==2.20.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 5)) (2018.11.29)
Collecting ipython>=4.0.0 (from ipykernel==4.8.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8))
Collecting pathlib2; python_version in "2.6 2.7 3.2 3.3" (from pickleshare==0.7.5->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 13))
Collecting futures (from tornado==5.1.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 17))
Collecting singledispatch (from tornado==5.1.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 17))
Collecting backports-abc>=0.4 (from tornado==5.1.1->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 17))
Collecting enum34; python_version == "2.7" (from traitlets==4.3.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 18))
Collecting backports.shutil-get-terminal-size; python_version == "2.7" (from ipython>=4.0.0->ipykernel==4.8.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8))
Collecting pygments (from ipython>=4.0.0->ipykernel==4.8.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8))
Requirement already satisfied, skipping upgrade: setuptools>=18.5 in /var/opt/dss/code-envs/python/test2/lib/python2.7/site-packages (from ipython>=4.0.0->ipykernel==4.8.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8)) (40.6.3)
Collecting prompt-toolkit<2.0.0,>=1.0.4 (from ipython>=4.0.0->ipykernel==4.8.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8))
Collecting scandir; python_version < "3.5" (from pathlib2; python_version in "2.6 2.7 3.2 3.3"->pickleshare==0.7.5->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 13))
Collecting wcwidth (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=4.0.0->ipykernel==4.8.2->-r /var/opt/dss/tmp/pip-requirements-install/req7162189643356614981.txt (line 8))
Building wheels for collected packages: scandir
  Running setup.py bdist_wheel for scandir: started
  Running setup.py bdist_wheel for scandir: finished with status 'error'
  Complete output from command /var/opt/dss/code-envs/python/test2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jSsFWK/scandir/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-u4MHKm --python-tag cp27:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-2.7
  copying scandir.py -> build/lib.linux-x86_64-2.7
  running build_ext
  building '_scandir' extension
  creating build/temp.linux-x86_64-2.7
  gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c _scandir.c -o build/temp.linux-x86_64-2.7/_scandir.o
  _scandir.c:14:20: fatal error: Python.h: No such file or directory
   #include <Python.h>
                      ^
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  
  ----------------------------------------
  Running setup.py clean for scandir
Failed to build scandir
Installing collected packages: scandir, pathlib2, backports.shutil-get-terminal-size, simplegeneric, pygments, ptyprocess, pexpect, enum34, ipython-genutils, traitlets, wcwidth, prompt-toolkit, pickleshare, ipython, futures, singledispatch, backports-abc, tornado, pyzmq, jupyter-core, jupyter-client, ipykernel
  Running setup.py install for scandir: started
    Running setup.py install for scandir: finished with status 'error'
    Complete output from command /var/opt/dss/code-envs/python/test2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jSsFWK/scandir/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-ry4fxI/install-record.txt --single-version-externally-managed --compile --install-headers /var/opt/dss/code-envs/python/test2/include/site/python2.7/scandir:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    copying scandir.py -> build/lib.linux-x86_64-2.7
    running build_ext
    building '_scandir' extension
    creating build/temp.linux-x86_64-2.7
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c _scandir.c -o build/temp.linux-x86_64-2.7/_scandir.o
    _scandir.c:14:20: fatal error: Python.h: No such file or directory
     #include <Python.h>
                        ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    
    ----------------------------------------
  Failed building wheel for scandir
notebook 5.4.0-dku-5.1-1 requires jinja2, which is not installed.
notebook 5.4.0-dku-5.1-1 requires nbconvert, which is not installed.
notebook 5.4.0-dku-5.1-1 requires nbformat, which is not installed.
notebook 5.4.0-dku-5.1-1 requires Send2Trash, which is not installed.
notebook 5.4.0-dku-5.1-1 requires terminado>=0.8.1, which is not installed.
Command "/var/opt/dss/code-envs/python/test2/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-jSsFWK/scandir/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-ry4fxI/install-record.txt --single-version-externally-managed --compile --install-headers /var/opt/dss/code-envs/python/test2/include/site/python2.7/scandir" failed with error code 1 in /tmp/pip-install-jSsFWK/scandir/

Can you please shed some light on this?
The issue was with my python installation for this - I was missing the python-devel package. installed that, and this is working fine now.

Thanks for the help, Clément
1,213 questions
1,246 answers
1,407 comments
11,780 users

©Dataiku 2012-2018 - Privacy Policy