Error loading PyMySQL python package in AppDaemon

andydmack · July 1, 2018, 4:43pm

Have been using the AppDaemon in hassio (add-on v 1.2, AppDaemon v3.x, Hassio v 72.1) with success for a number of months, and after a reset I am getting errors loading PyMySQL. Appears to be while compiling a cryptography dependency. Note that I also load SQLAlchemy, and it appears to load fine during startup of the add-on.

Any suggestions?

…
reading manifest file ‘src/cryptography.egg-info/SOURCES.txt’
reading manifest template ‘MANIFEST.in’
no previously-included directories found matching ‘docs/_build’
warning: no previously-included files matching ‘*’ found under directory ‘vectors’
writing manifest file ‘src/cryptography.egg-info/SOURCES.txt’
running build_ext
generating cffi module ‘build/temp.linux-armv7l-3.6/_padding.c’
creating build/temp.linux-armv7l-3.6
generating cffi module ‘build/temp.linux-armv7l-3.6/_constant_time.c’
generating cffi module ‘build/temp.linux-armv7l-3.6/_openssl.c’
building ‘_openssl’ extension
creating build/temp.linux-armv7l-3.6/build
creating build/temp.linux-armv7l-3.6/build/temp.linux-armv7l-3.6
gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/include/python3.6m -c build/temp.linux-armv7l-3.6/_openssl.c -o build/temp.linux-armv7l-3.6/build/temp.linux-armv7l-3.6/_openssl.o -Wconversion -Wno-error=sign-conversion
unable to execute ‘gcc’: No such file or directory
error: command ‘gcc’ failed with exit status 1

----------------------------------------

Command “/usr/bin/python3.6 -u -c “import setuptools, tokenize;file=’/tmp/pip-build-9sja1403/cryptography/setup.py’;f=getattr(tokenize, ‘open’, open)(file);code=f.read().replace(’\r\n’, ‘\n’);f.close();exec(compile(code, file, ‘exec’))” install --record /tmp/pip-m8142xpc-record/install-record.txt --single-version-externally-managed --compile” failed with error code 1 in /tmp/pip-build-9sja1403/cryptography/
You are using pip version 9.0.1, however version 10.0.1 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.
FATAL: Failed installing package PyMySQL
[cont-init.d] 81-python-packages.sh: exited 1.
[cont-finish.d] executing container finish scripts…
[cont-finish.d] 50-compiled-symlink.sh: executing…
[cont-finish.d] 50-compiled-symlink.sh: exited 0.
[cont-finish.d] 99-message.sh: executing…

            Oops! Something went wrong.

We are so sorry, but something went terribly wrong when
starting or running this add-on.

ReneTode · July 1, 2018, 10:10pm

you are trying to do very advanced stuff on a plaform that is developed for users that dont want to do advanced stuff.

it seems that the hassio doesnt have the needed pip version installed, but i dont know if it is possible at all to upgrade that on hassio.

if you were on linux (on what AD is build and has the biggest support) i would say update pi and then install pymysql.

now i only can say: go to the hassio part from the forum and find out if is possible to upgrade pip inside an addon.

andydmack · July 8, 2018, 6:54pm

Thanks but I was able to figure it out. For anyone else in a similar boat, I switched from PyMySQL and SQLAlchemy to mysql.connector and after some code changes (due to the new package), everything is fine now.

For the record, with AppDaemon you don’t need to use pip (you only need to include the packages as part of your hassio plugin configuration). Adds a lot of flexibility to hassio, while providing all of the benefits. In this case, I created a simple script that does a google image search based on the media being played on my TV, and displays it in the frontend.

ReneTode · July 8, 2018, 7:04pm

with “normal” appdaemon you need pip.
with hassio you add it to the addon.
but there are also things that you dont use pip for that you can install and use with AD, that you cant use with hassio. and not everything that you can pip also can be used in hassio (dont know why, but i had some of those things come by)

PyMySQL and SQLAlchemy are among those things that others tried to use with an hassio addon, that they didnt seem to get to work. (and people without hassio use that)

but why did you need mysql for a google search? that can be done with default libs.

andydmack · July 9, 2018, 3:07am

Fair enough - I only started using AppDaemon after using hassio. And still very much a rookie, but enjoying the challenge

As for PyMySQL, it’s for a different use case. I have a separate script that loads TV listings into a MySQL database and then use AppDaemon to query info about the show I’m watching (pulling the current channel from my Tivo) and also a list of other shows on now (so I can change the channel via Alexa or the HA frontend). This is all tied together with AppDaemon and needs the SQL connector. The Google search now pulls images based on what’s being watched.

Was able to do a lot with the built in libraries, but hit a wall and turned to AppDaemon.

ReneTode · July 9, 2018, 8:28am

did you already share that app?
i think others will be interested in it too.

i know i am.

andydmack · July 15, 2018, 10:08pm

I haven’t shared it yet, but happy to if there is interest.

I’m not sure the best way to share the code (posting on Github?). But in the meantime, here are the key components and some images:

TV Listings database - I use tv_grab_sd_json (part of XMLTV tools) to load data from http://www.schedulesdirect.org into a MySQL database I host on another server. I filter out only the channels I watch and keep the show title, description and timing information
Using a tcp sensor, I leverage the telnet interface exposed by my TiVo receiver to store the current channel being watched (or alternatively confirm if a recorded show is being watched instead)
I then use an AppDaemon script that reads the tcp sensor and queries the database to confirm the show details. This data is then stored in another sensor value in HA. Note, if I am using my PS4 or Plex - then I display details on the game or movie being watched instead.
In the same AppDaemon script, I also do an image lookup and use a Camera component to display the image (which could be a Tivo show, PS4 game, Plex movie, etc)
Finally, I created a Custom UI component which reads this sensor and cleanly displays it in a panel (see image)

I run another AppDaemon script to query whats on all my favourite channels and use a similar approach to display this info in a table (with channel logos - see below)

Each of the logos in the table are linked to a HA script that sends a telnet command to my Tivo to change the channel.

I am currently trying to create a custom Alexa intent, so I can ask to tune a particular show (eg Seinfeld) and then have it change the channel directly.

Happy to answer any questions about the approach and would love any suggestions on how to package and make available to others.

ReneTode · July 15, 2018, 10:50pm

great work.
i would share it on github with an explenation how others could install and use it.
make sure that all the files are clean from passwords

gremblin · August 15, 2018, 7:53pm

Hi @andydmack, I’m having an issue installing ‘imageio’ in AppDaemon. I’m getting a similar error that you had with PyMySQL. I added ‘imageio’ to the ‘python_packages’ list in the AppDaemon config and it fails out. Any ideas?

[cont-init.d] 81-python-packages.sh: executing…
Collecting imageio
Downloading https://files.pythonhosted.org/packages/a7/1d/33c8686072148b3b0fcc12a2e0857dd8316b8ae20a0fa66c8d6a6d01c05c/imageio-2.3.0-py2.py3-none-any.whl (3.3MB)
Collecting numpy (from imageio)
Downloading https://files.pythonhosted.org/packages/3a/20/c81632328b1a4e1db65f45c0a1350a9c5341fd4bbb8ea66cdd98da56fe2e/numpy-1.15.0.zip (4.5MB)
Collecting pillow (from imageio)
Downloading https://files.pythonhosted.org/packages/d3/c4/b45b9c0d549f482dd072055e2d3ced88f3b977f7b87c7a990228b20e7da1/Pillow-5.2.0.tar.gz (14.5MB)
Installing collected packages: numpy, pillow, imageio
Running setup.py install for numpy: started

…

Could not locate executable ifort
Could not locate executable ifc
customize LaheyFCompiler
Could not locate executable lf95
customize PGroupFCompiler
Could not locate executable pgfortran
customize AbsoftFCompiler
Could not locate executable f90
Could not locate executable f77
customize NAGFCompiler
customize VastFCompiler
customize CompaqFCompiler
Could not locate executable fort
customize IntelItaniumFCompiler
Could not locate executable efort
Could not locate executable efc
customize IntelEM64TFCompiler
customize GnuFCompiler
Could not locate executable g77
customize G95FCompiler
Could not locate executable g95
customize PathScaleFCompiler
Could not locate executable pathf95
customize NAGFORCompiler
Could not locate executable nagfor
don’t know how to compile Fortran code on platform ‘posix’
C compiler: gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC
compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/usr/include/python3.6m -c'
gcc: _configtest.c
/bin/sh: gcc: not found
/bin/sh: gcc: not found
failure.
removing: _configtest.c _configtest.o
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-build-n_rida6t/numpy/setup.py", line 410, in <module>
    setup_package()
  File "/tmp/pip-build-n_rida6t/numpy/setup.py", line 402, in setup_package
    setup(**metadata)
  File "/tmp/pip-build-n_rida6t/numpy/numpy/distutils/core.py", line 169, in setup
    return old_setup(**new_attr)
  File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/tmp/pip-build-n_rida6t/numpy/numpy/distutils/command/install.py", line 62, in run
    r = self.setuptools_run()
  File "/tmp/pip-build-n_rida6t/numpy/numpy/distutils/command/install.py", line 36, in setuptools_run
    return distutils_install.run(self)
  File "/usr/lib/python3.6/distutils/command/install.py", line 545, in run
    self.run_command('build')
  File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/tmp/pip-build-n_rida6t/numpy/numpy/distutils/command/build.py", line 47, in run
    old_build.run(self)
  File "/usr/lib/python3.6/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/tmp/pip-build-n_rida6t/numpy/numpy/distutils/command/build_src.py", line 148, in run
    self.build_sources()
  File "/tmp/pip-build-n_rida6t/numpy/numpy/distutils/command/build_src.py", line 159, in build_sources
    self.build_library_sources(*libname_info)
  File "/tmp/pip-build-n_rida6t/numpy/numpy/distutils/command/build_src.py", line 294, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "/tmp/pip-build-n_rida6t/numpy/numpy/distutils/command/build_src.py", line 377, in generate_sources
    source = func(extension, build_dir)
  File "numpy/core/setup.py", line 666, in get_mathlib_info
    raise RuntimeError("Broken toolchain: cannot link a simple C program")
RuntimeError: Broken toolchain: cannot link a simple C program

----------------------------------------
Command “/usr/bin/python3.6 -u -c “import setuptools, tokenize;file=‘/tmp/pip-build-n_rida6t/numpy/setup.py’;f=getattr(tokenize, ‘open’, open)(file);code=f.read().replace(‘\r\n’, ‘\n’);f.close();exec(compile(code, file, ‘exec’))” install --record /tmp/pip-w4j2dhmt-record/install-record.txt --single-version-externally-managed --compile” failed with error code 1 in /tmp/pip-build-n_rida6t/numpy/
You are using pip version 9.0.1, however version 18.0 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.
FATAL: Failed installing package imageio
[cont-init.d] 81-python-packages.sh: exited 1.
[cont-finish.d] executing container finish scripts…
[cont-finish.d] 50-compiled-symlink.sh: executing…
[cont-finish.d] 50-compiled-symlink.sh: exited 0.
[cont-finish.d] 99-message.sh: executing…
            Oops! Something went wrong.
We are so sorry, but something went terribly wrong when
starting or running this add-on.

Be sure to check the log above, line by line, for hints.

[cont-finish.d] 99-message.sh: exited 0.
[cont-finish.d] done.
[s6-finish] syncing disks.
[s6-finish] sending all processes the TERM signal.

andydmack · August 17, 2018, 11:45am

It definitely looks like the same errors I was seeing, but unfortunately I wasn’t able to figure out how to resolve it and found another package that was able to work for me (and changed my code to accommodate).