Installation of DSM Integration fails

labertouche · October 9, 2021, 2:11pm

This is what the ‘basic’ log is telling me:

Logger: homeassistant.components.synology_dsm.config_flow
Source: components/synology_dsm/config_flow.py:181
Integration: synology_dsm (documentation, issues)
First occurred: 4:02:45 PM (1 occurrences)
Last logged: 4:02:45 PM

{'api': None, 'code': -1, 'reason': 'Unknown', 'details': 'ConnectionError = <urllib3.connection.HTTPConnection object at 0x7f80ec3880>: Failed to establish a new connection: [Errno -3] Try again'}

This is the what the ‘extensive’ logs tells me:

2021-10-09 16:02:40 DEBUG (SyncWorker_4) [synology_dsm.synology_dsm] Creating new session

The last log is not an error, just confirmation “creating a new session”, but in the DSM logs I see nothing:

That last login attempt is me logging in to open the logfile. The one before is DDNS registration.

Appreciate your help!

DaveTX · October 9, 2021, 3:05pm

I find my DSM integration the roughest one to setup. I’ve repeated this several times and it feels like I’m groping through a canvas curtain at an object I’m not sure what it is….

Mine is working now after having again set it up again last night. My whole cycle was caused by an upgrade to DSM 7.x and the subsequent overload on my DSM system to rebuild facial recognition. The saturation of the CPU and Disk was causing my integration to timeout and I stupidly removed it thinking it was related to DSM 7.x upgrade software. Wasn’t that; it’s just so stinking busy.

So my point is make sure your DSM is fairly quiet - Look at resource monitoring and see.

The only way I’m successful in setting things up is:
Let HA find the integration.
Configure it via GUI
I use my IP address
I use my id/pwd for the special account that has admin privs only and all other access turned off.
I leave port number disabled. I seem to have problems with providing this.
I uncheck all the SSL boxes. (Don’t take this as gospel - it’s my setup and I haven’t spend the time to figure out why SSL isn’t working for this login….)

I have 2FA setup and enter the challenge key.

One debugging tip: Can you log into DSM with your special HA id/pwd? For example if you’ve got 2FA mucked up or you’ve recently moved to DSM 7.x there are delays or challenges that logging in clean via DSM will ensure you’ve sorted out first. Then retry your integration configuration.

armin-gh · October 9, 2021, 4:08pm

After creating the session my log continues to display the URL for the http-Request and the Result from the request:

2021-10-09 17:40:43 DEBUG (SyncWorker_1) [synology_dsm.synology_dsm] Request url: https://192.168.x.x:5001/webapi/query.cgi?api=SYNO.API.Info&version=1&method=query
2021-10-09 17:40:43 DEBUG (SyncWorker_1) [synology_dsm.synology_dsm] Request status_code: 200

I’m not sure what your type of HA installation allows you from SSH, if “wget” is installed you can try

wget --no-check-certificate "https://192.168.x.x:5001/webapi/query.cgi?api=SYNO.API.Info&version=1&method=query" -O syno.out

syno.out contains the output from the request.
This first request to the Synology API seems not to require Authentication, might be the reason why you see nothing on the Synology Log
Authentication is done after the call to SYNO.API.Info was successful, seems you do not get to this point.
2021-10-09 17:40:43 DEBUG (SyncWorker_1) [synology_dsm.synology_dsm] Request url: https://192.168.x.x:5001/webapi/entry.cgi?account=********&passwd=********&enable_device_token=yes&device_name=pi&format=sid&api=SYNO.API.Auth&version=7&method=login

Armin

jeroen.nijssen · October 9, 2021, 7:34pm

Hey, just wanted to add that i have the same problem… it seems like a bug

labertouche · October 11, 2021, 9:42am

Thanks all for helping out and coming with suggestions.

I created a ‘hass’ user (with admin rights) who can log in. In the DSM that works, not in HA.
The DSM integration is not found as a ‘new’ potential integration. It was so when I initially started with HA some weeks ago. Seems no way to force HA to scan for new equipment in the local network.
My DSM is used for storage, hosts a Plex Media Server and runs Surveillance Station. Looking at the logs, it doesn’t it a whole lot of time
Problems where with the latest DSM version (see above) and remain in DSM 7

This morning, I setup 2FA for the HA specific account: nothing.
I changed the IP address; HA does not see the DSM as a new integration.

I repeated all the steps I did (see initial post) and it.just.doesn’t.work.

My gut feeling tells me it has something to do with DDNS on the NAS.

HA comes with the following errors:

Logger: homeassistant.components.synology_dsm.config_flow
Source: components/synology_dsm/config_flow.py:181
Integration: Synology DSM (documentation, issues)
First occurred: 12:17:32 PM (5 occurrences)
Last logged: 12:26:22 PM

{'api': None, 'code': -1, 'reason': 'Unknown', 'details': 'ConnectionError = <urllib3.connection.HTTPConnection object at 0x7f9a2f8640>: Failed to establish a new connection: [Errno -3] Try again'}
{'api': None, 'code': -1, 'reason': 'Unknown', 'details': 'ConnectionError = <urllib3.connection.HTTPConnection object at 0x7f9417e820>: Failed to establish a new connection: [Errno -3] Try again'}
{'api': None, 'code': -1, 'reason': 'Unknown', 'details': 'ConnectionError = <urllib3.connection.HTTPConnection object at 0x7f9217b0a0>: Failed to establish a new connection: [Errno -3] Try again'}
{'api': None, 'code': -1, 'reason': 'Unknown', 'details': 'ConnectionError = <urllib3.connection.HTTPSConnection object at 0x7f93888070>: Failed to establish a new connection: [Errno -3] Try again'}
{'api': None, 'code': -1, 'reason': 'Unknown', 'details': 'ConnectionError = <urllib3.connection.HTTPSConnection object at 0x7f93e6f400>: Failed to establish a new connection: [Errno -3] Try again'}

Exact same thing as before. No connection possible.

armin-gh · October 13, 2021, 4:15pm

Hi
one additional location you might want to check is the logs in
/var/log/nginx
on the synology.

Nginx processes all requests on ports 80/443/5000/5001 on the Synology, perhaps nginx has a reason to refuse inbound requests to the Synology WebAPI.

Regarding DDNS:
Like already said, the integration works for me on multiple home assistant installatins. What I do not have is DDNS on the Synology, perhaps the configuration for the external name blocks something in nginx.
I have DDNS on my Router and access to services on the Synology is only allowed via VPN, with the exception of the VPN-Server of course.

Armin

labertouche · October 14, 2021, 6:09am

I have no /var/log/nginx
Is nginx a requirement?

armin-gh · October 14, 2021, 6:47am

That’s on the Synology, at least with DSM 7 I would expect to have it installed
You might have to enable SSH in the settings on Synology, the system folders are not exposed in File Station
I still think your Home Assistant is ok, but the Synology refuses requests to its WebApi, hence the error

Armin

labertouche · October 14, 2021, 7:03am

When I SSH into my Synology I see 3 files:
error_default.log
error.log
error.log.1.xz

The error.log has the most recent timestamp.
When I do a tail -f /var/log/nginx/error.log and try to install the DSM integration I see an error message on HA (Failed to Connect) but the error.log on Synology is unchanged.

Is there a way to get extensive logging?

You might be on to something…

Thanks,

Alex

armin-gh · October 14, 2021, 5:00pm

Hi
you might be able to switch on the access log for nginx, but I think it won’t tell us much more as the Synology seems to refuse the connection on TCP-Level, the nginx-Server never processes the request.
Any luck to run the “wget” command I wrote about earlier? wget might throw a better error message than the Python-Exception in the integration
Might also be worth to run “wget” from any other computer in your network, most Linux distributions have it installed by default, you can also run from powershell on any windows computer.

wget  "http://192.168.x.x:5000/webapi/query.cgi?api=SYNO.API.Info&version=1&method=query" -O syno.out

produces a file here in JSON format ( have a DS416play) and this is the request failing in the integration.

Armin

labertouche · October 15, 2021, 9:39am

I managed to execute the command. The output is a 69122 character long file.

What am I looking for?

Alex

armin-gh · October 15, 2021, 11:27am

Hi
Means that from where you started the command it can reach the Synology, using the IP/Hostname, port and Protocol as used in the wget command. Additionally ignoring a possible mismatch on the certificate
I’m at work now, will send later how my file retrieved from wget looks like, but the content does not really matter, retrieving ~70k means that the Synology responds
Using the same parameters in the integration it should work too… that much about the theory…

Depending on your type of HA installation the docker environment could be a problem too, that the integration can’t reach to the Synology from inside the docker environment running the HA server process

Armin

labertouche · October 15, 2021, 12:24pm

Hi Armin,

I used the wget command from my Mac in the same network and the syno.out gives me a lot of lines.
You would think the connection to the Synology works just fine right?
File is to large to copy/paste. I agree with your theory. But as mentioned in the initial post: I tried all possible combinations you van possibly think of.

Alex

armin-gh · October 15, 2021, 3:22pm

Hi
looks like this, a file in JSON-Format where Synology provides information about the functions available via http-Requests

Size likely depends on the packages installed in Synology, mine is around 90k

This is the documentation:

Synology accepts the connection from the Mac, leaves two options…

Synology blocks the requests coming from HA
HA does not get to the Synology through the network

We can try the following to make HA to send the same http-Request
Add the following to configuration.yaml:

notify:
  - name: SynologyInfo
    platform: rest
    resource: https://192.168.0.99:5001/webapi/query.cgi?api=SYNO.API.Info&version=1&method=query
    verify_ssl: false

This adds the same request to HA as a notification, might require to restart HA
Then in HA go to “Developer Tools”/Services

in the “Services” field you should see this notification service

Select it and enter some random text in the message (it insists on having something there…)

the press the “Call service”-Button
if it goes through and retrieves a response from Synology, the button just turns green and then blue again, I have not yet managed to show the response.
when I use an incorrect IP, the button turns red and an Error is logged

My hope is that this has more details than the error message from the Synology integration

we might also see just the same:

but at least we would know if this is a problem with the Synology integration or if there is a problem in reaching the Synology-Server from HA in general.

hope I did not add more confusion with my lengthy posts…
Armin

labertouche · October 18, 2021, 7:32am

Hi Armin,

Not lengthy at all but very helpful!
When calling the service, the button does turn green for a few seconds.
The logs show NO Synology errors.
So you would think the connection is successful right?

When I use the wrong IP address I get two error messages in the logfile:

Logger: homeassistant.helpers.script.websocket_api_script
Source: components/rest/notify.py:196
First occurred: 9:45:42 AM (1 occurrences)
Last logged: 9:45:42 AM

websocket_api script: Error executing script. Unexpected error for call_service at pos 1: HTTPSConnectionPool(host='192.168.1.215', port=5001): Max retries exceeded with url: /webapi/query.cgi?api=SYNO.API.Info&version=1&method=query&message=hello (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f75d09910>: Failed to establish a new connection: [Errno 113] Host is unreachable'))
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
OSError: [Errno 113] Host is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f75d09910>: Failed to establish a new connection: [Errno 113] Host is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.1.215', port=5001): Max retries exceeded with url: /webapi/query.cgi?api=SYNO.API.Info&version=1&method=query&message=hello (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f75d09910>: Failed to establish a new connection: [Errno 113] Host is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 371, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 571, in _async_call_service_step
    await service_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 1491, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1526, in _execute_service
    await handler.job.target(service_call)
  File "/usr/src/homeassistant/homeassistant/components/notify/__init__.py", line 174, in _async_notify_message_service
    await self.async_send_message(**kwargs)
  File "/usr/src/homeassistant/homeassistant/components/notify/__init__.py", line 149, in async_send_message
    await self.hass.async_add_executor_job(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/rest/notify.py", line 196, in send_message
    response = requests.get(
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='192.168.1.215', port=5001): Max retries exceeded with url: /webapi/query.cgi?api=SYNO.API.Info&version=1&method=query&message=hello (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f75d09910>: Failed to establish a new connection: [Errno 113] Host is unreachable'))

and

Logger: homeassistant.components.websocket_api.http.connection
Source: components/rest/notify.py:196
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 9:45:42 AM (1 occurrences)
Last logged: 9:45:42 AM

[547166043728] Error handling message: Unknown error
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
OSError: [Errno 113] Host is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f75d09910>: Failed to establish a new connection: [Errno 113] Host is unreachable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='192.168.1.215', port=5001): Max retries exceeded with url: /webapi/query.cgi?api=SYNO.API.Info&version=1&method=query&message=hello (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f75d09910>: Failed to establish a new connection: [Errno 113] Host is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 26, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 525, in handle_execute_script
    await script_obj.async_run(msg.get("variables"), context=context)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1219, in async_run
    await asyncio.shield(run.async_run())
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 353, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 371, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 571, in _async_call_service_step
    await service_task
  File "/usr/src/homeassistant/homeassistant/core.py", line 1491, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1526, in _execute_service
    await handler.job.target(service_call)
  File "/usr/src/homeassistant/homeassistant/components/notify/__init__.py", line 174, in _async_notify_message_service
    await self.async_send_message(**kwargs)
  File "/usr/src/homeassistant/homeassistant/components/notify/__init__.py", line 149, in async_send_message
    await self.hass.async_add_executor_job(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/rest/notify.py", line 196, in send_message
    response = requests.get(
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='192.168.1.215', port=5001): Max retries exceeded with url: /webapi/query.cgi?api=SYNO.API.Info&version=1&method=query&message=hello (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f75d09910>: Failed to establish a new connection: [Errno 113] Host is unreachable'))

armin-gh · October 18, 2021, 4:32pm

Hi
ok, that means that HA does receive a response when sending the HTTP-Request to the Synology
And that the problem ist somewhere with the Synology integration.

Wondering if some garbage data was left in the internal configuration files of HA when you discarded the integration during initial install or when you tried with the various combinations of parameters.

Can you locate a file “core.config_entries” in a folder “.storage” on HA? The .storage folder is where HA keeps the configurations for devices, entities, Lovelace, …
there sould be exactly one entry with domain “synology_dsm” and the connection data should be what you entered in the GUI ( I masked some of the data below)

            {
                "entry_id": "a66dc3f084e0b093f061399c4236ad89",
                "version": 1,
                "domain": "synology_dsm",
                "title": "DS416",
                "data": {
                    "host": "192.168.0.x",
                    "port": "5001",
                    "ssl": true,
                    "verify_ssl": false,
                    "username": "userid",
                    "password": "secret",
                    "mac": [
                        "00-11-32-aa-bb-cc",
                        "00-11-32-aa-bb-dd"
                    ]
                },
                "options": {},
                "pref_disable_new_entities": false,
                "pref_disable_polling": false,
                "source": "ssdp",
                "unique_id": "uuuuuuuuuuuuu",
                "disabled_by": null
            },

Armin

labertouche · October 19, 2021, 9:38am

Hey!

I checked the “core.config_entries” and see a DSM entry here:

 {
                "entry_id": "cd3dac2599fdad2cb8494284d214efa1",
                "version": 1,
                "domain": "synology_dsm",
                "title": "BLACKBOX (192.x.x.x)",
                "data": {},
                "options": {},
                "pref_disable_new_entities": false,
                "pref_disable_polling": false,
                "source": "ignore",
                "unique_id": "00113271322D",
                "disabled_by": null
            },

It does not have the “data” entries shown in your example:

"host": "192.168.0.x",
                    "port": "5001",
                    "ssl": true,
                    "verify_ssl": false,
                    "username": "userid",
                    "password": "secret",
                    "mac": [
                        "00-11-32-aa-bb-cc",
                        "00-11-32-aa-bb-dd"
                    ]

I removed the complete Synology and rebooted HA.
Then: Tada! New device discovered.

I enter my credentials and went to the next screen: 2FA
Haven’t setup 2FA, but I quickly logged in to DSM using a webbrowser and clicked “I don’t have my phone with me”. Then I used that code to get in.

IT WORKS!

Many many thanks Armin! Very much appreciated!

Now to find out why I get to enter a 2FA code even while that’s disabled in DSM.

armin-gh · October 19, 2021, 3:57pm

Hi,
very good, perhaps that helps the other user reporting a similar issue.
That’s what I wanted to suggest, to remove the whole section, of course after making a copy of the file. I guess the incomplete entry stopped it from discovering the device again.

I was never asked for 2FA and also have it not yet enabled on my Synology.
I believe you wrote earlier you have Quick Connect enabled and also a Synology Account ( for DDNS?), perhaps that triggers something. Did it put the internal name/IP or the external name into this config file?

Armin

labertouche · October 20, 2021, 8:54am

Thanks again for your help Armin, much appreciated!

I never activated 2FA, and don’t use quick connect. I only used the Synology DDNS service, but for internal traffic the internal ip address is used of course. Based on that HA found the integration.

Hope other people with similar problems will find this post!

Kudos Armin!

Alex.

Johnnyyr · November 11, 2021, 7:58pm

I checkt the core.config_entries nothing from Syno.
And i can´t connect. It works fine 6 month and now no connection .
Logs:
Logger: homeassistant.components.synology_dsm.config_flow
Source: components/synology_dsm/config_flow.py:190
Integration: synology_dsm (documentation, issues)
First occurred: 20:25:17 (12 occurrences)
Last logged: 20:53:03

{‘api’: None, ‘code’: -1, ‘reason’: ‘Unknown’, ‘details’: “ReadTimeout = HTTPConnectionPool(host=‘192.168.2.240’, port=5000): Read timed out. (read timeout=30)”}

Any ideas?