Dashboard 'robots.txt' not found

Should i stop AD3 first?

i dont think its neccesary, but i would, to make sure.

When i deleted the files from /resin-data/addons/data/a0d7b954_appdaemon3/compiled they where also deleted from /mnt/data/resin-data/addons/data/a0d7b954_appdaemon3/compiled, so one path must be a mirror of the other (i think).

And now… magic!

2018-03-13 20:02:53.705226 INFO HADashboard: New dashboard connected: HA Main Panel
2018-03-13 20:03:00.877722 INFO HADashboard: New dashboard connected: Lights
2018-03-13 20:06:13.025573 INFO Loading custom skin 'all_black'
2018-03-13 20:06:13.080797 INFO Compiling dashboard 'PanelMain'
2018-03-13 20:06:16.533068 INFO Loading custom widget 'baseweather'
2018-03-13 20:06:16.538860 INFO Loading custom widget 'basenew_weather'
2018-03-13 20:06:16.544408 INFO Loading custom widget 'new_weather.yaml'
2018-03-13 20:06:16.545756 INFO Loading custom widget 'weather.yaml'
2018-03-13 20:06:18.246833 INFO function [get_dashboard] finished in 5222 ms
2018-03-13 20:06:18.430501 INFO HADashboard: Dashboard disconnected
2018-03-13 20:06:21.265703 INFO HADashboard: New dashboard connected: HA Main Panel
2018-03-13 20:07:00.466494 INFO Loading custom skin 'all_black'
2018-03-13 20:07:00.533275 INFO Compiling dashboard 'PanelLights'
2018-03-13 20:07:05.656589 INFO Loading custom widget 'baseweather'
2018-03-13 20:07:05.659691 INFO Loading custom widget 'basenew_weather'
2018-03-13 20:07:05.662447 INFO Loading custom widget 'new_weather.yaml'
2018-03-13 20:07:05.663602 INFO Loading custom widget 'weather.yaml'
2018-03-13 20:07:07.402448 INFO function [get_dashboard] finished in 6936 ms
2018-03-13 20:07:07.420414 INFO HADashboard: Dashboard disconnected
2018-03-13 20:07:08.647188 INFO HADashboard: New dashboard connected: Lights

No errors on the log! But i’m not sure that deleting the files was the solution.

When i viewed HA logs, the error stopped happening at 2018-03-13 13:02:09863291. By that time i know for sure that the system rebooted but i don’t remember why (i’ve upgraded to 0.65.3 but don’t remember the time and changed a widget inside one of the dashboard pages).

Could it be something with HA version 0.65 that was solved on 0.65.4 (I don’t know on what version the problem started but was 0.65.0 when i saw it for the first time…)?

Could it be a problem inside one of the pages of the dashboard? The error happened with all pages…

I’ll keep an eye on the log to see if it happens again.

Thank you very much for your help!

Spoke too soon…

2018-03-13 20:17:08.709649 INFO Loading custom skin 'all_black'
2018-03-13 20:17:08.922307 INFO function [get_dashboard] finished in 213 ms
2018-03-13 20:17:08.935488 INFO Compiling dashboard 'robots.txt'
2018-03-13 20:17:08.939491 INFO HADashboard: Dashboard disconnected
2018-03-13 20:17:10.488944 WARNING Dashboard 'robots.txt' not found
2018-03-13 20:17:10.491457 WARNING ------------------------------------------------------------
2018-03-13 20:17:10.492916 WARNING Unexpected error during DASH creation
2018-03-13 20:17:10.493971 WARNING ------------------------------------------------------------
2018-03-13 20:17:10.506242 WARNING Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/appdaemon/dashboard.py", line 822, in get_dashboard
    with open(include_path, "r") as include_file:
FileNotFoundError: [Errno 2] No such file or directory: '/config/appdaemon/compiled/html/default/robots.txt_head.html'

2018-03-13 20:17:10.507424 WARNING ------------------------------------------------------------
2018-03-13 20:17:10.508885 INFO function [get_dashboard] finished in 1761 ms
2018-03-13 20:17:11.058616 INFO HADashboard: New dashboard connected: HA Main Panel

This is strange. I use a custom skin (same as default but with a black background) and it is called in the URL http://192.168.1.200:5050/PanelMain?skin=all_black .

When i first loaded a page the error appeared but i’ve noticed that the error was coming from an missing file or directory called /config/appdaemon/compiled/html/default/robots.txt_head.html that i think is supposed to be from the default skin (I’m i right @ReneTode?).

Then removed the call for the skin on the URL http://192.168.1.200:5050/PanelMain to load the default skin. After that loaded the custom skin again with http://192.168.1.200:5050/PanelMain?skin=all_black and no error again…

Right after AD3 started and dashboard page opened for the first time:

root@hassio:/resin-data/addons/data/a0d7b954_appdaemon3/compiled/html# ls -la
total 12
drwxr-xr-x 3 root root 4096 Mar 13 20:06 .
drwxr-xr-x 5 root root 4096 Mar  8 12:02 ..
drwxr-xr-x 2 root root 4096 Mar 13 20:07 all_black

After loading the URL without calling the custom skin:

root@hassio:/resin-data/addons/data/a0d7b954_appdaemon3/compiled/html# ls -la
total 16
drwxr-xr-x 4 root root 4096 Mar 13 20:28 .
drwxr-xr-x 5 root root 4096 Mar  8 12:02 ..
drwxr-xr-x 2 root root 4096 Mar 13 20:29 all_black
drwxr-xr-x 2 root root 4096 Mar 13 20:28 default

this file is created by HDdashboard because it finds a file robots.txt in your dashboardpath and its loaded somewhere with the default skin.

do you have a dashboard in your frontend?
i saw you also use grafana, and that also uses the robots.txt
something you are calling in your browser seems to call for that file

I don’t have any file with that name anywhere!

I know Grafana uses it because it appears on the search for robots.txt.

What puzzles me now is why the error appeared when loading a custom skin, saying it was trying to load a file from the default skin… And when the default skin file and folder structure was created the errors disappeared…

i know, but if you use something from grafana in your homeassistant gui, it could be that something like grafana tries to open something like that file.

something like that could happen if grafana (or something else) also points to the same port (as hadashboard) on your device.

I use Grafana by itself, no graphs inside HA.

I can check the ports but I don’t think so.

On HA the file you show with lrwxrwxrwx is a symlink and points to the file named on the right of the ->
in your case /data/compiled

I had a quick read on docker to see what its all about. Seems its similar to a obsd jail so it will have a false root somewhere. Its been 10 years since I retired so I’m a bit rusty on this.

To to find out where things are in the base OS try this.

ssh to the HA root and run

touch qwerty

This will create an empty file called ‘qwerty’ in the container root
You should be able to cd to the /config/appdaemon/compiled directory within the docker.

cd /config/appdaemon/compiled
touch asdfg

then ssh to the OS root and search for the file without sending errors to the bit bucket

root@hassio:/# find . -name qwerty 

As you are in the system root you can use either the . (here) or the /(root) in the command. You shouldn’t see any permission errors but its good practice to check.
The result will be the false root of the docker container. Make a note of it and do the same for the file asdfg
to get the full path to the /data/compiled linked directory

Having the start points will help in working out where things are in the real world.
When you have the locations you can delete the 2 files (rm filename) as their work is done.

You’ve no doubt worked out we are in different timezones

1 Like

we already located the compiled dir now.
but thx for the pointers.

the mystery stays where the files inside the compiled dir come from.
AD generates them, but based on what?
normaly only based on files that are in the dashboard dir.
the error says it cant find the file, but why is it asking for that file?

it seems like i was starting at the wrong foot.
something is calling for the url http://dashboardserver/robots.txt

What do you mean by that?

somewhere something tries to load a dashboard called robots.txt

so a program or browser that you have active tries to load this url

http://theAddressFromYourDashboardServer/robots.txt

I just use HADashboard from a wall mounted tablet running Fully Kiosk Browser, Only open on the browser to make some quick tests.

But… Fully Kiosk Browser has a remote admin feature that is accessed by the URL http://192.168.1.120:2323 and a REST interface to send commands like http://192.168.1.120:2323/?cmd=getCamshot&password=[pass]

Can this be the problem?

Chrome dev tools shows that whe the remote admin page is open, this files are requested:

jquery.min.js
fully-header-web-90.png
fully-loading.png
inject.js
resize-tooltip.html
lato-regular.ttf
lato-bold.ttf

No errors on the console…

the remote admin from fully has another address. so nothing there.

you need to look inside fully, or inside your homeassistant screen.

You’re right. Sorry, bad night sleep…

I’ll contact Fully’s developer to ask him about this.

Opened all HA pages with dev tools and nothing calls for robots.txt.

it can be inside a page.
like inside a dashboard a cam url can be called.

so you need to look at anything you open that is remotely connected to dashboard
open a page and look at you errorlog
no error then open something else.
untill you know what action from you triggers the error.

I’ll investigate that. One of my dashboard pages have three cams that stopped working when i started to use encryption and it’s giving a 404 (Not Found) error on all three of them.

But since yesterday the errors stopped…

without errors, there is nothing to find :wink:

using encryption inside HA and/or dashboard gives problems showing cams.
the best way is to use a proxyserver to connect from outside to your home.
like this:

The problem with the cameras was that i was using the link from outside the get the feed. Changed to my internal IP, got another robots.txt error on the first access to the cameras dash page and then no more errors.