HASSBIAN crashes, my pi3 crashes. How to troubleshoot

I experience frequent crashes these last 2 days. Could you help me in troubleshooting, what shall I look for?

Same problem. I started out in /var/log/syslog but no luck thus far. Let me know if you have a better lead.

To me it all started when I upgraded to 0.44.1 (it failed), so then I upgraded again, but it started crashing all the time. So I reverted back to 0.43.2 … but now continues crashing

Again and again,after few hours it crashed again.

I have no clue of what I should do, where to look.

Any help?

What shall I do, just reformat the SD card?

Hi, I just started with Hassbian and HA last week and installed it on a RPI3. Same issues.
I found that the issue (in my case) is related to WiFi. If I setup and access HA trough wifi it crashes quite often. I noted also that ssh sessions get frozen from time to time.
I deleted wpa_supplicant.conf setup and got connected to wired connection. No issues so far after that.
I tried also on a RPI2 with a wifi dongle and apparently there are no issues, but I am struggling to understand the HA configuration stuff now.
Hope this helps.
Adrian

You should type “cat /var/log/messages” and see if there are any crashes or stack traces or dumps.

Now is not crashed. Shall I wait when it crashes, and then launch that command, correct?

Now is not crashed, I will copy/paste the result, althoug the log is too long to paste here, but I notice I have a huge number of the below log

May 17 08:40:57 hassbian rsyslogd-2007: action 'action 17' suspended, next retry is Wed May 17 08:42:27 2017 [try http://www.rsyslog.com/e/2007 ]

then also this

May 17 08:48:17 hassbian kernel: [28506.248767] kworker/1:2     D    0 29919                                                                                                                                                                                                                                                   2 0x00000000
May 17 08:48:18 hassbian kernel: [28506.248784] Workqueue: events_freezable mmc_                                                                                                                                                                                                                                             rescan
May 17 08:48:21 hassbian kernel: [28506.248804] [<80713980>] (__schedule) from [                                                                                                                                                                                                                                             <80713ea4>] (schedule+0x50/0xa8)
May 17 08:48:26 hassbian kernel: [28506.248812] [<80713ea4>] (schedule) from [<8                                                                                                                                                                                                                                             05b6188>] (__mmc_claim_host+0xb8/0x1cc)
May 17 08:48:32 hassbian kernel: [28506.248820] [<805b6188>] (__mmc_claim_host)                                                                                                                                                                                                                                              from [<805b62cc>] (mmc_get_card+0x30/0x34)
May 17 08:48:37 hassbian kernel: [28506.248828] [<805b62cc>] (mmc_get_card) from                                                                                                                                                                                                                                              [<805bed5c>] (mmc_sd_detect+0x2c/0x80)
May 17 08:48:42 hassbian kernel: [28506.248836] [<805bed5c>] (mmc_sd_detect) fro                                                                                                                                                                                                                                             m [<805b8f90>] (mmc_rescan+0x1c8/0x38c)
May 17 08:48:45 hassbian rsyslogd-2007: action 'action 17' suspended, next retry                                                                                                                                                                                                                                              is Wed May 17 08:50:10 2017 [try http://www.rsyslog.com/e/2007 ]
May 17 08:48:47 hassbian kernel: [28506.248847] [<805b8f90>] (mmc_rescan) from [                                                                                                                                                                                                                                             <80136570>] (process_one_work+0x158/0x454)
May 17 08:48:52 hassbian kernel: [28506.248857] [<80136570>] (process_one_work)                                                                                                                                                                                                                                              from [<801368d0>] (worker_thread+0x64/0x568)
May 17 08:48:58 hassbian kernel: [28506.248865] [<801368d0>] (worker_thread) fro                                                                                                                                                                                                                                             m [<8013c7d8>] (kthread+0x10c/0x124)
May 17 08:49:05 hassbian kernel: [28506.248874] [<8013c7d8>] (kthread) from [<80                                                                                                                                                                                                                                             108148>] (ret_from_fork+0x14/0x2c)
May 17 08:50:23 hassbian kernel: [28629.129463] kworker/1:2     D    0 29919                                                                                                                                                                                                                                                   2 0x00000000
May 17 08:50:27 hassbian rsyslogd-2007: action 'action 17' suspended, next retry                                                                                                                                                                                                                                              is Wed May 17 08:51:51 2017 [try http://www.rsyslog.com/e/2007 ]
May 17 08:50:30 hassbian kernel: [28629.129479] Workqueue: events_freezable mmc_                                                                                                                                                                                                                                             rescan
May 17 08:50:38 hassbian kernel: [28629.129500] [<80713980>] (__schedule) from [                                                                                                                                                                                                                                             <80713ea4>] (schedule+0x50/0xa8)
May 17 08:50:44 hassbian kernel: [28629.129508] [<80713ea4>] (schedule) from [<8                                                                                                                                                                                                                                             05b6188>] (__mmc_claim_host+0xb8/0x1cc)
May 17 08:50:52 hassbian kernel: [28629.129516] [<805b6188>] (__mmc_claim_host)                                                                                                                                                                                                                                              from [<805b62cc>] (mmc_get_card+0x30/0x34)
May 17 08:50:54 hassbian kernel: [28629.129523] [<805b62cc>] (mmc_get_card) from                                                                                                                                                                                                                                              [<805bed5c>] (mmc_sd_detect+0x2c/0x80)
May 17 08:51:01 hassbian kernel: [28629.129532] [<805bed5c>] (mmc_sd_detect) fro                                                                                                                                                                                                                                             m [<805b8f90>] (mmc_rescan+0x1c8/0x38c)
May 17 08:51:07 hassbian kernel: [28629.129542] [<805b8f90>] (mmc_rescan) from [                                                                                                                                                                                                                                             <80136570>] (process_one_work+0x158/0x454)
May 17 08:51:14 hassbian kernel: [28629.129551] [<80136570>] (process_one_work)                                                                                                                                                                                                                                              from [<801368d0>] (worker_thread+0x64/0x568)
May 17 08:51:20 hassbian kernel: [28629.129559] [<801368d0>] (worker_thread) fro                                                                                                                                                                                                                                             m [<8013c7d8>] (kthread+0x10c/0x124)
May 17 08:51:24 hassbian kernel: [28629.129568] [<8013c7d8>] (kthread) from [<80                                                                                                                                                                                                                                             108148>] (ret_from_fork+0x14/0x2c)
May 17 08:52:06 hassbian rsyslogd-2007: action 'action 17' suspended, next retry                                                                                                                                                                                                                                              is Wed May 17 08:53:31 2017 [try http://www.rsyslog.com/e/2007 ]
May 17 08:52:21 hassbian kernel: [28752.010186] kworker/1:2     D    0 29919                                                                                                                                                                                                                                                   2 0x00000000
May 17 08:52:29 hassbian kernel: [28752.010202] Workqueue: events_freezable mmc_                                                                                                                                                                                                                                             rescan
May 17 08:52:33 hassbian kernel: [28752.010222] [<80713980>] (__schedule) from [                                                                                                                                                                                                                                             <80713ea4>] (schedule+0x50/0xa8)
May 17 08:52:38 hassbian kernel: [28752.010231] [<80713ea4>] (schedule) from [<8                                                                                                                                                                                                                                             05b6188>] (__mmc_claim_host+0xb8/0x1cc)
May 17 08:52:42 hassbian kernel: [28752.010239] [<805b6188>] (__mmc_claim_host)                                                                                                                                                                                                                                              from [<805b62cc>] (mmc_get_card+0x30/0x34)
May 17 08:52:47 hassbian kernel: [28752.010247] [<805b62cc>] (mmc_get_card) from                                                                                                                                                                                                                                              [<805bed5c>] (mmc_sd_detect+0x2c/0x80)
May 17 08:52:53 hassbian kernel: [28752.010255] [<805bed5c>] (mmc_sd_detect) fro                                                                                                                                                                                                                                             m [<805b8f90>] (mmc_rescan+0x1c8/0x38c)
May 17 08:52:58 hassbian kernel: [28752.010265] [<805b8f90>] (mmc_rescan) from [                                                                                                                                                                                                                                             <80136570>] (process_one_work+0x158/0x454)
May 17 08:53:05 hassbian kernel: [28752.010275] [<80136570>] (process_one_work)                                                                                                                                                                                                                                              from [<801368d0>] (worker_thread+0x64/0x568)
May 17 08:53:11 hassbian kernel: [28752.010283] [<801368d0>] (worker_thread) fro                                                                                                                                                                                                                                             m [<8013c7d8>] (kthread+0x10c/0x124)
May 17 08:53:16 hassbian kernel: [28752.010292] [<8013c7d8>] (kthread) from [<80                                                                                                                                                                                                                                             108148>] (ret_from_fork+0x14/0x2c)
May 17 08:53:16 hassbian kernel: [28822.951331] kthreadd invoked oom-killer: gfp                                                                                                                                                                                                                                             _mask=0x27000c0(GFP_KERNEL_ACCOUNT|__GFP_NOTRACK), nodemask=0, order=1, oom_scor                                                                                                                                                                                                                                             e_adj=0
May 17 08:53:16 hassbian kernel: [28822.951337] kthreadd cpuset=/ mems_allowed=0
May 17 08:53:16 hassbian kernel: [28822.951349] CPU: 1 PID: 2 Comm: kthreadd Not                                                                                                                                                                                                                                              tainted 4.9.24-v7+ #993
May 17 08:53:16 hassbian kernel: [28822.951351] Hardware name: BCM2835
May 17 08:53:16 hassbian kernel: [28822.951372] [<8010fb3c>] (unwind_backtrace)                                                                                                                                                                                                                                              from [<8010c058>] (show_stack+0x20/0x24)
May 17 08:53:16 hassbian kernel: [28822.951381] [<8010c058>] (show_stack) from [                                                                                                                                                                                                                                             <80455200>] (dump_stack+0xd4/0x118)
May 17 08:53:16 hassbian kernel: [28822.951391] [<80455200>] (dump_stack) from [                                                                                                                                                                                                                                             <8026c998>] (dump_header+0x9c/0x1f4)
May 17 08:53:16 hassbian kernel: [28822.951403] [<8026c998>] (dump_header) from                                                                                                                                                                                                                                              [<80210364>] (oom_kill_process+0x3e0/0x4e4)
May 17 08:53:16 hassbian kernel: [28822.951412] [<80210364>] (oom_kill_process)                                                                                                                                                                                                                                              from [<802107cc>] (out_of_memory+0x124/0x334)
May 17 08:53:16 hassbian kernel: [28822.951422] [<802107cc>] (out_of_memory) fro                                                                                                                                                                                                                                             m [<802158f0>] (__alloc_pages_nodemask+0xcf4/0xdd0)
May 17 08:53:16 hassbian kernel: [28822.951433] [<802158f0>] (__alloc_pages_node                                                                                                                                                                                                                                             mask) from [<8011aaf4>] (copy_process.part.5+0xec/0x1770)
May 17 08:53:16 hassbian kernel: [28822.951442] [<8011aaf4>] (copy_process.part.                                                                                                                                                                                                                                             5) from [<8011c308>] (_do_fork+0xc8/0x408)
May 17 08:53:16 hassbian kernel: [28822.951451] [<8011c308>] (_do_fork) from [<8                                                                                                                                                                                                                                             011c6b8>] (kernel_thread+0x40/0x48)
May 17 08:53:16 hassbian kernel: [28822.951460] [<8011c6b8>] (kernel_thread) fro                                                                                                                                                                                                                                             m [<8013d828>] (kthreadd+0x1e0/0x268)
May 17 08:53:16 hassbian kernel: [28822.951469] [<8013d828>] (kthreadd) from [<8                                                                                                                                                                                                                                             0108148>] (ret_from_fork+0x14/0x2c)
May 17 08:53:16 hassbian kernel: [28822.951472] Mem-Info:
May 17 08:53:16 hassbian kernel: [28822.951484] active_anon:109185 inactive_anon                                                                                                                                                                                                                                             :109209 isolated_anon:0
May 17 08:53:16 hassbian kernel: [28822.951484]  active_file:347 inactive_file:3                                                                                                                                                                                                                                             65 isolated_file:0
May 17 08:53:16 hassbian kernel: [28822.951484]  unevictable:2 dirty:2 writeback                                                                                                                                                                                                                                             :1 unstable:0
May 17 08:53:16 hassbian kernel: [28822.951484]  slab_reclaimable:1868 slab_unre                                                                                                                                                                                                                                             claimable:3107
May 17 08:53:16 hassbian kernel: [28822.951484]  mapped:456 shmem:2841 pagetable                                                                                                                                                                                                                                             s:921 bounce:0
May 17 08:53:16 hassbian kernel: [28822.951484]  free:5974 free_pcp:0 free_cma:1                                                                                                                                                                                                                                             024
May 17 08:53:16 hassbian kernel: [28822.951493] Node 0 active_anon:436740kB inac                                                                                                                                                                                                                                             tive_anon:436836kB active_file:1388kB inactive_file:1460kB unevictable:8kB isola                                                                                                                                                                                                                                             ted(anon):0kB isolated(file):0kB mapped:1824kB dirty:8kB writeback:4kB shmem:113                                                                                                                                                                                                                                             64kB writeback_tmp:0kB unstable:0kB pages_scanned:4752 all_unreclaimable? yes
May 17 08:53:16 hassbian kernel: [28822.951505] Normal free:23896kB min:16384kB                                                                                                                                                                                                                                              low:20480kB high:24576kB active_anon:436740kB inactive_anon:436836kB active_file                                                                                                                                                                                                                                             :1388kB inactive_file:1460kB unevictable:8kB writepending:12kB present:966656kB                                                                                                                                                                                                                                              managed:945520kB mlocked:8kB slab_reclaimable:7472kB slab_unreclaimable:12428kB                                                                                                                                                                                                                                              kernel_stack:1536kB pagetables:3684kB bounce:0kB free_pcp:0kB local_pcp:0kB free                                                                                                                                                                                                                                             _cma:4096kB
May 17 08:53:16 hassbian kernel: lowmem_reserve[]: 0 0
May 17 08:53:16 hassbian kernel: [28822.951513] Normal: 403*4kB (UMEH) 280*8kB (                                                                                                                                                                                                                                             UMEH) 206*16kB (UMEH) 69*32kB (UMEH) 38*64kB (UMEH) 15*128kB (UME) 6*256kB (M) 3                                                                                                                                                                                                                                             *512kB (M) 1*1024kB (M) 1*2048kB (M) 1*4096kB (C) = 23948kB
May 17 08:53:16 hassbian kernel: 4063 total pagecache pages
May 17 08:53:16 hassbian kernel: [28822.951559] 446 pages in swap cache
May 17 08:53:16 hassbian kernel: [28822.951563] Swap cache stats: add 27024, del                                                                                                                                                                                                                                             ete 26578, find 1573/1835
May 17 08:53:16 hassbian kernel: [28822.951565] Free swap  = 0kB
May 17 08:53:16 hassbian kernel: [28822.951567] Total swap = 102396kB
May 17 08:53:16 hassbian kernel: [28822.951569] 241664 pages RAM
May 17 08:53:16 hassbian kernel: [28822.951571] 0 pages HighMem/MovableOnly
May 17 08:53:16 hassbian kernel: [28822.951573] 5284 pages reserved
May 17 08:53:16 hassbian kernel: [28822.951575] 2048 pages cma reserved
May 17 08:53:16 hassbian kernel: [28822.951577] [ pid ]   uid  tgid total_vm                                                                                                                                                                                                                                                   rss nr_ptes nr_pmds swapents oom_score_adj name
May 17 08:53:16 hassbian kernel: [28822.951606] [  137]     0   137     2483                                                                                                                                                                                                                                                   190       7       0       34             0 systemd-journal
May 17 08:53:16 hassbian kernel: [28822.951613] [  144]     0   144     2897                                                                                                                                                                                                                                                    29       8       0      108         -1000 systemd-udevd
May 17 08:53:16 hassbian kernel: [28822.951620] [  430]     0   430     1268                                                                                                                                                                                                                                                    12       7       0       38             0 cron
May 17 08:53:16 hassbian kernel: [28822.951626] [  431]   105   431     1000                                                                                                                                                                                                                                                    60       6       0       25             0 avahi-daemon
May 17 08:53:16 hassbian kernel: [28822.951632] [  432]     0   432     8036                                                                                                                                                                                                                                                   161      10       0       97             0 rsyslogd
May 17 08:53:16 hassbian kernel: [28822.951638] [  437]     0   437      849                                                                                                                                                                                                                                                    20       5       0       32             0 systemd-logind
May 17 08:53:16 hassbian kernel: [28822.951645] [  438]   104   438     1373                                                                                                                                                                                                                                                    37       7       0       52          -900 dbus-daemon
May 17 08:53:16 hassbian kernel: [28822.951651] [  446]   105   446      969                                                                                                                                                                                                                                                    13       5       0       43             0 avahi-daemon
May 17 08:53:16 hassbian kernel: [28822.951657] [  455]     0   455     1789                                                                                                                                                                                                                                                    14       7       0       91             0 wpa_supplicant
May 17 08:53:16 hassbian kernel: [28822.951663] [  515]     0   515     6161                                                                                                                                                                                                                                                    35      11       0     1184             0 python3
May 17 08:53:16 hassbian kernel: [28822.951669] [  549]     0   549     6900                                                                                                                                                                                                                                                    21       6       0       14             0 rngd
May 17 08:53:16 hassbian kernel: [28822.951675] [  555]   109   555     8257                                                                                                                                                                                                                                                   167       9       0       42             0 mosquitto
May 17 08:53:16 hassbian kernel: [28822.951680] [  556] 65534   556      573                                                                                                                                                                                                                                                     4       5       0       27             0 thd
May 17 08:53:16 hassbian kernel: [28822.951686] [  656]  1000   656   198820                                                                                                                                                                                                                                                   687      10       0       66             0 presence
May 17 08:53:16 hassbian kernel: [28822.951691] [  760]     0   760      641                                                                                                                                                                                                                                                    19       5       0       51             0 dhcpcd
May 17 08:53:16 hassbian kernel: [28822.951697] [  761]  1001   761   329113   2                                                                                                                                                                                                                                             12886     613       0    15082             0 hass
May 17 08:53:16 hassbian kernel: [28822.951703] [  762]     0   762     1966                                                                                                                                                                                                                                                     1       7       0      114         -1000 sshd
May 17 08:53:16 hassbian kernel: [28822.951709] [  794]     0   794     1013                                                                                                                                                                                                                                                     1       7       0       32             0 agetty

Now the pi3 crashed (I can;t reach it in the network), shall I restart it (I need to unplug and plug again the power) and then run the code?

Did you have it wired or wireless? You should try to connect a keyboard and monitor to the pi to see why you even have no connectivity.

I am on ethernet. Bare in mind that it has been connected to the network for weeks/months without any problem.

Just since last day or 2 I am having this sort of problems. I do not think is related to connectivity, rather HASS crashing (or maybe SD card, but is strange since is relatively new)

now I am noticing that the file
home-assistant_v2

is huge, like 16Gbyte and growing rapidly

16.820.000 and after 1-2 minutes 16.840.000

Even if things in HA crashes, your PI should still be pingable. You should honestly get a keyboard and mouse because if you restart, your real issue could be wiped out. With a HA log that size, you could have turned in debug lodging and ran out of disk space. You might only be able to find out why by plugging in a monitor and keyboard…

can I just delet that file, while HASS running, or?

now is 16.910.000 definetely growing way too fast

where should I look that?

I have that in configuration. Shall I remove it?

What is the configuration?