CPU 100% in a docker

MarkR · March 8, 2020, 6:39pm

Hello

I noticed my CPU was 100% due to HA and I have emptied my MYSQL database and still it persisits. Any idea what I can do to see what is causing it please? I stop the docker and cpu drops right off. So I know it is something in HA

Cheers
Mark

sj3fk3 · March 8, 2020, 7:08pm

tip: if you press “c” in top, you can see the whole command-line. But I’m sure it will show homeassistant. Could be any number of things. What version are you running? Any idea if this is new or could be an exciting pre-condition?

p.s.
It should not really matter whether you run it in docker or not, you could check but running it in an virtual-env if you know how to do that.

MarkR · March 8, 2020, 7:13pm

cheers yeah just says home assistant. I am running the current version and I did have this issue a few days ago

sj3fk3 · March 8, 2020, 7:16pm

You could try:

strace -p 27189

and have a look if you see anything funky going on… It’s like looking into the matrix, sometimes you can see a glitch

MarkR · March 8, 2020, 7:19pm

boat loads of these

futex(0x7ffe85b4d4b4, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998712}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe85b4d4b4, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998923}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe85b4d4b4, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999183}) = -1 EAGAIN (Resource temporarily unavailable)

sj3fk3 · March 8, 2020, 7:26pm

That could be something, but could also be a side effect… See: https://blog.packagecloud.io/eng/2015/11/15/strace-cheat-sheet/

if you “grep them out” (add

| grep -v futex

is there anything else weird you see?

MarkR · March 8, 2020, 7:27pm

rolled back to Home Assistant 0.105.4

still the same… arrghh

sj3fk3 · March 8, 2020, 7:28pm

Yeah, I don’t think it’s the version, more likely a device that’s acting up.

MarkR · March 8, 2020, 7:30pm

thanks for the replies nothing obvious, but I guess time to start stripping crap out of my config

epoll_pwait(3, [{EPOLLIN, {u32=58, u64=140209207377978}}], 25, 0, NULL, 8) = 1
futex(0x7f85e43adc44, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
sendto(53, "2d73\r\nContent-Type: image/jpeg\r\n"..., 11643, 0, NULL, 0) = 11643
futex(0x7ffe29c2f054, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999360}) = -1 EAGAIN (Resource temporarily unavailable)
getpid()                                = 1
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f85e1736000
futex(0x7f85e43adc44, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
recvfrom(58, "Hg\37\355Sb9\306\334z\236\224\322*\365\346\235sg\nKq\36\325\221\266\216A\355QCi"..., 262144, 0, NULL, NULL) = 23787
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999070}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999196}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999531}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999275}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999183}) = -1 EAGAIN (Resource temporarily unavailable)
mremap(0x7f85e1736000, 266240, 24576, MREMAP_MAYMOVE) = 0x7f85e1736000
munmap(0x7f85e1736000, 24576)           = 0
futex(0x7f85e43b5eb4, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_pwait(3, [], 25, 0, NULL, 8)      = 0
futex(0x7ffe29c30644, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998416}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30644, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998677}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7f85e43afa64, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
sendto(53, "5ceb\r\nHg\37\355Sb9\306\334z\236\224\322*\365\346\235sg\nKq\36\325\221\266"..., 23795, 0, NULL, 0) = 23795
getpid()                                = 1
futex(0x7f85e43adc44, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_pwait(3, [{EPOLLIN, {u32=59, u64=140209207377979}}], 25, 269, NULL, 8) = 1
futex(0x7ffe29c30644, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998928}) = -1 EAGAIN (Resource temporarily unavailable)
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f85e1736000
futex(0x7f85e43adc44, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
recvfrom(59, "Content-Type: image/jpeg\r\nConten"..., 262144, 0, NULL, NULL) = 2947
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998719}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998996}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999735}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998936}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998970}) = -1 EAGAIN (Resource temporarily unavailable)
munmap(0x7f85e1736000, 266240)          = 0
futex(0x7f85e43b5d44, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_pwait(3, [{EPOLLIN, {u32=59, u64=140209207377979}}], 25, 0, NULL, 8) = 1
futex(0x7ffe29c30644, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998783}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30644, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999388}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30644, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999034}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30644, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998917}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30644, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998784}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7f85e43adc44, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
sendto(52, "b83\r\nContent-Type: image/jpeg\r\nC"..., 2954, 0, NULL, 0) = 2954
getpid()                                = 1
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f85e1736000
futex(0x7f85e43adc44, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
recvfrom(59, "\372U\233\247\313\376\25\237\346\240\270M\354B\36\t\333\234\17\\w\240\10\341;\241\372\22)\337\362\315"..., 262144, 0, NULL, NULL) = 18824
mremap(0x7f85e1736000, 266240, 20480, MREMAP_MAYMOVE) = 0x7f85e1736000
munmap(0x7f85e1736000, 20480)           = 0
futex(0x7f85e43b5d44, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
epoll_pwait(3, [{EPOLLIN, {u32=59, u64=140209207377979}}, {EPOLLIN, {u32=58, u64=140209207377978}}], 25, 0, NULL, 8) = 2
futex(0x7ffe29c30644, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999197}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30644, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998762}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30644, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999194}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30644, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999430}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7f85e43b5d44, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
sendto(52, "4988\r\n\372U\233\247\313\376\25\237\346\240\270M\354B\36\t\333\234\17\\w\240\10\341;\241"..., 18832, 0, NULL, 0) = 18832
futex(0x7ffe29c2f054, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999119}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c2f054, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999137}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c2f054, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998913}) = -1 EAGAIN (Resource temporarily unavailable)
getpid()                                = 1
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f85e1736000
futex(0x7f85e43b5d44, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
recvfrom(59, "\1/\16v/\n;\320b\t\226\3174\363\30\30\4\363\355M\3368\fq\217\326\200\33*\217\220\16"..., 262144, 0, NULL, NULL) = 3079
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998544}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999364}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999539}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998909}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4999430}) = -1 EAGAIN (Resource temporarily unavailable)
munmap(0x7f85e1736000, 266240)          = 0
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f85e1736000
futex(0x7f85e43adc44, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7f85f05aa904, FUTEX_WAKE_PRIVATE, 1) = 1
recvfrom(58, "Content-Type: image/jpeg\r\nConten"..., 262144, 0, NULL, NULL) = 35423
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998761}) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x7ffe29c30244, FUTEX_WAIT_PRIVATE, 2, {tv_sec=0, tv_nsec=4998976}) = -1 EAGAIN (Resource temporarily unavailable)

MarkR · March 9, 2020, 8:14am

I have tracked it down to the camera STREAM component. I removed the 1 camera stream I had setup and we are back to normal I have 4 other cameras setup but not using stream.

Didn’t have this issue with it until a few days ago… strange

ReDaLeRt · March 16, 2020, 12:02am

Which HA version are you using? 0.106.x?

ReDaLeRt · March 16, 2020, 12:25am

I’m suffering from the same issue: HA docker at 100% CPU constantly

MarkR · March 16, 2020, 1:05am

yep and 0.105… I remove stream and all is back to normal.

ReDaLeRt · March 16, 2020, 8:02pm

See: https://github.com/home-assistant/core/issues/32876