0.5.1-rc2 is running great although it did fall over this afternoon with the ‘not enough space’ error. I’ve read your comments on the thread for previous cases of this. It’s not possible to tell the queue size on this version is it? Here’s my log if it’s any help. Please let me know if I can provide anything else.
Thanks
2020-03-15T20:53:32.695217177Z Camera_process started for front: 36
2020-03-15T20:53:32.706870315Z Camera_process started for side: 37
2020-03-15T20:53:32.717708336Z Camera_process started for back: 38
2020-03-15T20:53:32.718656172Z Starting process for front: 36
2020-03-15T20:53:32.725574945Z Starting process for side: 37
2020-03-15T20:53:32.732257925Z Starting process for back: 38
2020-03-15T20:53:32.738542468Z Camera_process started for frontdoor: 39
2020-03-15T20:53:32.753685121Z Starting process for frontdoor: 39
2020-03-15T20:53:32.792982202Z * Serving Flask app "detect_objects" (lazy loading)
2020-03-15T20:53:32.793070885Z * Environment: production
2020-03-15T20:53:32.793588914Z WARNING: Do not use the development server in a production environment.
2020-03-15T20:53:32.793791577Z Use a production WSGI server instead.
2020-03-15T20:53:32.793863224Z * Debug mode: off
2020-03-17T12:42:32.895044268Z Detection appears to be stuck. Restarting detection process
2020-03-17T12:42:32.895470741Z Waiting for detection process to exit gracefully...
2020-03-17T12:42:32.918762899Z ../src/plasma/store.cc:738: Disconnecting client on fd 6
2020-03-17T12:42:32.936826974Z Starting detection process: 24080
2020-03-17T12:43:00.890455815Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 87 objects to free up 80246016 bytes. The number of bytes in use (before this eviction) is 399783680.
2020-03-17T12:43:03.689770230Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 87 objects to free up 80246016 bytes. The number of bytes in use (before this eviction) is 399558656.
2020-03-17T12:43:06.450724241Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 87 objects to free up 80246016 bytes. The number of bytes in use (before this eviction) is 399333632.
2020-03-17T12:43:09.252161926Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 89 objects to free up 80477952 bytes. The number of bytes in use (before this eviction) is 399799808.
2020-03-17T12:43:12.124826983Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 95 objects to free up 80482560 bytes. The number of bytes in use (before this eviction) is 399803648.
2020-03-17T12:43:14.846103501Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 94 objects to free up 80481792 bytes. The number of bytes in use (before this eviction) is 399111680.
2020-03-17T12:43:17.710246128Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 94 objects to free up 80020992 bytes. The number of bytes in use (before this eviction) is 399342080.
2020-03-17T12:43:25.232561348Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 96 objects to free up 80022528 bytes. The number of bytes in use (before this eviction) is 399541248.
2020-03-17T12:43:37.765146771Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 87 objects to free up 80246016 bytes. The number of bytes in use (before this eviction) is 399696640.
2020-03-17T12:43:43.245745051Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 87 objects to free up 80246016 bytes. The number of bytes in use (before this eviction) is 399696640.
2020-03-17T12:43:48.886712205Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 87 objects to free up 80246016 bytes. The number of bytes in use (before this eviction) is 399696640.
2020-03-17T12:43:54.404359806Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 87 objects to free up 80246016 bytes. The number of bytes in use (before this eviction) is 399696640.
2020-03-17T12:43:59.824610111Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 87 objects to free up 80246016 bytes. The number of bytes in use (before this eviction) is 399696640.
2020-03-17T12:44:05.426071106Z ../src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 87 objects to free up 80246016 bytes. The number of bytes in use (before this eviction) is 399696640.
....
The not enough space messages are not necessarily a problem. They are an indication that there are more frames in cache and frigate isn’t processing them fast enough. In that situation, the cache frees up some space automatically to make room for the newer frames. Unless your video feeds stop completely, it should continue just fine. If it happens continuously, then I would recommend lowering the fps or using take_frame to reduce the fps.
For the queue size, I had to switch to a different type of queue that doesn’t deadlock with multi processing. This queue does not have a way to measure the queue size.
It did indeed stop working unfortunately. The message was being repeated in the log every X seconds for two hours until I restarted it. I believe the debug video stream was still running (not frozen) before I restarted, but there were no persons detected when there should have been.
I only have four low res 640x480 streams running at 8fps so I don’t think it should be hitting any limits. I did have a fifth camera until a squirrel chewed through the cable, but even then it ran fine 24x7 on previous versions.
I’ll try and do some more debugging if it does it again, and then reduce the fps to 5, to see if it makes a difference.
I’ve seen references to reconnect flags… for example… -reconnect 1 -reconnect_at_eof 1 -reconnect_streamed 1 -reconnect_delay_max 2000… but these references seem to be relatively old.
I’ve also seen references to 4.x versions coming in while searching which seems to possibly address some of the issues I’ve been seeing. Have you tested 4.x?
You shouldn’t need to reduce the fps at that resolution. I will look and see how it would be possible for the debug stream to keep working, but detection to stop.
Not sure if this is of any help. The FPS on each camera started going very erratic at the same time the plasma messages started, and both the coral fps and inference value stopped updating from the same time. Where it goes back to normal at about 1445 is after I restarted it.
I would still suggest running the ffmpeg command from my earlier post directly to see if it exits too. I have an idea for something that might help in the code.
Regarding your first paragraph - would you expect the values in the log message to change?
For my issue the same values repeat over and over, even across restarts. It’s always:
frigate-new | /arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 29 objects to free up 80201472 bytes. The number of bytes in use (before this eviction) is 398512640.
The version I stated using was 0.5.0-rc1 since I didn’t have a Coral device.
I’ve started that ffmpeg command in the container.
How will the version of frigate make a difference? If the issue is the ffmpeg parameters it will fail on both versions.
This seems to be an issue with how the parent process deals with the issue since in all cases, when the issue happens, a restart of the parent process fixes it.
I was hoping to confirm that ffmpeg does behave the same in both container versions. I wasn’t restarting the parent process in 0.3.0, so I am wondering if it wasn’t failing as frequently for some reason. I would also expect it to behave the same, so I don’t yet see how 0.3.0 would have been more stable.
After several hours, 0.5.1rc2, keep showing following message and it won’t recover:
/arrow/cpp/src/plasma/eviction_policy.cc:134: There is not enough space to create this object, so evicting 87 objects to free up 80246016 bytes. The number of bytes in use (before this eviction) is 399657472.
I have 2 cameras and it usually affects one of them. CPU usage increases and so does the fps and detection. The debug output shows grey/blue rapidly flashing image.
