I’ve noticing that my Node-Red is only using one core of the 6 available. Is there a way to let Node-Red use more than one CPU core?
After a bit of searching and reading up on the subject…
‘Let’ - no. ‘Make’ - yes, but with caveats.
Node-RED (the main program) is written in JavaScript and runs under Node.js. Node was built specifically as a single-thread processor (to avoid any need for multi-process locking) in that all JavaScript code run is executed using just one thread. I read that some of the parts of node.js such as the V8 engine, DNS lookups etc will spawn additional threads, so there could be as many as 10 to 20 threads going under node.js itself. However, this still has node.js running Node-RED as a single thread which I understand will only ever use one core.
Node.js can, of course, be run as multiple instances, so I guess you could try running Node-RED more than once, each one taking one CPU core to itself. Of course the challenge is getting more than one Node-RED running, with different ports, and then working out how to actually use the various instances.
Node.js can, apparently, run child processes, so the single executing JS program can itself run worker processes, again the main thread doing the main work and distributing tasks as required. Of more practical use, Node.js also now includes the cluster module. This permits the creation of worker processes, each of which can execute on a different CPU core. Naturally, the master process has to do the work of creating and managing the worker processes.
Since Node-RED is a client program run under node.js, there are limited options for using clustering. I did find
which appears to use the clustering module to permit opening worker processes (which will use additional / all CPU cores) directly from a flow.
I have to say that I am certainly not going to try using this myself, since the overheads and risks almost certainly outweigh any slight benefit.
Since Node-RED runs on just one CPU core, if Node-RED saturates (non terminating loop, runs close to memory capacity and garbage collector runs repeatedly) then that just kills Node-RED and one core. If NR was busy using all my CPU cores, my machine and Home Assistant would stop too.
If I need more than one Node-RED, I can use multiple machines, eg Raspberry Pi. These can be easily set up with just Node-RED, and offer physical redundancy as well as ‘extra processing’.
Any benefit from multi-core operation is likely to be limited. Multi-process management takes up resource itself, and multi-threading really only adds benefit when concurrent processing is required to get through a lot of heavy work quickly.
Seriously - if your single core running Node-RED ever runs to full capacity, it is probably due to a non-terminating loop or a badly behaved contrib-node. Not something you want across all the cores at the same time.
Interesting subject though.
Thanks @Biscuit for taking the time to elaborate this complete and comprehensive answer.
You mentioned “non-terminating loop or a badly behaved contrib-node”. Probably one or both of these is what is causing my Node-RED to start eating a Core (causing delayed response on all the flows) after letting run for a few days. I have not being able to locate what part of the flows are causing this.
This started happening after I created a sub node with an extensive flow inside that also uses a bunch of the “Home Assistant Entities” nodes. So this sub node is the culprit for sure. I just don’t know how to find what part of it is causing the issue.
I made sure to put node.done(); after each function that send messages asynchronously. but other than that, I don’t know how to track what flow or node is eating resources.
Do you have an Idea on where I could find information on that regard?
This kind of issue is always very challenging to debug.
Having had a couple of issues with Node-RED falling over due to memory leakage, I am aware that consuming CPU core comes not only from getting stuck in a non-terminating loop, but also from memory leaks. When Node starts to run out of memory, the garbage collector scavenger comes into play. Normally garbage collection is frequent, uses little resource, and interleaves between normal execution. The scavenger routine however will halt normal processing and takes over, which is why it only runs as a last resort.
As a first step you might wish to look at your NR memory use, to check if you have an ongoing leak. If you are using function nodes, then the use of ‘var’ should be replaced with ‘let’ and ‘const’ as a first step. Keeping a message going with increasing data, splits without joins, http calls, timers and the like are other potential contributory reasons for a memory leak.
@Biscuit I came back here to say thank you. you point me in the right direction.
I made the GC to work every 5 minutes and that solved 80% of my problems. I improved my flows and function nodes and that solved 10% of my problems. I is now just matter of continuing improving my flows and functions to make it work better.
Thanks Again!
When running out of memory, forcing the GC to run can solve the issue but this is always going to be a short-term fix. Ideally memory use should be stable over time.
I should have added this flow for you before - this is what I use to continually monitor the NR memory use. Memory used will climb after a reboot, but should plateau after a few hours. If memory continually creeps upwards (over days and weeks) there is a leak somewhere. Of course, finding the leak can be the real challenge.
[
{
"id": "5e0e066c7957262a",
"type": "group",
"z": "f91b7d4d17ac6c09",
"style": {
"stroke": "#999999",
"stroke-opacity": "1",
"fill": "none",
"fill-opacity": "1",
"label": true,
"label-position": "nw",
"color": "#a4a4a4"
},
"nodes": [
"1a8578e1722bf90e",
"6b707944048e0f23",
"cc933977e642df17",
"539b284c2782480c",
"f812f2fa619bc191",
"5ac316e1e62fa3ba",
"1d9c03393e9d5de0",
"23a33f172f4dfe2d",
"50f0410a6a167e9c"
],
"x": 94,
"y": 579,
"w": 1032,
"h": 242
},
{
"id": "1a8578e1722bf90e",
"type": "function",
"z": "f91b7d4d17ac6c09",
"g": "5e0e066c7957262a",
"name": "Read Memory Use",
"func": "\nmsg.payload = {\n \"memory\": process.memoryUsage(),\n \"heap\": v8.getHeapStatistics(), \n \"space\": v8.getHeapSpaceStatistics()}\n\nreturn msg;",
"outputs": 1,
"timeout": "",
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [
{
"var": "v8",
"module": "v8"
},
{
"var": "process",
"module": "process"
}
],
"x": 450,
"y": 700,
"wires": [
[
"cc933977e642df17",
"f812f2fa619bc191"
]
]
},
{
"id": "6b707944048e0f23",
"type": "inject",
"z": "f91b7d4d17ac6c09",
"g": "5e0e066c7957262a",
"name": "",
"props": [
{
"p": "payload"
},
{
"p": "topic",
"vt": "str"
}
],
"repeat": "",
"crontab": "",
"once": false,
"onceDelay": 0.1,
"topic": "",
"payload": "",
"payloadType": "date",
"x": 260,
"y": 620,
"wires": [
[
"1a8578e1722bf90e"
]
]
},
{
"id": "cc933977e642df17",
"type": "debug",
"z": "f91b7d4d17ac6c09",
"g": "5e0e066c7957262a",
"name": "debug 92",
"active": false,
"tosidebar": true,
"console": false,
"tostatus": false,
"complete": "false",
"statusVal": "",
"statusType": "auto",
"x": 640,
"y": 640,
"wires": []
},
{
"id": "539b284c2782480c",
"type": "inject",
"z": "f91b7d4d17ac6c09",
"g": "5e0e066c7957262a",
"name": "Every 1 mins",
"props": [
{
"p": "payload"
},
{
"p": "topic",
"vt": "str"
}
],
"repeat": "60",
"crontab": "",
"once": false,
"onceDelay": 0.1,
"topic": "",
"payload": "",
"payloadType": "date",
"x": 220,
"y": 700,
"wires": [
[
"1a8578e1722bf90e"
]
]
},
{
"id": "f812f2fa619bc191",
"type": "delay",
"z": "f91b7d4d17ac6c09",
"g": "5e0e066c7957262a",
"name": "",
"pauseType": "rate",
"timeout": "5",
"timeoutUnits": "seconds",
"rate": "1",
"nbRateUnits": "5",
"rateUnits": "minute",
"randomFirst": "1",
"randomLast": "5",
"randomUnits": "seconds",
"drop": true,
"allowrate": false,
"outputs": 1,
"x": 680,
"y": 700,
"wires": [
[
"5ac316e1e62fa3ba"
]
]
},
{
"id": "5ac316e1e62fa3ba",
"type": "change",
"z": "f91b7d4d17ac6c09",
"g": "5e0e066c7957262a",
"name": "Format",
"rules": [
{
"t": "set",
"p": "payload",
"pt": "msg",
"to": "(\t\t/* FUNCTION: format value according to setting or best fit k/M/G */\t/* keep a set MB value for graph plotting on same axis */ \t $form:= function($val, $set) {(\t $type($val) = \"string\" ? $val:=0;\t $x:=$val>1000000000 ? \"G\" : $val>1000000 ? \"M\" : $val>1000 ? \"K\" : \"b\";\t $factor:= $exists($set) ? $uppercase($set) : $x;\t $fix:= $factor=\"K\" ? \"kB\" : $factor=\"M\" ? \"MB\" : $factor=\"G\" ? \"GB\" : \"bytes\";\t $fac:= $factor=\"K\" ? 1024 : $factor=\"M\" ? 1048576 : $factor=\"G\" ? 1073741824 : 1;\t $rnd:= $factor=\"K\" ? 0 : $factor=\"M\" ? 1 : $factor=\"G\" ? 2 : 0;\t $text:=$round($val/$fac, $rnd) & \" \" & $fix;\t {\"value\": $val, \"text\": $text, \"MB\": $round($val/1048576,1)}\t )};\t\t/* some fields are duplicated or unuseful, tidy heap names */\t $heap:=payload.heap~>|$|{},['does_zap_garbage']|;\t $heap:=$keys($heap).(\t $x:=$split($,\"_\")[$!=\"size\" and $!=\"total\" and $!=\"memory\" and $!=\"of\"]~>$join(\"_\");\t {$x: $lookup($heap, $)}\t )~>$merge();\t $memory:=payload.memory~>|$|{},['heapTotal','heapUsed', 'external']|;\t\t/* modify space to flattened object with space name prefixing field */\t/* ignore the read only block, and strip out space name field */\t $sarray:=payload.space;\t $slist:=$sarray.space_name[$!=\"read_only_space\"];\t $space:=$slist.(\t $name:=$;\t $obj:=$sarray[space_name=$name];\t $okeys:=$keys($obj)[$!=\"space_name\"];\t $okeys.{$substringBefore($name, \"_space\") & \"O\" & $substringBefore($, \"_size\") : $lookup($obj, $)}\t )~>$merge();\t \t $all:=$merge([{\"time_number\": $now()}, $memory, $heap, $space]);\t\t/* format everything except the basic 'numbers' */\t $result:=$keys($all).(\t $key:=$;\t $val:=$lookup($all, $key);\t {$key: $contains($key,\"number\") ? $val : $form($val)};\t )~>$merge();\t\t/* code to calculate the percentage space used for each group in space */\t $spaceList:=$keys($result)[$contains(\"O\")].$substringBefore(\"O\")~>$distinct();\t $spacepc:=$spaceList.(\t $space:= $lookup($result, $ & \"Ospace\").value;\t $used:= $lookup($result, $ & \"Ospace_used\").value;\t $phys:= $lookup($result, $ & \"Ophysical_space\").value;\t $aval:= $lookup($result, $ & \"Ospace_available\").value;\t $space>0? {$: {\t \"space\": $space,\t \"physical\": $round($phys*100/$space,0),\t \"used\": $round($used*100/$space,0),\t \"available\": $round($aval*100/$space,0)\t }})~>$merge();\t\t $result~>|$|{\"percent\": $spacepc}|;\t\t)",
"tot": "jsonata"
}
],
"action": "",
"property": "",
"from": "",
"to": "",
"reg": false,
"x": 860,
"y": 700,
"wires": [
[
"1d9c03393e9d5de0"
]
]
},
{
"id": "1d9c03393e9d5de0",
"type": "function",
"z": "f91b7d4d17ac6c09",
"g": "5e0e066c7957262a",
"name": "Save Details",
"func": "let memMon = flow.get(\"Memory\") || [];\n\n\n\nif (memMon.length > 300) { memMon.pop() }\n\nmemMon.unshift({ \"Time\": msg.payload.time_number, \"rss\": msg.payload.rss, \"heap\": msg.payload.heap, \"usedheap\": msg.payload.used_heap })\n\nflow.set(\"Memory\", memMon);\nmsg.payload = memMon;\nreturn msg;",
"outputs": 1,
"timeout": "",
"noerr": 0,
"initialize": "",
"finalize": "",
"libs": [],
"x": 1030,
"y": 700,
"wires": [
[
"23a33f172f4dfe2d"
]
]
},
{
"id": "23a33f172f4dfe2d",
"type": "change",
"z": "f91b7d4d17ac6c09",
"g": "5e0e066c7957262a",
"name": "Chart Array",
"rules": [
{
"t": "set",
"p": "payload",
"pt": "msg",
"to": "(\t $table:=payload^(Time);\t [{\"series\":[\"RSS\", \"HEAP\", \"UsedHeap\"],\t \"data\": [[$table.rss.MB], [$table.heap.MB], [$table.usedheap.MB]],\t \"labels\": [$table.Time]}];\t)",
"tot": "jsonata"
}
],
"action": "",
"property": "",
"from": "",
"to": "",
"reg": false,
"x": 770,
"y": 780,
"wires": [
[
"50f0410a6a167e9c"
]
]
},
{
"id": "50f0410a6a167e9c",
"type": "ui_chart",
"z": "f91b7d4d17ac6c09",
"g": "5e0e066c7957262a",
"name": "",
"group": "a354928d8634a7db",
"order": 22,
"width": 0,
"height": 0,
"label": "Memory Graph",
"chartType": "line",
"legend": "false",
"xformat": "HH:mm:ss",
"interpolate": "linear",
"nodata": "",
"dot": false,
"ymin": "",
"ymax": "",
"removeOlder": 1,
"removeOlderPoints": "",
"removeOlderUnit": "3600",
"cutout": 0,
"useOneColor": false,
"useUTC": false,
"colors": [
"#1f77b4",
"#aec7e8",
"#ff7f0e",
"#2ca02c",
"#98df8a",
"#d62728",
"#ff9896",
"#9467bd",
"#c5b0d5"
],
"outputs": 1,
"useDifferentColor": false,
"className": "",
"x": 960,
"y": 780,
"wires": [
[]
]
},
{
"id": "a354928d8634a7db",
"type": "ui_group",
"name": "Time Graph",
"tab": "689fbe1ea21c95ac",
"order": 1,
"disp": true,
"width": "18",
"collapse": false,
"className": ""
},
{
"id": "689fbe1ea21c95ac",
"type": "ui_tab",
"name": "RevPi Node-RED Memory",
"icon": "dashboard",
"order": 2,
"disabled": false,
"hidden": false
}
]
I did write a (very long, very tedious and sleep-inducing) reply last year about how Node-RED memory works… Just in case anyone (?) is at all interested
The important key things to do / not do seem to be:
The big one that made the difference for me, was to stop using ‘var’ in function nodes. Lots of loops repeatedly calling functions and declaring global variables soon eats up the memory.
Good luck!
Very interesting post in deed. I’ll read it after work.
I disabled the forced GC and I’m using your flow to track the memory. any way to pass that data to HA? So I can create some automations.
Try this flow. Uses a WebSocket Sensor to pass the NR memory back to HA.
I will leave you to experiment! It does open the opportunity to re-start NR (HA addon) using an automation (eg during the night, when memory exceeds a given figure)
[{"id":"539b284c2782480c","type":"inject","z":"249a680bf3f915e2","g":"53c4c81afcce308f","name":"Every 1 mins","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"60","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":160,"y":1560,"wires":[["1a8578e1722bf90e"]]},{"id":"1a8578e1722bf90e","type":"function","z":"249a680bf3f915e2","g":"53c4c81afcce308f","name":"Read Memory Use","func":"\nmsg.payload = {\n \"memory\": process.memoryUsage(),\n \"heap\": v8.getHeapStatistics(), \n \"space\": v8.getHeapSpaceStatistics()}\n\nreturn msg;","outputs":1,"timeout":"","noerr":0,"initialize":"","finalize":"","libs":[{"var":"v8","module":"v8"},{"var":"process","module":"process"}],"x":370,"y":1560,"wires":[["f812f2fa619bc191","3c575eaca7a42b1b","c879c5ac63f83116"]]},{"id":"3c575eaca7a42b1b","type":"ha-sensor","z":"249a680bf3f915e2","g":"53c4c81afcce308f","name":"NR Memory","entityConfig":"c650a3aa3b38025f","version":0,"state":"payload.memory.rss","stateType":"msg","attributes":[{"property":"memory","value":"payload.memory","valueType":"msg"},{"property":"heap","value":"payload.heap","valueType":"msg"},{"property":"space","value":"payload.space","valueType":"msg"},{"property":"timestamp","value":"","valueType":"date"}],"inputOverride":"allow","outputProperties":[],"x":650,"y":1580,"wires":[[]],"server":""},{"id":"c650a3aa3b38025f","type":"ha-entity-config","server":"","deviceConfig":"","name":"SC NR Memory","version":"6","entityType":"sensor","haConfig":[{"property":"name","value":"Node Red Memory"},{"property":"icon","value":""},{"property":"entity_category","value":""},{"property":"device_class","value":""},{"property":"unit_of_measurement","value":""},{"property":"state_class","value":""}],"resend":true,"debugEnabled":false}]
@Biscuit thanks for the help, I was able to clean up my code and I have no memory leaks. but my issue is still happening. after a few days, my subflow hogs the CPU usage. I created new topic in hope of some light. You seems to be very knowledgeable, do mind taking a look?
I looked already, but without a flow the guesswork is far too wide.
If CPU use goes up over a period of time, then I would first suspect memory leak, with loads of garbage collection attempts. You say your memory (Node-RED memory?) is OK, but I would still be interested to see if lots of large memory was being allocated and then released, meaning work for the GC.
After that, it must be something to do with too much going on - loops and iterations. Have you tried doing some logging of the flow?
You are using a subflow - I would also wonder if there is an overhead with running subflows in Node-RED… Does the subflow itself ever terminate? Node-RED works well for flows where the message goes from start to finish and then gets destroyed. Continuous loops, generating copy messages (multiple outputs from a node) and things like splits without joins can gum up the works.