How to fill in and submit the login web form?

Hi, I would like to log in to the site and then perform scraping. Unfortunately the site doesn’t provide any API. The form on the page expects the variables “username” and “password”. Then there is a CSRF token in a hidden form field.

If I use the Postman app and enter my login credentials and a valid CSRF token in body/form-data, I can log in. The page status returns 200 and I get a valid cookie.
The cURL code is as follows:

curl --location --request POST 'https://schoolclassname.tridnistranky.cz/prihlaseni/' \
--form 'login="username"' \
--form 'password="password"' \
--form 'csrf="valid csrf token"'

Now I’d like to automate the same thing using node-red. I created a flow that gets a valid csrf token and sends it back to the form. Unfortunately, I’m stuck at this point and don’t know exactly how to submit the form. I’ve tried several methods and none of them work.

var csrf = msg.payload[0].value;
var login = 'username';
var password = 'password';

msg.followRedirects = true

msg.headers = {};
//msg.headers['Content-Type'] = 'application/x-www-form-urlencoded';
//msg.headers['Accept'] = 'application/json';
//msg.headers['X-Auth-User'] = login;
//msg.headers['X-Auth-Key'] = password;
//msg.headers['X-CSRF-Token'] = csrf;

//msg.payload = {
//    'login': login,
//    'password': password,
//    'csrf': csrf
//}

msg.payload = "login=" + login;
msg.payload += "&password=" + password;
msg.payload += "&csrf=" + csrf;

return msg;

Here is the code for the function I use to create the headers. I tried several variations and combinations. I’ve left the previous variations as comments. I also tried defining the content-type of the header up in the “http request” node.

Here is my whole current flow.

[{"id":"630f1967ec6bb45e","type":"tab","label":"Flow 2","disabled":false,"info":"","env":[]},{"id":"dffb09e214114abb","type":"inject","z":"630f1967ec6bb45e","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":120,"y":40,"wires":[["a5fc156472af18a5"]]},{"id":"a5fc156472af18a5","type":"http request","z":"630f1967ec6bb45e","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://schoolclassname.tridnistranky.cz/prihlaseni/","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"credentials":{},"x":130,"y":100,"wires":[["1c093d059381b830"]]},{"id":"54b05baf06a47cf9","type":"debug","z":"630f1967ec6bb45e","name":"debug 2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":400,"y":280,"wires":[]},{"id":"1c093d059381b830","type":"html","z":"630f1967ec6bb45e","name":"Raed CSRF","property":"payload","outproperty":"payload","tag":"form#login-form>input#csrf","ret":"attr","as":"single","x":130,"y":160,"wires":[["ee1da4af691fdae1"]]},{"id":"ee1da4af691fdae1","type":"function","z":"630f1967ec6bb45e","name":"set login variables","func":"var csrf = msg.payload[0].value;\nvar login = 'username';\nvar password = 'password';\n\nmsg.followRedirects = true\n\n//msg.headers = {};\n//msg.headers['Content-Type'] = 'application/x-www-form-urlencoded';\n//msg.headers['Accept'] = 'application/json';\n\n//msg.headers['X-Auth-User'] = login;\n//msg.headers['X-Auth-Key'] = password;\n//msg.headers['X-CSRF-Token'] = csrf;\n\n//msg.payload = {\n//    'login': login,\n//    'password': password,\n//    'csrf': csrf\n//}\n\nmsg.payload = \"login=\" + login;\nmsg.payload += \"&password=\" + password;\nmsg.payload += \"&csrf=\" + csrf;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":150,"y":220,"wires":[["37575f27a5835a67","bb3514dd0702ade4"]]},{"id":"bb3514dd0702ade4","type":"http request","z":"630f1967ec6bb45e","name":"","method":"POST","ret":"txt","paytoqs":"query","url":"https://schoolclassname.tridnistranky.cz/prihlaseni/","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Content-Type","keyValue":"","valueType":"other","valueValue":"application/x-www-form-urlencoded"}],"credentials":{},"x":130,"y":280,"wires":[["54b05baf06a47cf9"]]},{"id":"37575f27a5835a67","type":"debug","z":"630f1967ec6bb45e","name":"debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":400,"y":220,"wires":[]}]

If I log in to the site in my browser and watch what happens, I see the following redirection.

Currently nodered is returning me error 405.

Can you please check my node-red code, it’s the first flow I’m creating. Thank you very much for your advice.

Does this have anything to do with HA? This not the node red forum :slight_smile:

Of course, I will further process the data in HA. I just didn’t get that far… :grinning:
Sorry to bother you and I’ll try to ask on the node-red forum. :innocent:

If someone will solve the same problem, the question is asked on the Node-Red forum.
Is there any way to close this topic? I don’t see any button…