Hi, I would like to log in to the site and then perform scraping. Unfortunately the site doesn’t provide any API. The form on the page expects the variables “username” and “password”. Then there is a CSRF token in a hidden form field.
If I use the Postman app and enter my login credentials and a valid CSRF token in body/form-data, I can log in. The page status returns 200 and I get a valid cookie.
The cURL code is as follows:
curl --location --request POST 'https://schoolclassname.tridnistranky.cz/prihlaseni/' \
--form 'login="username"' \
--form 'password="password"' \
--form 'csrf="valid csrf token"'
Now I’d like to automate the same thing using node-red. I created a flow that gets a valid csrf token and sends it back to the form. Unfortunately, I’m stuck at this point and don’t know exactly how to submit the form. I’ve tried several methods and none of them work.
var csrf = msg.payload[0].value;
var login = 'username';
var password = 'password';
msg.followRedirects = true
msg.headers = {};
//msg.headers['Content-Type'] = 'application/x-www-form-urlencoded';
//msg.headers['Accept'] = 'application/json';
//msg.headers['X-Auth-User'] = login;
//msg.headers['X-Auth-Key'] = password;
//msg.headers['X-CSRF-Token'] = csrf;
//msg.payload = {
// 'login': login,
// 'password': password,
// 'csrf': csrf
//}
msg.payload = "login=" + login;
msg.payload += "&password=" + password;
msg.payload += "&csrf=" + csrf;
return msg;
Here is the code for the function I use to create the headers. I tried several variations and combinations. I’ve left the previous variations as comments. I also tried defining the content-type of the header up in the “http request” node.
Here is my whole current flow.
[{"id":"630f1967ec6bb45e","type":"tab","label":"Flow 2","disabled":false,"info":"","env":[]},{"id":"dffb09e214114abb","type":"inject","z":"630f1967ec6bb45e","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":120,"y":40,"wires":[["a5fc156472af18a5"]]},{"id":"a5fc156472af18a5","type":"http request","z":"630f1967ec6bb45e","name":"","method":"GET","ret":"txt","paytoqs":"ignore","url":"https://schoolclassname.tridnistranky.cz/prihlaseni/","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"credentials":{},"x":130,"y":100,"wires":[["1c093d059381b830"]]},{"id":"54b05baf06a47cf9","type":"debug","z":"630f1967ec6bb45e","name":"debug 2","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":400,"y":280,"wires":[]},{"id":"1c093d059381b830","type":"html","z":"630f1967ec6bb45e","name":"Raed CSRF","property":"payload","outproperty":"payload","tag":"form#login-form>input#csrf","ret":"attr","as":"single","x":130,"y":160,"wires":[["ee1da4af691fdae1"]]},{"id":"ee1da4af691fdae1","type":"function","z":"630f1967ec6bb45e","name":"set login variables","func":"var csrf = msg.payload[0].value;\nvar login = 'username';\nvar password = 'password';\n\nmsg.followRedirects = true\n\n//msg.headers = {};\n//msg.headers['Content-Type'] = 'application/x-www-form-urlencoded';\n//msg.headers['Accept'] = 'application/json';\n\n//msg.headers['X-Auth-User'] = login;\n//msg.headers['X-Auth-Key'] = password;\n//msg.headers['X-CSRF-Token'] = csrf;\n\n//msg.payload = {\n// 'login': login,\n// 'password': password,\n// 'csrf': csrf\n//}\n\nmsg.payload = \"login=\" + login;\nmsg.payload += \"&password=\" + password;\nmsg.payload += \"&csrf=\" + csrf;\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":150,"y":220,"wires":[["37575f27a5835a67","bb3514dd0702ade4"]]},{"id":"bb3514dd0702ade4","type":"http request","z":"630f1967ec6bb45e","name":"","method":"POST","ret":"txt","paytoqs":"query","url":"https://schoolclassname.tridnistranky.cz/prihlaseni/","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[{"keyType":"Content-Type","keyValue":"","valueType":"other","valueValue":"application/x-www-form-urlencoded"}],"credentials":{},"x":130,"y":280,"wires":[["54b05baf06a47cf9"]]},{"id":"37575f27a5835a67","type":"debug","z":"630f1967ec6bb45e","name":"debug 1","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":400,"y":220,"wires":[]}]
If I log in to the site in my browser and watch what happens, I see the following redirection.
Currently nodered is returning me error 405.
Can you please check my node-red code, it’s the first flow I’m creating. Thank you very much for your advice.
