I am trying to scrape a download number value from a public website but the trial and error approach with always restarting home assistant is quite cumbersome to nail down the exact value.
Is there some easier test platform like the developer tools in home assistant to try if the sensor syntax is correct before restarting everytime?
So far I have this:
#Grab PrusaPrinter Download Counter
- platform: scrape
resource: https://www.prusaprinters.org/social/101152-pixelwave/about
name: Prusa Download Counter
select: 'strong[_ngcontent-prusa-front-app-c181]'
value_template: '{{ value }}'
But it throws “unknown” …
Hellis81
January 28, 2022, 8:08am
2
I think you can just reload the sensors.
I generally find Node red better for these purposes as you can use regex or substring to get the value.
…tried reloading all of them. But unfortunately only restart works.
Node Red I have not touched yet …
Hellis81
January 28, 2022, 8:18am
4
Looking at the source i’d say it’s impossible to scrape.
Damn … its that “971” value:
Hellis81
January 28, 2022, 8:24am
6
I know. But look at the source instead.
koying
January 28, 2022, 9:34am
7
Just send a POST to “https://www.prusaprinters.org/graphql/ ” with JSON payload:
{"operationName":"UserProfileSocial","variables":{"id":"101152"},"query":"query UserProfileSocial($id: ID!) {\n user(id: $id) {\n ...AvatarUserFragment\n hashId\n publicUsername\n email\n place\n offersPrintOnDemand\n offersTechSupport\n offersTechService\n offersShowAndTell\n isFollowedByMe\n downloadCount\n printsCount\n eduProjectsCount\n userGcodesCount\n followersCount\n followingCount\n makesCount\n collectionsCount\n eventsCount\n rating\n isPublicSpace\n publicSpaceUrl\n isTeacher\n promotionLinks {\n status\n url\n __typename\n }\n badgesProfileLevel {\n profileLevel\n __typename\n }\n badgesSelection {\n order\n userBadge {\n id\n name\n description\n levelsCount\n currentBadgeDisplayName\n currentBadgeLevel {\n name\n image\n __typename\n }\n __typename\n }\n __typename\n }\n coordinates {\n lat\n lng\n __typename\n }\n locationSharing\n bio\n donationLinks {\n id\n title\n url\n __typename\n }\n socialLinks {\n id\n socialType\n url\n __typename\n }\n printers {\n id\n name\n __typename\n }\n languages {\n id\n name\n __typename\n }\n techService {\n techServiceBio\n techServicePrinters {\n id\n name\n __typename\n }\n techServiceRatingAvg\n techServiceRatingCount\n __typename\n }\n __typename\n }\n}\n\nfragment AvatarUserFragment on UserType {\n id\n publicUsername\n avatarFilePath\n slug\n badgesProfileLevel {\n profileLevel\n __typename\n }\n __typename\n}\n"}
you’ll get the value in the response JSON
{
"data": {
"user": {
[...]
"downloadCount": 971,
How do I integrate this into Home Assistant as a sensor output?
koying:
{"operationName":"UserProfileSocial","variables":{"id":"101152"},"query":"query UserProfileSocial($id: ID!) {\n user(id: $id) {\n ...AvatarUserFragment\n hashId\n publicUsername\n email\n place\n offersPrintOnDemand\n offersTechSupport\n offersTechService\n offersShowAndTell\n isFollowedByMe\n downloadCount\n printsCount\n eduProjectsCount\n userGcodesCount\n followersCount\n followingCount\n makesCount\n collectionsCount\n eventsCount\n rating\n isPublicSpace\n publicSpaceUrl\n isTeacher\n promotionLinks {\n status\n url\n __typename\n }\n badgesProfileLevel {\n profileLevel\n __typename\n }\n badgesSelection {\n order\n userBadge {\n id\n name\n description\n levelsCount\n currentBadgeDisplayName\n currentBadgeLevel {\n name\n image\n __typename\n }\n __typename\n }\n __typename\n }\n coordinates {\n lat\n lng\n __typename\n }\n locationSharing\n bio\n donationLinks {\n id\n title\n url\n __typename\n }\n socialLinks {\n id\n socialType\n url\n __typename\n }\n printers {\n id\n name\n __typename\n }\n languages {\n id\n name\n __typename\n }\n techService {\n techServiceBio\n techServicePrinters {\n id\n name\n __typename\n }\n techServiceRatingAvg\n techServiceRatingCount\n __typename\n }\n __typename\n }\n}\n\nfragment AvatarUserFragment on UserType {\n id\n publicUsername\n avatarFilePath\n slug\n badgesProfileLevel {\n profileLevel\n __typename\n }\n __typename\n}\n"}
I put this in config:
- platform: rest
resource: https://www.prusaprinters.org/graphql/
method: POST
payload: '{"operationName":"UserProfileSocial","variables":{"id":"101152"},"query":"query UserProfileSocial($id: ID!) {\n user(id: $id) {\n ...AvatarUserFragment\n hashId\n publicUsername\n email\n place\n offersPrintOnDemand\n offersTechSupport\n offersTechService\n offersShowAndTell\n isFollowedByMe\n downloadCount\n printsCount\n eduProjectsCount\n userGcodesCount\n followersCount\n followingCount\n makesCount\n collectionsCount\n eventsCount\n rating\n isPublicSpace\n publicSpaceUrl\n isTeacher\n promotionLinks {\n status\n url\n __typename\n }\n badgesProfileLevel {\n profileLevel\n __typename\n }\n badgesSelection {\n order\n userBadge {\n id\n name\n description\n levelsCount\n currentBadgeDisplayName\n currentBadgeLevel {\n name\n image\n __typename\n }\n __typename\n }\n __typename\n }\n coordinates {\n lat\n lng\n __typename\n }\n locationSharing\n bio\n donationLinks {\n id\n title\n url\n __typename\n }\n socialLinks {\n id\n socialType\n url\n __typename\n }\n printers {\n id\n name\n __typename\n }\n languages {\n id\n name\n __typename\n }\n techService {\n techServiceBio\n techServicePrinters {\n id\n name\n __typename\n }\n techServiceRatingAvg\n techServiceRatingCount\n __typename\n }\n __typename\n }\n}\n\nfragment AvatarUserFragment on UserType {\n id\n publicUsername\n avatarFilePath\n slug\n badgesProfileLevel {\n profileLevel\n __typename\n }\n __typename\n}\n"}'
But sensor output is this:
koying
January 28, 2022, 2:40pm
11
They require the content-type
- platform: rest
resource: https://www.prusaprinters.org/graphql/
method: POST
payload: '{"operationName":"UserProfileSocial","variables":{"id":"101152"},"query":"query UserProfileSocial($id: ID!) {\n user(id: $id) {\n ...AvatarUserFragment\n hashId\n publicUsername\n email\n place\n offersPrintOnDemand\n offersTechSupport\n offersTechService\n offersShowAndTell\n isFollowedByMe\n downloadCount\n printsCount\n eduProjectsCount\n userGcodesCount\n followersCount\n followingCount\n makesCount\n collectionsCount\n eventsCount\n rating\n isPublicSpace\n publicSpaceUrl\n isTeacher\n promotionLinks {\n status\n url\n __typename\n }\n badgesProfileLevel {\n profileLevel\n __typename\n }\n badgesSelection {\n order\n userBadge {\n id\n name\n description\n levelsCount\n currentBadgeDisplayName\n currentBadgeLevel {\n name\n image\n __typename\n }\n __typename\n }\n __typename\n }\n coordinates {\n lat\n lng\n __typename\n }\n locationSharing\n bio\n donationLinks {\n id\n title\n url\n __typename\n }\n socialLinks {\n id\n socialType\n url\n __typename\n }\n printers {\n id\n name\n __typename\n }\n languages {\n id\n name\n __typename\n }\n techService {\n techServiceBio\n techServicePrinters {\n id\n name\n __typename\n }\n techServiceRatingAvg\n techServiceRatingCount\n __typename\n }\n __typename\n }\n}\n\nfragment AvatarUserFragment on UserType {\n id\n publicUsername\n avatarFilePath\n slug\n badgesProfileLevel {\n profileLevel\n __typename\n }\n __typename\n}\n"}'
value_template: "{{ value_json.data.user.downloadCount }}"
headers:
Content-Type: application/json
1 Like
Awesome! Works … outputs exactly the number. Thanks a lot for the quick reply.
Sorry to res an old thread, but with the domain name change to printables, it seems this sensor no longer works, is there a work around for this does anyone know?
Thanks! I just wanted to post the same question … I tried the obvious to change the query request from “resource: Printables ” to “resource: Printables ” but it did not work …
Ok found the solution replace: “Printables ”
… with the current API location: “https://api.printables.com/graphql/​ ”
Tested and works again for me!
Amazing! I was almost there myself, cant believe I didnt try api.printab… I feel stupid now! Thanks for the update though!
1 Like
… the query request does not seem to work anymore?
Ok … found out the fix myself. One topic in the JSON query seems to be outdated so you just have to remove “currentBadgeDisplayName\n”. Works again!
#Grab PrusaPrinter Download Counter
- platform: rest
resource: https://api.printables.com/graphql/
method: POST
payload: '{"operationName":"UserProfileSocial","variables":{"id":"101152"},"query":"query UserProfileSocial($id: ID!) {\n user(id: $id) {\n ...AvatarUserFragment\n hashId\n publicUsername\n email\n place\n offersPrintOnDemand\n offersTechSupport\n offersTechService\n offersShowAndTell\n isFollowedByMe\n downloadCount\n printsCount\n eduProjectsCount\n userGcodesCount\n followersCount\n followingCount\n makesCount\n collectionsCount\n eventsCount\n rating\n isPublicSpace\n publicSpaceUrl\n isTeacher\n promotionLinks {\n status\n url\n __typename\n }\n badgesProfileLevel {\n profileLevel\n __typename\n }\n badgesSelection {\n order\n userBadge {\n id\n name\n description\n levelsCount\n currentBadgeLevel {\n name\n image\n __typename\n }\n __typename\n }\n __typename\n }\n coordinates {\n lat\n lng\n __typename\n }\n locationSharing\n bio\n donationLinks {\n id\n title\n url\n __typename\n }\n socialLinks {\n id\n socialType\n url\n __typename\n }\n printers {\n id\n name\n __typename\n }\n languages {\n id\n name\n __typename\n }\n techService {\n techServiceBio\n techServicePrinters {\n id\n name\n __typename\n }\n techServiceRatingAvg\n techServiceRatingCount\n __typename\n }\n __typename\n }\n}\n\nfragment AvatarUserFragment on UserType {\n id\n publicUsername\n avatarFilePath\n slug\n badgesProfileLevel {\n profileLevel\n __typename\n }\n __typename\n}\n"}'
name: Prusa Download Counter
value_template: "{{ value_json.data.user.downloadCount }}"
headers:
Content-Type: application/json
A shorter query for just the download count:
#Grab PrusaPrinter Download Counter
- platform: rest
resource: https://api.printables.com/graphql/
method: POST
payload: '{"operationName":"UserProfileSocial","variables":{"id": "101152"},"query":"query UserProfileSocial($id: ID!) {\n user(id: $id) {\n publicUsername\n downloadCount\n}}" }'
name: Prusa Download Counter
value_template: "{{ value_json.data.user.downloadCount }}"
headers:
Content-Type: application/json
Came here to make the same finding! Thanks again!
