Scraping shows no value

Configuration
1

Hi all, I am looking to scrape the prayer times from this website,
https://www.muis.gov.sg/ or https://muslim.sg/

Reason being, the islamic prayer time integration does not give the same timings as the local Muslim governing body in my country.

Up until a week ago, the scrape was working fine to be included in my automations. However, suddenly it was not giving any value/data after being okay for 2 months.

I went on to have a look at the sites and notice that nothing changed visually.

Using multiscrape, I included this in yaml.

- resource: https://www.muis.gov.sg
  scan_interval: 28800
  sensor:
    - unique_id: zuhur
      name: Zuhur
      select: "#PrayerTimeControl1_Zohor"

I have been trying to regain the timings as sensor to no avail for the past week and picking up knowledge on css selectors and such.

2

Have you looked at the native integration in HA ?
image

3

Yes. I tried that at first. However,the values/timings were off

4

What do you mean with ‘off’, fixed values off or just plain wrong?

5

The timings for 2 out of the 5 prayer times were not the same. For example. Asar is 4:03pm on the website. But the integration shows 4:01pm

6

Note that I am not muslim so bear with me…only looking at this purely technical
I checked 4 websites for prayer times for asr today and got 4:55 / 4:56 /4:56 / 4:57
My integration show 4:56
I have no clue why they differ but from where I stand…I would not trust any website before one knows what is behind its definition, could it be that it looks more to sun-rise/set and these indeed differe it one is a bit more east/west?

EDIT: the same applies to the integration of course but first one would need to know the true time, only then one can verify the integration

7

The website is the governing council for muslims in Singapore. Their calculations take precedence in these matters. The timings indicated in the website are fix-ed and calibrated according to various factors.

Looking at this technically, I am guessing that the websites have altered their page source resulting in my scrape sensors no longer being active.
(I have no programming/coding background and only recently started learning those because of HA)

Am I able to scrape the timings from either of these websites?

8

I have no clue what and how many websites exist so cannot asnwer this.
The ones I chose to feedback the differences are random , i.e. popping up via google. Also, governmental sites may not be correct at times :slight_smile:
If the integration is ‘off’ then I would report an issue there.

9

Sure, in this case, we can trust the timing provided on the websites mentioned on the first post.

I shall bring up an issue with the dev of that integration.

In the meantime, I would like to learn if I can scrape the timings off either of these websites. Would you be able to point a direction that I can follow?

10

I looked at your websites just now and they changed to :01 and also mine changed to :57 (from :56)
So… I am a bit confused now, should this not be stable per day?

11

I checked the docs and (learning whilst reading) apparently there are many calculations.
From this, it shuld cover your MUIS, if not I would raise a ticket/issue
Islamic Prayer Times - Home Assistant (home-assistant.io)

12

It looks like this change is being caused by the website now using javascript to inject that data into the website. And apparently the Scrape integration cannot handle that.

However, following this post: Scrape HTTPS JavaScript generated content I found a solution for this.
The prayer times data is now delivered as JSON through a separate api link.
So if you create a RESTful Sensor that retrieves the prayer times from the api website, and a Template sensor to retrieve the JSON attribute it should be working again.
Something like this:

sensor:
  - platform: rest
    name: Singapore Prayer times
    resource: https://www.muis.gov.sg/api/pagecontentapi/GetPrayerTime
    scan_interval: 28800
    json_attributes:
      - Subuh
      - Syuruk
      - Zohor
      - Asar
      - Maghrib
      - Isyak

template:
  - sensor:
      - name: Zuhur praying time
        state: "{{ state_attr('sensor.singapore_prayer_times', 'Zohor') }}"
1 Like
13

YOU ARE AMAZING. I have managed to have all the timings back. At the same time, I learnt a bit more than I did yesterday. How do I buy you a coffee?

14

Great! You are welcome :smiley:
About the coffee: no, that’s OK. When I started with Home Assistant two years ago or so I also was helped several times via this community, so I am happy to be able to return this now when I can.
May be you can do the same sometimes when you have the time?
And I am still learning a lot as well while following the forums and looking into questions.
In this case I found the solution by reading this post from @Troon and this post from @ikea.

1 Like
15

Hi. Do you mind posting your configuration? I’m still trying to figure this one.

16

Sure, which part of the config is creating a stumbling block for you currently?

17

I managed to get the prayer times, but they are not triggering my automation. I suspect it’s because the time is in 12hr format and I can’t figure how to change to 24hr.

18

I have working on mine. it works great.

However, one thing that bothers me is my time templates. I can share it here (It’s ugly and the time shows every minute in my logbook)

I hope you or someone else can make it better.

I shall share it tomorrow as I am currently replying you through ny phone.

19 
- platform: template
  sensors:
    time_now:
      friendly_name: Time
      unique_id: timeX
      value_template: >-
        {{ now().strftime('%-I:%M') }}
- platform: time_date
  display_options:
    - "time"
    - "date"
### adhan except subuh timing in 24hour format ###
- platform: template
  sensors:
    zuhur_format:
      friendly_name: zuhur 24 hour format
      value_template: >-
        {% set t = states('sensor.zuhur1') %}
        {{ (t[:1] | int + 12) ~ t[-3:] }}
- platform: template
  sensors:
    asar_format:
      friendly_name: asar 24 hour format
      value_template: >-
        {% set t = states('sensor.asar1') %}
        {{ (t[:1] | int + 12) ~ t[-3:] }}
- platform: template
  sensors:
    maghrib_format:
      friendly_name: maghrib 24 hour format
      value_template: >-
        {% set t = states('sensor.maghrib1') %}
        {{ (t[:1] | int + 12) ~ t[-3:] }}
- platform: template
  sensors:
    isyak_format:
      friendly_name: isyak 24 hour format
      value_template: >-
        {% set t = states('sensor.isyak1') %}
        {{ (t[:1] | int + 12) ~ t[-3:] }}

Currently Zuhur is before 1300hrs, So I had to use the original format of the time rather than change it to 24hours.

Screenshot 2023-10-06 074010
Screenshot 2023-10-06 0740102151×1247 114 KB

This is what I meant by ugly. However, the automation works great.

20

Are you aware that you can filter out (exclude) specific entries in the logbook?

I personally know nothing about Muslim praying times, so please explain: is there always one time shown per day per location on this muis website, and are those times give in 12-hour format?
If so, then why isn’t the am / pm code given on that site?
How do you know whether it is in the morning or the evening?
Or is this indicated by those icons in front of the times?