Scraping shows no value

Configuration
7

The website is the governing council for muslims in Singapore. Their calculations take precedence in these matters. The timings indicated in the website are fix-ed and calibrated according to various factors.

Looking at this technically, I am guessing that the websites have altered their page source resulting in my scrape sensors no longer being active.
(I have no programming/coding background and only recently started learning those because of HA)

Am I able to scrape the timings from either of these websites?

8

I have no clue what and how many websites exist so cannot asnwer this.
The ones I chose to feedback the differences are random , i.e. popping up via google. Also, governmental sites may not be correct at times :slight_smile:
If the integration is ‘off’ then I would report an issue there.

9

Sure, in this case, we can trust the timing provided on the websites mentioned on the first post.

I shall bring up an issue with the dev of that integration.

In the meantime, I would like to learn if I can scrape the timings off either of these websites. Would you be able to point a direction that I can follow?

10

I looked at your websites just now and they changed to :01 and also mine changed to :57 (from :56)
So… I am a bit confused now, should this not be stable per day?

11

I checked the docs and (learning whilst reading) apparently there are many calculations.
From this, it shuld cover your MUIS, if not I would raise a ticket/issue
Islamic Prayer Times - Home Assistant (home-assistant.io)

12

It looks like this change is being caused by the website now using javascript to inject that data into the website. And apparently the Scrape integration cannot handle that.

However, following this post: Scrape HTTPS JavaScript generated content I found a solution for this.
The prayer times data is now delivered as JSON through a separate api link.
So if you create a RESTful Sensor that retrieves the prayer times from the api website, and a Template sensor to retrieve the JSON attribute it should be working again.
Something like this:

sensor:
  - platform: rest
    name: Singapore Prayer times
    resource: https://www.muis.gov.sg/api/pagecontentapi/GetPrayerTime
    scan_interval: 28800
    json_attributes:
      - Subuh
      - Syuruk
      - Zohor
      - Asar
      - Maghrib
      - Isyak

template:
  - sensor:
      - name: Zuhur praying time
        state: "{{ state_attr('sensor.singapore_prayer_times', 'Zohor') }}"
1 Like
13

YOU ARE AMAZING. I have managed to have all the timings back. At the same time, I learnt a bit more than I did yesterday. How do I buy you a coffee?

14

Great! You are welcome :smiley:
About the coffee: no, that’s OK. When I started with Home Assistant two years ago or so I also was helped several times via this community, so I am happy to be able to return this now when I can.
May be you can do the same sometimes when you have the time?
And I am still learning a lot as well while following the forums and looking into questions.
In this case I found the solution by reading this post from @Troon and this post from @ikea.

1 Like
15

Hi. Do you mind posting your configuration? I’m still trying to figure this one.

16

Sure, which part of the config is creating a stumbling block for you currently?

17

I managed to get the prayer times, but they are not triggering my automation. I suspect it’s because the time is in 12hr format and I can’t figure how to change to 24hr.

18

I have working on mine. it works great.

However, one thing that bothers me is my time templates. I can share it here (It’s ugly and the time shows every minute in my logbook)

I hope you or someone else can make it better.

I shall share it tomorrow as I am currently replying you through ny phone.

19 
- platform: template
  sensors:
    time_now:
      friendly_name: Time
      unique_id: timeX
      value_template: >-
        {{ now().strftime('%-I:%M') }}
- platform: time_date
  display_options:
    - "time"
    - "date"
### adhan except subuh timing in 24hour format ###
- platform: template
  sensors:
    zuhur_format:
      friendly_name: zuhur 24 hour format
      value_template: >-
        {% set t = states('sensor.zuhur1') %}
        {{ (t[:1] | int + 12) ~ t[-3:] }}
- platform: template
  sensors:
    asar_format:
      friendly_name: asar 24 hour format
      value_template: >-
        {% set t = states('sensor.asar1') %}
        {{ (t[:1] | int + 12) ~ t[-3:] }}
- platform: template
  sensors:
    maghrib_format:
      friendly_name: maghrib 24 hour format
      value_template: >-
        {% set t = states('sensor.maghrib1') %}
        {{ (t[:1] | int + 12) ~ t[-3:] }}
- platform: template
  sensors:
    isyak_format:
      friendly_name: isyak 24 hour format
      value_template: >-
        {% set t = states('sensor.isyak1') %}
        {{ (t[:1] | int + 12) ~ t[-3:] }}

Currently Zuhur is before 1300hrs, So I had to use the original format of the time rather than change it to 24hours.

Screenshot 2023-10-06 074010
Screenshot 2023-10-06 0740102151×1247 114 KB

This is what I meant by ugly. However, the automation works great.

20

Are you aware that you can filter out (exclude) specific entries in the logbook?

I personally know nothing about Muslim praying times, so please explain: is there always one time shown per day per location on this muis website, and are those times give in 12-hour format?
If so, then why isn’t the am / pm code given on that site?
How do you know whether it is in the morning or the evening?
Or is this indicated by those icons in front of the times?

21

Thank you for sharing. (logbook filters)
I have not come across it before.

In regards to the praying times, there is a 24 hour format provided by this website,
an affiliate of MUIS.
However, I did not successfully manage to retrieve/scrape the timings from that website.

How do we know whether the 12 hour format praying times are for AM or PM:
I would say it’s because of the arrangement and the meaning of the prayers.
Subuh means dawn
Syuruk means sunrise
etc

22

Ah! I misunderstood and assumed these names to be city names with every city each own local time for a prayer…
OK, so these names are the six prayers to be done each day at the given time, and these are all always either in the morning or in the afternoon?
If so, then this might be a solution:

sensor:
  - platform: rest
    name: Singapore Prayer times
    resource: https://www.muis.gov.sg/api/pagecontentapi/GetPrayerTime
    scan_interval: 28800
    json_attributes:
      - Subuh
      - Syuruk
      - Zohor
      - Asar
      - Maghrib
      - Isyak

template:
  - sensor:
      - name: Subuh praying time
        state: "{{ as_timestamp(strptime(state_attr('sensor.singapore_prayer_times', 'Subuh')~' AM', '%I:%M %p')) | timestamp_custom('%H:%M') }}"
      - name: Isyak praying time
        state: "{{ as_timestamp(strptime(state_attr('sensor.singapore_prayer_times', 'Isyak')~' PM', '%I:%M %p')) | timestamp_custom('%H:%M') }}"

Like this you can create template sensors for each prayer with the correct AM or PM code.
What this template does is:

  1. it reads the 12-hour time as a string from the rest sensor
  2. it adds either “ AM” or “ PM” to complete the 12-hour string format
  3. it converts this 12-hour format string to a timestamp
  4. it converts that timestamp into a 24-hour format
1 Like
23

Thank you, you’re awesome!!

I will be trying these out. I would love to learn more about this, do you have notes/documentation for the template you just created?

(just to correct you, it’s five prayer times)
“Syuruk” indicates to us humans that the time period to perform the subuh period at its appropriate time has ended.

24

I do have some documentation sources that I collected (see below), but I learned, and still do learn, the most from trying to understand code that is posted in this community by others.

Some applicable coding sources:
HA Templating
HA Template integration
Jinja Template Designer Documentation
Expressions
Python - datetime — Basic date and time types

1 Like
25

Hello guys, great solution by the way. it works in getting the prayer times, but the timing that i get is from 29 Dec 2023

image
image852×305 14.7 KB

27 
sensor:
  - platform: rest
    name: Singapore Prayer times
    resource: "https://www.muis.gov.sg/api/pagecontentapi/GetPrayerTime?v={{ as_timestamp(now()) | int }}"
    scan_interval: 28800
    json_attributes:
      - Subuh
      - Syuruk
      - Zohor
      - Asar
      - Maghrib
      - Isyak
      - PrayerDate
      - Hijri

template:
  - sensor:
      - name: Zuhur praying time
        state: "{{ strptime(state_attr('sensor.singapore_prayer_times', 'Zohor')~' PM', '%I:%M %p').strftime('%H:%M') }}"
  - sensor:
      - name: Subuh praying time
        state: "{{ strptime(state_attr('sensor.singapore_prayer_times', 'Subuh')~' AM', '%I:%M %p').strftime('%H:%M') }}"
  - sensor:
      - name: Syuruk praying time
        state: "{{ strptime(state_attr('sensor.singapore_prayer_times', 'Syuruk')~' AM', '%I:%M %p').strftime('%H:%M') }}"
  - sensor:
      - name: Asar praying time
        state: "{{ strptime(state_attr('sensor.singapore_prayer_times', 'Asar')~' PM', '%I:%M %p').strftime('%H:%M') }}"
  - sensor:
      - name: Maghrib praying time
        state: "{{ strptime(state_attr('sensor.singapore_prayer_times', 'Maghrib')~' PM', '%I:%M %p').strftime('%H:%M')  }}"
  - sensor:
      - name: Isyak praying time
        state: "{{ strptime(state_attr('sensor.singapore_prayer_times', 'Isyak')~' PM', '%I:%M %p').strftime('%H:%M') }}"
  - sensor:
      - name: Prayer Date
        state: "{{ state_attr('sensor.singapore_prayer_times', 'PrayerDate') }}"
  - sensor:
      - name: Hijri Date
        state: "{{ state_attr('sensor.singapore_prayer_times', 'Hijri') }}"
  - sensor: 
      - name: Time Now
        state: '{{ now().strftime("%H:%M") }}'

This is working