Companion app for iOS 2024.5: Let me Assist you 🍎

Blog
1

Hey, this is Bruno. It has been a great few months since I joined Nabu Casa to work full-time on the Home Assistant iOS app, again a big thanks to our Home Assistant Cloud subscribers for making this possible. Today, I would like to recap what has been introduced since the last blog post for version 2024.1 and discuss more about how Assist is being integrated into the Apple ecosystem - along with a number of other improvements to the app.

Assist on iOS

I am very happy to bring Assist for iOS with a new native UI, integrated with iOS shortcuts and widgets, so let’s start with a quick demo:

In this demo, you see Assist being triggered from the iPhone 15 Pro “action button” and lock screen widget. Those are a few ways of interacting with Assist, but as it’s an iOS shortcut, you can trigger it in all sorts of creative ways. You can create an automation triggered by scanning an NFC tag that launches Assist, or even add it to an existing shortcut you already use.

“What if I don’t have an iPhone 15 pro to use the action button?”

It would be ideal if we could replace Siri with Assist in the power button long press, but since Apple doesn’t allow that. The second best option becomes the action button. For iPhones without it, there is a third option, the accessibility feature called “back tap”. Follow this guide on the Apple page to learn how to configure it.

Widgets

If back tap doesn’t work for you, Assist can be activated via our widgets from the home screen or with our very first iOS lock screen widget, which looks great:

Accessibility feature alternative

There is “one more thing” that you can use to launch Assist, iOS has an accessibility feature called “Voice Control” which allows a user to basically navigate an iPhone completely using voice commands. On top of that they allow you to add “custom commands” so… Yes! You can just add “Ok Nabu”, then use the option Run shortcut and have it trigger “Assist In App”.

There are a couple of things to take into consideration. I tried this last weekend, and although it’s pretty cool, it’s not as fast as calling Siri. You will also have full voice control enabled, so you trigger other commands every once in a while.

Other highlights

  • We have introduced the ability to run your Action widgets in the background, so you can quickly trigger them instead of having to wait for the app to launch to see the result.

  • This version introduced a simple way to create an automation for your iOS Action: simply tap on the “create automation” button.

  • We started improving our widgets’ performance and capabilities. I know you all have great ideas, and I am currently working on the foundation to allow those to happen, but please continue sending feature requests to our Home Assistant Community.

  • When we first released CarPlay integration, we missed out on displaying all areas due to a limitation in the number of items allowed on a list. Since 2024.2, we have worked around this by using the first and last rows of the areas list as pagination arrows, so you can now see all your rooms.

  • The Actions UI in CarPlay was aligned to display the same information as in Apple Watch, previously we were displaying the “Identifier” and ”Text”, now we display just the “Text” so we keep consistency between platforms.

  • We also added a button to reload your widgets, since sometimes they may become blank for a few users after an update. For actions, we added a button to update server-created Actions manually, so you don’t need to reopen the app for that (but remember to restart HA after adding or removing Actions from YAML).

  • Memory usage and performance issues on our macOS client are big topics in our GitHub right now, and we are aware of them and are working towards improving them. This release is the first iteration in that direction.

Before wrapping up, I would like to say how important reporting issues on GitHub is for the development of these companion apps. We are addressing them as fast as we can and with the attention necessary to make sure your problems are resolved. I would also like to take this opportunity to invite iOS devs (aspiring or established) who wish to contribute to jump in. Drop me a message so we can work together and move faster, I will be happy to help you get started.

This is a companion discussion topic for the original entry at https://www.home-assistant.io/blog/2024/05/22/companion-app-for-ios-20245-assist
15 Likes
2

What are the pre-requisites here? Is home assistant cloud required? I cannot find a way how to bring up assistant using shortcuts

3

Hi,

Great news and features :ok_hand:

As far as I know (and tried) it’s not possible to sent notifications to Carplay.

Is this on the roadmap or is it working but don’t I know how to do it?

Best regards,
Rien

1 Like
4

Home Assistant Cloud is not required for Assist, you can use your voice assistants configured in Home Assistant with iOS.
After you have an Assist pipeline configured you can use it with the App and if you want to use shortcuts you just need to search for “Assist In App” shortcut

5

Currently is not on the roadmap, it’s on my radar to investigate a nice and safe way to implement that but not as a top priority for now

6

When I launch the Assist in App from the available widget, it does not hear me at all until I cancel the mic and then click the mic button again. That makes it unusable as a voice command option. Am I doing something wrong?

7

Works perfectly on my iPhone, thank you very much!

2 Likes
8

New to home assistant and in the planning right now. Does the IOS app work on the apple watch? I know it does in carplay which is awesome to have.

9

Yes, it does

1 Like
10

Exactyle the same thing is happening here…

11

This update broke all my Apple Watch actions, worked fine on all previous versions and also still work in the iPhone widget but no longer on the Apple Watch.

The Apple Watch loads the trigger and then buzzes shortly and nothing happens.

Edit:
It’s like the ios.action_fired is not defined in the Companion App anymore since there’s a “Create automation” button to execute and an example code under it. The widget on the iPhone can still launch it, but the Apple Watch no longer knows how.

12
1 Like
13

That’s weird, it has work reliably for me, can you open a GitHub issue with detailed information?

14

Are there instructions available for adding the Lock Screen widget. I installed the update but am not seeing the widget on the list of possible widgets to add.

15

If you search for home assistant it should be there, otherwise please try rebooting your phone.

16

Thank you Bruno! A reboot did the trick. Really appreciate all your work on this iOS app!

1 Like
17

Wow this does not work well for me at all.

Tap widget and ask “What’s the lounge room temperature?”

Wait…
Wait…
Wait…

Ask again

Wait…
Wait…
Wait…

Ask again

Wait…
Wait…
Wait…

Tap listening icon.

Tap microphone icon.

Ask again.

Hears: “And now? Yeah.”
“Sorry I couldn’t understand that”

Tap microphone button. Ask again.

Finally get correct answer.

Check HA logs:

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/connection.py:167
integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 09:23:48 (217 occurrences)
Last logged: 10:34:40

[140270936648640] Received binary message for non-existing handler 1
[140270942187712] Received binary message for non-existing handler 1
[140270942187712] Received binary message for non-existing handler 2
[140272882262464] Received binary message for non-existing handler 1
[140272882262464] Received binary message for non-existing handler 2

18

I created a feature request for some assistant functionality that i would like to see. Lets make it happen!.. Initiate Voip call from home assistant - #4 by nickrout

19

Thanks for reporting, can you open a detailed github issue with information like… whats your device, whats your assist pipeline configuration, if you were using external mic (airpods for example) or not?

20

Unfortunately I have removed all the wyoming/piper stuff now.

I don’t really have a use for a fancy voice remote control. I prefer to have everything automated.

2 Likes