Home Organization with AI Help: Teaching Jarvis Where You Left Your Stuff

Share your Projects!
, ,
1

Imagine this: you’re elbow-deep in a project and need that one specific component. You ask “Hey Jarvis, where’s the flux capacitor?” and instantly get back “Second drawer, office desk.” Sounds like sci-fi, right? Well, it’s closer to reality than you think, and honestly, it works pretty damn well already.

Look, we all have too much crap. I’ve got 30 students at the lab, multiple 3D printers, a rocketry project, electric skates with Mad Max-inspired names, and enough ESP32s to start my own semiconductor shortage. The typical solution to this chaos? Manually register everything in an inventory system. Yeah… that’s never going to happen. The sheer thought of typing “M3 bolt, quantity 47, drawer 3A” for every single item makes me want to take a nap instead.

But here’s where it gets interesting: AI can actually help now. The workflow is beautifully simple - you snap a photo of your glorious mess, Claude looks at it and identifies most of what’s there, you make a few corrections because AI isn’t perfect (yet), and boom, everything gets registered in your Homebox inventory system. No manual typing, no decision fatigue about whether that thing should be categorized as “electronics” or “tools” or “why-did-I-buy-this.” The AI figures it out, you approve, done.

Now, I’ll be honest about what works today versus what’s still in the “wouldn’t it be cool if” category. Right now you need Claude.ai (the web interface that actually supports MCP connections), my custom Home Assistant addon that runs the MCP server, and Homebox (also running as an HA addon). You take the photo, upload to Claude, it detects the items using its vision capabilities, you review and correct anything it missed or got wrong, then Claude calls the MCP server which registers everything into Homebox. It’s not quite real-time conversation with a camera feed yet, but we’ve got to start somewhere, right?

Here’s why I had to build a custom MCP server instead of using the existing ones: the native Home Assistant MCP is super limited and can’t talk to Homebox at all. I tried for way too long to make it work with Nabu Casa URLs - spoiler alert, it doesn’t. So I said screw it and implemented my own MCP server specifically for this use case. You can find the whole thing at github.com/oangelo/homebox-mcp if you want to see how it works or contribute.

What’s still on the TODO list? The big one is getting this integrated with Home Assistant’s Assist so you can actually voice-ask “where did I put the screwdriver?” and get an answer. Homebox already has a Home Assistant integration, but I haven’t explored it much yet - seems pretty limited from what I’ve seen. The dream scenario is having your camera feed open and just chatting in real-time with an AI that’s watching what you’re organizing, but that’s definitely future territory.

Let me share a few things I’ve learned that might save you some headaches. First, when you’re working alongside an AI like this, it genuinely feels different - less like a solo chore and more like you’ve got a weirdly competent assistant. Second, decision fatigue is absolutely real. We’re all consumers drowning in stuff, and every item needs a home, which means another decision. The AI taking over the “where should this go” and “how should I describe it” decisions is genuinely liberating. Third, and this is critical: do NOT try to organize your entire house in one go. Start small. Pick one drawer, one shelf, maybe one cabinet. Get the system calibrated, see what works for your specific chaos, then expand. Otherwise you’ll burn out before you finish your first room.

I’ve been running Home Assistant for years and never posted anything to the forums before. That should tell you how worthwhile I think this is. The system isn’t perfect, there are rough edges, and I’m absolutely open to pull requests and suggestions. But even in its current state, it’s solved a problem I’ve been avoiding for literal years.

So yeah, if you’re drowning in maker supplies, electronic components, or just the general detritus of modern life, maybe give this a shot. Worst case, you’ll have some fun playing with AI and MCP. Best case? You’ll finally know where that damn flux capacitor ended up.

2

Would this also work for dementia patients?
People with ADHD as well as OCD?
People with spouses obsessed with neatness that put everything away?

You would need the camera feed with them to observe what they relocate, and compare it to a baseline.

“Your aunt visited yesterday and tidied up your room. She put the Playboy magazine you had under your mattress in the left bottom drawer when she changed the sheets. By the way, it is last months issue - did you want me to order this months one too?”

3

Ha! The Playboy magazine scenario is perfect because it shows both the potential and the absurdity. To make that work, you’d need “under the mattress, left side” as an official Homebox location, which is… a special kind of organizational honesty.

You’re absolutely right that the current system only works if you actively register things. For dementia patients, ADHD folks, or anyone with an aggressively tidy spouse, passive observation would be transformative.

Indoor cameras 24/7? Hard pass - privacy nightmare. But outdoor cameras for workshop and garden tools? Now that’s actually viable and useful. My garage is a black hole where tools disappear, and I’d have zero privacy concerns with a camera watching my workbench.

Side note: I have both ADHD and OCD (the chaotic combo of “where did I put that” meets “it must be PRECISELY organized”), and even the manual version works well for me. The AI removing the friction of categorizing and describing things makes all the difference. But honestly, who doesn’t have at least a touch of these nowadays?

Real question: would it be creepy or convenient if your home knew your aunt reorganized your stuff?

4

Yes, your AI should know where everything belongs, not just where it currently is. This would frustrate compulsive tidy-uppers whose whole life revolves around moving things around.
In the PlayBoy scenario, under the mattress may be the correct location, and an audit would suggest a relocation to restore privacy.

Which means if somebody moves an item by more than six inches, it has effectively disappeared as it is not where you expected it to be. Would this resolve the consequential enormous mental anguish that occurs?

My neighbour has an extensive collection of screwdrivers, a lot identical, as people borrowing them from his workshop and returning them, often unasked, don’t put them back exactly where they found them, hence he has to go buy another one or drop everything to launch a search expedition to find something he is sure he has, but cannot put his finger on it at time of need. His wife laughs at him, being one of the prime offenders. We are bracing for murder one day. It shows lack of respect for his condition, which is normally a plus, as he can put his finger quickly on everything if it is not moved.

A peg board on the wall with an outline of where everything is supposed to be is one solution. Asking AI where a ‘thingy’ is, and playing back a video history to see when it was there last and where it was moved to (and most importantly by whom) is probably a big ask of current technology.

“What was moved since I went to the shop to buy the paper (and this months Playboy tucked inside)” may be a valid request…

Tool locations? Certain manufacturers now have BlueTooth tracking devices built-in on expensive tools and batteries so they can be tracked - a boon on large construction sites where tools go missing when they are plugged in to recharge. Stealing a workman’s tools is one of the lowest dog acts imaginable, and often results in instant dismissal. Having a portable CCTV recording can often be bypassed by a determined thief. Having the evidence on hand makes it far easier to dismiss a thief and blacken their reputation as the word gets around.

Incorporating that tracking into asset management and current location would require enormous technology that is probably a tad too expensive for my neighbour to deploy, and he would probably be out of prison for time for murder before he paid for the technology bill.

5

I think we’re solving different problems here. Your neighbor needs things in exact positions - I don’t. I just need to know if something exists and roughly where it probably is.

My use case is simpler: I’ve got 20 boxes of electronic components, sensors, cables, ESP32s. The question isn’t “is my screwdriver exactly 6 inches from where I left it” - it’s “do I even HAVE a JST connector?” and “which box should I check first?” Opening 5 boxes to find out wastes time. Asking Claude “do I have what I need for this project” and getting an instant answer solves my actual problem.

Video playback tracking who moved what, audit trails, pegboards with outlines - those are interesting solutions for different problems. But they’re also more complex than what’s needed for my scenario.

Here’s my take: it’s more useful to solve real problems with technology we have TODAY than to wait for the perfect system that solves everything. The current setup works well for “organized chaos in boxes” - and that’s a common enough problem to be worth sharing.

6

The neighbour sighs, and just asks his wife where everything is.

Asking Jeeves if you have a JST connector, and where it is are two different challenges. One is an inventory issue, the other a storage issue. You possibly have a problem with the first, but my neighbour has a HUUGE problem with the second. He knows he has exactly twenty seven Phillips #2 screwdrivers with a green handle, it is just he cant put his hands on one right now, as his pressing need is to tighten up a loose screw, not launch a search party to find one of them, because his wife helpfully put them in one box and put them in a cupboard somewhere, without telling him, when or where.

7

Get a label maker.

Putting labels on drawers/shelves/toolboxes/tackleboxes and such, and having the discipline to consistently put the appropriate things where they belong, is much simpler, cheaper, and better use of time/effort, than trying to wrangle an ai to keep track of it for you.

It may not solve all the problems, like other people, but that solution is simple (lock shit up).

good luck

8

Having open or transparent boxes where all the contents is visible is another option. You just need to go on the prowl every now and again to refresh your memory.
When the inventory becomes vast (stuff, not junk for hoarders - often in many cupboards, many shelves, many buildings), Jeeves should keep location and inventory information for you to preserve the seven remaining memory cells still functioning in your brain.

The most reliable way to find stuff is to move house.

9

I already have labels on my boxes.

But I’m reminded of my wife saying: “we don’t need smart switches, we can get our butt off the couch and turn on the light by pressing the switch.” But she can’t see that when arriving home, it’s much more practical to have automation that turns on the front light, garage, hallway, and kitchen so that nighttime arrival is comfortable.

My grandmother is super organized, it seems natural to her. Everything has its place, she only buys things after she’s already thought about where to put them. It seems this came from her upbringing. I, like many others, was raised by TV and the internet - I have no idea how to naturally organize a home, despite being capable of inventing solutions for first-world problems while living in the third world.

Just like every smart switch must have a physical fallback switch when something goes wrong, labels on boxes are essential. But it’s impossible to have labels that are specific enough to eliminate ambiguity and always make it obvious where each item should go.

Some practical problems that labels don’t solve:

1. Scalability: With 3D printer projects, electric skates, home automation, rocketry… You accumulate hundreds of components. Creating a labeling nomenclature system that covers all cases becomes a byzantine nightmare.

2. Temporal context: “M3 x 20mm screw” might be in 3 different boxes depending on the active project. The label “M3 screws” doesn’t tell you which box has the length you need RIGHT NOW.

3. Attribute-based search: You’re not looking for “box 23”, you’re looking for “that temperature sensor that can handle 300°C” or “that 100µF capacitor left over from the power supply project”. No label system solves this without having a catalog… which is exactly what Homebox is.

4. Remote access: If you’re not home and someone needs to get something specific - your wife, a friend helping with a project - you can simply search the system and say “blue box, third shelf from the left”. Without this, it’s a 20-minute call trying to describe “that box with green tape near the tools”.

And yes, my boxes aren’t transparent, so that tip never helped me in most cases. Plus, transparent boxes don’t help when the box is full of many small items - you still can’t see what you need without dumping everything out.

10

My neighbour would probably want your grandmother to visit his wife for a fireside chat.

1 Like
11

Have any of you tried one of the box organizing apps like ToteScan?

I came to the forums today with somewhat similar organizing questions. These apps seem like a better option than I’ve come up with so far on my own but they all appear to use cloud storage & some require subscriptions.

12

Good question about ToteScan and similar apps. Here’s my take: I want home automation and AI to reduce decision-making, not just catalog things. I need the system to help me answer questions like “what can I build with what I have?”, “what should I repair vs. replace?”, “what parts do I need to buy?”, and “what’s sitting here unused for months?”

We’re seeing this play out right now with OpenClaw (formerly Clawdbot/Moltbot). People are running it despite security concerns and nondeterministic behavior because the utility is worth the trade-off. They want real autonomy, not a locked-down “safe” experience.

Closed apps like ToteScan lock you into what the developer allows. They’re generic by design - built to fit many people, which means they won’t take risks with AI that could be incredible but also problematic. That’s a safe business decision, but it limits what’s possible.

Here’s the critical part: if you don’t control your data, you can’t give it to the AI tools you actually want to use. With Homebox + MCP + Home Assistant, I own the data and can delegate complex decisions to whatever AI model works best. Today’s models aren’t perfect, but nothing suggests they won’t improve rapidly.

Yes, it’s more complex than downloading an app. But the difference between a tool that serves the developer’s business model and one that serves your actual needs is worth the setup effort.

1 Like
13

Thank you for your thoughtful, considerate reply.
I was already thinking ToteScan etc would end up being disappointing.