I read a post in LinkedIn this week about a Covid document reader (built using AI capabilities) which scanned information from paper documents and sent what it found to a database for measurement and processing.
It got me thinking:
- Is it possible to quickly create a Power App which takes the pain out of a manual process like inventory checking or stock-taking? Something where it's not just a document but an item located on a shelf?
- How could I do this using the AI capabilities of Microsoft 365?
- Would I need special skills or insight?
In this post, I'll take you through what I discovered. Spoiler alert, it genuinely blew my mind....it's also made me sad in equal measures.
Do You Need Experience with AI
I know nothing about AI. That's not strictly true. I know a little about the 'concept' of AI. I understand the idea of rules being generated based on data but I know very little about the complexities of the algorithms involved. If I were left to my own devices, I wouldn't know how to get started building something with Artificial Intelligence.
In terms of Microsoft 365, I'm possibly a core customer. A classic use case of knowing about 'some' of the opportunities within Microsoft 365 but curious how far a Citizen Developer can go with a moderate to low level of development experience.
The Good
It took me about five minutes. In that amount of time, I'd figured out that the answer to the first question - "is it possible to create an intelligent app to recognise objects" - is a big YES! Not only that, it's incredibly simple and took me about 15-minutes to build.
Full disclosure. Up to this point, my fingers had never touched anything remotely resembling AI. So the 15 minute claim is a genuine one.
There isn't much you need to get started quickly:
- Some Images
- An AI Model
- A Power App
The Model is the key element. This is what you train to recognise objects and what you interact with via the Power App.
Creating one couldn't be simpler. If you are building a Power App, head to the AI Builder Menu Item. You can find the same in Power Automate if you want it.
Click Build and you'll head into a Wizard to step through the process (more on that in a minute).
The Bad
Before we dive into a build, it's time for a HUGE warning. Whilst the idea might be great, it's not cheap.
Having blown my mind with a whole new world of potential ideas, the realisation that at a minimum it would cost $500 per month!!! got me a bit disheartened ...
You do get a free trial of the premium connector needed for this (AI Builder) and get to see how it will work before making any more decisions.
There is also the potential to research your use case further. Head over to the AI Builder Calculator page to get a quick estimate on what your outlay could be.
I imagined an example like this:
- For a stock-take of 1000 items in my small retail outlet, I'd take the recommended 50 photographs per item to train the Model (the minimum to train the model for 1 item is 15 images in different positions and ideally locations/lighting).
- I imagined I would only need to Train once per month. Then every other image taken is compared against the model. So on an average month, let's say I'm taking and processing 60000 images overall and training once.
- Plugging this into the calculator, my likely outlay would be $1000 as I would need 2 AI builder add on units at $500 each.
Stepping into the shoes of a small shop or warehouse owner. That just doesn't seem worth it.
Working in an auction house however, selling cars or other items at volume, once a model is trained to recognize different objects, the ongoing catalogue and metadata processing could be made vastly more efficient. That could make this kind of figure a reasonable ROI.
Especially given the actual App creation is so simple!
The Not So Ugly
The decision to use what I'm about to demonstrate is one to consider carefully.
Taking account of the efficiencies you could achieve by simply snapping a picture on your phone and having complex cataloging, counting and meta data setup activities all happen for you, it may be worth it for your business.
Add to that how ridiculously simple the build is and there may be a compelling case. Here's how you'd get started:
The Build
Step 1 - You need lots of images
As mentioned, the minimum required for object recognition AI is 15 images. Ideally, you will take these of the same object in different settings and lighting conditions.
Microsoft recommends you take upwards of 50 images per item. With a laptop or desktop, this can be cumbersome so I created the following process to help me.
- Install the Office Lens app on your phone
- Sign in to Microsoft 365 and OneDrive on your PC
- Use Office Lens to take and upload a Photo
- For each photo you take, save to OneDrive. This allows you to easily import them via a desktop and train the AI without connecting to a phone or downloading them.
- Take lots of pictures to get ready for Step 2
Step 2 - Train a Model
- Head to make.powerapps.com
- On the left-hand navigation menu, choose AI Builder
- Choose Build to create your first AI model
- I chose the Object Detection Model from the list of options
- Give your model a name e.g. "Stock Count"
Once you start a model, you can start to provide it with data about the objects you wish to recognise. I chose "Placemat" as something to hand to test the theory. Choose any object in your house or office to test with.
- Create an object within the model - give it a name that you use when tagging images.
- Secondly add images that represent that object (not stock images, the real thing...warts and all!)
- Finally, tag your images
- Do this by drawing a tight box around the image using the mouse and selecting the object tag which matches the object e.g in my case "Placemat"
Keep going until you have tagged between 15 and 50+ images for each item.
When complete, the Next button becomes enabled and you can click it to begin training your Model.
Depending on the volume of items in your model and the number of images, this may take a while. It's worth noting that for each item in your model, take a similar volume of images. This will result in a better model training result.
Pro Tip
Natural Images Work Best
Don't create artificial lighting for your images. Photos of items in their natural setting and positions works much better to train your model.
Step 3 - Use the Model In Your App
This last step couldn't be easier.
- Create or load up any Canvas app.
- Choose a Screen where you want the Object recognition to operate
- In the Insert menu, choose AI Builder and then the Object Detector Control
When the control is added to the screen, use the Properties Pane to choose the model you just created (in my case "Stock Count". You have various options but I left everything as is for my test.
Now Save, Publish and Share Your Power app.
I have the Power Apps mobile app installed so to test it, I only needed to open the app on my phone, fire it up and then when I pressed "Detect" on the control, my phone camera opened. I snapped a picture of another table mat and hey presto! the image on screen was recognised and tagged as a "Placemat". At this point, its possible to count, assign meta data and perform may automated tasks to speed up processes.
It really was that easy!
The more items you train in the model and the higher the volume of images, the better the model will be at recognising your objects.
What I had learned was that with some quite simple steps, you've got a very quick and efficient way of counting hundreds of hundreds of items using your phone.
Quality-wise, I cant vouch for results over time but It's certainly been an interesting experiment to try.
If you are interested or currently experimenting in this area, I'd love to hear from you and learn more about where the AI Model builder is being used in the wild. Let me know about your experiences good or bad! (Drop a comment or find me on social).
Thanks for now.
