bradleyheathhays
Hey, can't say how grateful I am to have found this forum! Although I'm not from the robotics community, I am a crafter of sorts and have an idea for a little device I'd like to build, and I need your advice on where to go from here.
My task is fairly simple. I work in an industry that uses 6-8" computer tablets, and when a part of the screen changes color to indicate an available 'job,' whoever hits the on-screen 'button' first gets the job, thusly increasing their income. Basically, I just need to create a device that senses when that particular part of the screen changes color, and then automatically responds with some type of actuator (I assume) to make contact with the screen using a stylus device.
Sounds simple? I hope so!
I know this may take some work, but I'm def willing to make an investment in time and effort.
Without going into further detail I should stop short and simply ask...is this the place I need to be to begin developing such an idea? Also, I'd like to try to develop this idea simultaneously at as many locations as possible here on the net; so if there's another place besides here that may be of help I would surely appreciate some direction.
Looking forward to your ideas and questions!
Welcome to the forum.
Well, ideally an API would exist for your application so you could create an interface to send a "button click" event. But I'm going to assume no such interface exists and thus you'd like to replicate the actions of a human with a robot on this touch screen tablet?
I'd recommend the EZ-Robot Dev Kit. You could use the camera to look at your tablet and use the object feature under the camera control to train the camera to look for a picture that matches you color on the screen, then use a servo or two from the kit to make an arm that lowers a stylus to the tablet to replicate pressing the button. You'll make a rig to hold all this together but the DEV kit would give you the primary parts.
One think to consider is keeping your tablet signed in and awake, you could not want it to go sleep or shut the screen off otherwise the robot would not be able to see what was going on.
Or for a cooler design, get a JD robot and have him hold a stylus...to hit the button on the screen. You'd have to create an action to move the servo arms down to a preset location to so you'd need to make sure your tablet and stylus don't move. Hopefully that idea makes sense. The ARC software will make it pretty easy to create your robot button masher.
You will also need to consider power. A battery will only last so long, but if you are using servos you need a power supply with plenty of amps, and that could easily be the most expensive part if the project (a wall wart power adapter is not going to be sufficient even for one or two servos).
Alan
Hi Justin. Thanks for the direction!
I'm not entirely sure what an API is but yes I do want to replicate the actions of a human.
The EZ Robot Dev Kit sounds like it's got a lot of what I'd be looking for. The small part of the screen the optical sensor would need to be trained to is only about 1" x 1". Ideally the system I'd like to design would be programmed so that it need not necessarily recognize any type of picture, but would just activate the stylus when it sensed any kind change on the screen at all. The way my tablet works is that the only change this part of the screen will ever make indicates a 'button' needs to be pushed. The optical sensor 'recognizer' and interfacing it with a servo is the part I need help with...the building of a mounting apparatus or ring as you mentioned is something I should be able to handle ok.
Thanks for the advice on the tablet going to sleep. With my setup the tablet is always active so should be no problem there.
Your JD robot suggestion sounds interesting. However I can get this to work would be great, although keeping things compact is def a priority. Glad to hear there's more than one idea so far though.
Thank you too techguru.
Bradley,
What kind of App ? Is an Android tablet, Apple IPAD, Microsoft mobile ?
12v is fine for the EZ-B but too high for the servos (7.4v optimal), but a voltage regulator is much cheaper than a power supply.
Alan
ptp that's a good question. The tablet is very industry specific and is not available to the general public. Not sure about what kind of software it uses.
so that door is closed.
DETECTION PROBLEM
you will need to monitor the screen, and you have a few options (simple to complex) :
https://www.adafruit.com/product/1334
you can interface them with EZB controller via I2C but you will need to build a closed opaque case, leaving open only the area where you want to monitor the color, to maximize the contrast and help the sensor to detect the color. The sensor needs to be higher than touch sensor.
Video camera you will need to take in consideration the location camera, light conditions, tablet glass reflection, and if you want to build a mobile solution (you mention car) you will need to build a custom cage to hold everything, then you need to train the software to detect the specific condition (color, shape, etc).
Custom Software If you have an Android tablet, and you have full control over it, you can install third party application to capture the screen. You will have a perfect screenshot to process the image. But you will need to develop a custom application. If you are not familiar with software development, and image processing this is not an option.
TOUCH PROBLEM
recent touch screens are capacitive touch screen not force or pressure touch screens, this means:
That means you need to emulate very low electrical voltage, there are sensors to do that, but you can easy do one using a floating digital pin check this video:
https://www.youtube.com/watch?v=JDgDMBquBw0
or you can use hardware:
Servo:
https://synthiam.com/Shop/AccessoriesDetails.aspx?prevCat=9&productNumber=13
Linear Actuator:
http://www.robotshop.com/en/firgelli-technologies-l12-actuator-30mm-210-1-12v-position-controller.html
both will need a capacitive touching pen