bradleyheathhays
Hey, can't say how grateful I am to have found this forum! Although I'm not from the robotics community, I am a crafter of sorts and have an idea for a little device I'd like to build, and I need your advice on where to go from here.
My task is fairly simple. I work in an industry that uses 6-8" computer tablets, and when a part of the screen changes color to indicate an available 'job,' whoever hits the on-screen 'button' first gets the job, thusly increasing their income. Basically, I just need to create a device that senses when that particular part of the screen changes color, and then automatically responds with some type of actuator (I assume) to make contact with the screen using a stylus device.
Sounds simple? I hope so!
I know this may take some work, but I'm def willing to make an investment in time and effort.
Without going into further detail I should stop short and simply ask...is this the place I need to be to begin developing such an idea? Also, I'd like to try to develop this idea simultaneously at as many locations as possible here on the net; so if there's another place besides here that may be of help I would surely appreciate some direction.
Looking forward to your ideas and questions!
Well, ideally an API would exist for your application so you could create an interface to send a "button click" event. But I'm going to assume no such interface exists and thus you'd like to replicate the actions of a human with a robot on this touch screen tablet?
I'd recommend the EZ-Robot Dev Kit. You could use the camera to look at your tablet and use the object feature under the camera control to train the camera to look for a picture that matches you color on the screen, then use a servo or two from the kit to make an arm that lowers a stylus to the tablet to replicate pressing the button. You'll make a rig to hold all this together but the DEV kit would give you the primary parts.
One think to consider is keeping your tablet signed in and awake, you could not want it to go sleep or shut the screen off otherwise the robot would not be able to see what was going on.
Or for a cooler design, get a JD robot and have him hold a stylus...to hit the button on the screen. You'd have to create an action to move the servo arms down to a preset location to so you'd need to make sure your tablet and stylus don't move. Hopefully that idea makes sense. The ARC software will make it pretty easy to create your robot button masher.
Alan
1) I'm not entirely sure what an API is but yes I do want to replicate the actions of a human.
2) The EZ Robot Dev Kit sounds like it's got a lot of what I'd be looking for. The small part of the screen the optical sensor would need to be trained to is only about 1" x 1". Ideally the system I'd like to design would be programmed so that it need not necessarily recognize any type of picture, but would just activate the stylus when it sensed any kind change on the screen at all. The way my tablet works is that the only change this part of the screen will ever make indicates a 'button' needs to be pushed. The optical sensor 'recognizer' and interfacing it with a servo is the part I need help with...the building of a mounting apparatus or ring as you mentioned is something I should be able to handle ok.
3) Thanks for the advice on the tablet going to sleep. With my setup the tablet is always active so should be no problem there.
4) Your JD robot suggestion sounds interesting. However I can get this to work would be great, although keeping things compact is def a priority. Glad to hear there's more than one idea so far though.
Thank you too techguru.
5) The operational environment for this little gadget will be in a vehicle so I will have a continuous 12v power supply. Would this be appropriate for powering 1-2 servos?
What kind of App ? Is an Android tablet, Apple IPAD, Microsoft mobile ?
Alan
so that door is closed.
DETECTION PROBLEM
you will need to monitor the screen, and you have a few options (simple to complex) :
1) Color Sensor
If is a basic detection e.g. specific spot color, red or green square there are cheap color sensors:
https://www.adafruit.com/product/1334
you can interface them with EZB controller via I2C
but you will need to build a closed opaque case, leaving open only the area where you want to monitor the color, to maximize the contrast and help the sensor to detect the color. The sensor needs to be higher than touch sensor.
2) Video camera
you will need to take in consideration the location camera, light conditions, tablet glass reflection, and if you want to build a mobile solution (you mention car) you will need to build a custom cage to hold everything, then you need to train the software to detect the specific condition (color, shape, etc).
3) Custom Software
If you have an Android tablet, and you have full control over it, you can install third party application to capture the screen. You will have a perfect screenshot to process the image. But you will need to develop a custom application. If you are not familiar with software development, and image processing this is not an option.
recent touch screens are capacitive touch screen not force or pressure touch screens, this means:
That means you need to emulate very low electrical voltage, there are sensors to do that, but you can easy do one using a floating digital pin check this video:
https://www.youtube.com/watch?v=JDgDMBquBw0
or you can use hardware:
Servo:
http://www.ez-robot.com/Shop/AccessoriesDetails.aspx?prevCat=9&productNumber=13
Linear Actuator:
http://www.robotshop.com/en/firgelli-technologies-l12-actuator-30mm-210-1-12v-position-controller.html
both will need a capacitive touching pen
if you want to learn robotics, the EZB controller is the right tool, no wires (wifi) easy to connect, user friendly software ARC and a nice community.
EZB Controller requires a PC or a mobile tablet to run the software logic.
If you want to build a mobile solution (car) you will need a tablet or smartphone always on running your application.
this can be overkill if you want to build a portable solution, always on, minimal user interaction.
Couple things...
SENSING
The simplest detection method, the color sensor, seems like it would work in my case. For the most part it's just as you mentioned...red or green square. The 1"x1" screen portion that needs to be monitored is a dedicated 'button' section of the screen that has a symbol in the middle, which is a different color than the rest of the button. When that entire area changes color and alerts, both the symbol and the non-symbol portion of the button change color, significantly enough that I would think a sensor could be calibrated to alert to the change.
Building a closed opaque case sounds entirely doable.
TOUCH
Thankfully the screen reacts to both human and stylus touch. I'm thinking I can use the stylus that came with the tablet and integrate it with the servo/actuator part of the plan. But thanks for thinking ahead to that potential issue.
CONTROLLER
I'll go whatever direction you suggest. The idea of learning robotics is appealing as I'm pretty much a tinkerer anyway. Controlling the device via wifi through a mobile pad sounds fairly practical. But I'm not entirely sure what you may have meant by overkill...maybe that all these things are more than I was expecting? If I have to go to some lengths to get this done I'm ok with that. In the end I think the effort will pay off.
Sorry I missed your post earlier, but, any more thoughts after my update here?
My last post had also an advise, to avoid wrong expectations.
some people think EZB is autonomous, so i raised the tablet/pc requirement.
you mentioned a mobile device (car), so i raised you will need a controlled/stable solution to be able to take with you and removed when needed.
finally i mentioned user interaction: stop, disconnect, connect, load application, start, etc.
Is doable, you will need to get familiar with the controller, software, sensors, actuators etc.
Is there anywhere else I should go here on the net to look for help developing this idea?
If everything is new to you, the Learn section is the best place to start.
then you can try to search the forum for previous posts or issues.
you will need to get the EZR hardware, you can go "a la carte" or go with the developer kit (a good bundle)
other topics:
I2C sensor you will need to know: how it works, how do you connect to EZB, and how you control (EZ-Script).
if you plan to use a servo, you will need a DC/DC converters from 12V to 7v one example:
https://www.amazon.com/gp/product/B00RSOOBYC/
I can't recommend any software to do this for a tablet, I would only be able to help with a PC version of the tablet app. However, I would recommend going to a tablet software forum and asking your question there.
Just a thought