Welcome to the forum.

Well, ideally an API would exist for your application so you could create an interface to send a "button click" event. But I'm going to assume no such interface exists and thus you'd like to replicate the actions of a human with a robot on this touch screen tablet?

I'd recommend the EZ-Robot Dev Kit. You could use the camera to look at your tablet and use the object feature under the camera control to train the camera to look for a picture that matches you color on the screen, then use a servo or two from the kit to make an arm that lowers a stylus to the tablet to replicate pressing the button. You'll make a rig to hold all this together but the DEV kit would give you the primary parts.

One think to consider is keeping your tablet signed in and awake, you could not want it to go sleep or shut the screen off otherwise the robot would not be able to see what was going on.

Or for a cooler design, get a JD robot and have him hold a stylus...to hit the button on the screen. You'd have to create an action to move the servo arms down to a preset location to so you'd need to make sure your tablet and stylus don't move. Hopefully that idea makes sense. The ARC software will make it pretty easy to create your robot button masher.

You will also need to consider power. A battery will only last so long, but if you are using servos you need a power supply with plenty of amps, and that could easily be the most expensive part if the project (a wall wart power adapter is not going to be sufficient even for one or two servos).

Alan

Hi Justin. Thanks for the direction!

1. I'm not entirely sure what an API is but yes I do want to replicate the actions of a human.

2. The EZ Robot Dev Kit sounds like it's got a lot of what I'd be looking for. The small part of the screen the optical sensor would need to be trained to is only about 1" x 1". Ideally the system I'd like to design would be programmed so that it need not necessarily recognize any type of picture, but would just activate the stylus when it sensed any kind change on the screen at all. The way my tablet works is that the only change this part of the screen will ever make indicates a 'button' needs to be pushed. The optical sensor 'recognizer' and interfacing it with a servo is the part I need help with...the building of a mounting apparatus or ring as you mentioned is something I should be able to handle ok.

3. Thanks for the advice on the tablet going to sleep. With my setup the tablet is always active so should be no problem there.

4. Your JD robot suggestion sounds interesting. However I can get this to work would be great, although keeping things compact is def a priority. Glad to hear there's more than one idea so far though.

Thank you too techguru.

5. The operational environment for this little gadget will be in a vehicle so I will have a continuous 12v power supply. Would this be appropriate for powering 1-2 servos?

Bradley,

What kind of App ? Is an Android tablet, Apple IPAD, Microsoft mobile ?

12v is fine for the EZ-B but too high for the servos (7.4v optimal), but a voltage regulator is much cheaper than a power supply.

Alan

ptp that's a good question. The tablet is very industry specific and is not available to the general public. Not sure about what kind of software it uses.

Quote:

The tablet is very industry specific and is not available to the general public.

so that door is closed.

DETECTION PROBLEM

you will need to monitor the screen, and you have a few options (simple to complex) :

1. Color Sensor If is a basic detection e.g. specific spot color, red or green square there are cheap color sensors:

https://www.adafruit.com/product/1334

you can interface them with EZB controller via I2C but you will need to build a closed opaque case, leaving open only the area where you want to monitor the color, to maximize the contrast and help the sensor to detect the color. The sensor needs to be higher than touch sensor.

2. Video camera you will need to take in consideration the location camera, light conditions, tablet glass reflection, and if you want to build a mobile solution (you mention car) you will need to build a custom cage to hold everything, then you need to train the software to detect the specific condition (color, shape, etc).

3. Custom Software If you have an Android tablet, and you have full control over it, you can install third party application to capture the screen. You will have a perfect screenshot to process the image. But you will need to develop a custom application. If you are not familiar with software development, and image processing this is not an option.

TOUCH PROBLEM

recent touch screens are capacitive touch screen not force or pressure touch screens, this means:

Quote:

Capacitive touchscreen displays rely on the electrical properties of the human body to detect when and where on a display the user touches. Because of this, capacitive displays can be controlled with very light touches of a finger and generally cannot be used with a mechanical stylus or a gloved hand.

That means you need to emulate very low electrical voltage, there are sensors to do that, but you can easy do one using a floating digital pin check this video:

https://www.youtube.com/watch?v=JDgDMBquBw0

or you can use hardware:

Servo:

https://synthiam.com/Shop/AccessoriesDetails.aspx?prevCat=9&productNumber=13

Linear Actuator:

http://www.robotshop.com/en/firgelli-technologies-l12-actuator-30mm-210-1-12v-position-controller.html

both will need a capacitive touching pen

CONTROLLER

if you want to learn robotics, the EZB controller is the right tool, no wires (wifi) easy to connect, user friendly software ARC and a nice community.

EZB Controller requires a PC or a mobile tablet to run the software logic.

If you want to build a mobile solution (car) you will need a tablet or smartphone always on running your application.

this can be overkill if you want to build a portable solution, always on, minimal user interaction.

Again thanks ptp! So glad this is looking doable.

Couple things...

SENSING The simplest detection method, the color sensor, seems like it would work in my case. For the most part it's just as you mentioned...red or green square. The 1"x1" screen portion that needs to be monitored is a dedicated 'button' section of the screen that has a symbol in the middle, which is a different color than the rest of the button. When that entire area changes color and alerts, both the symbol and the non-symbol portion of the button change color, significantly enough that I would think a sensor could be calibrated to alert to the change.

Building a closed opaque case sounds entirely doable.

TOUCH Thankfully the screen reacts to both human and stylus touch. I'm thinking I can use the stylus that came with the tablet and integrate it with the servo/actuator part of the plan. But thanks for thinking ahead to that potential issue.

CONTROLLER I'll go whatever direction you suggest. The idea of learning robotics is appealing as I'm pretty much a tinkerer anyway. Controlling the device via wifi through a mobile pad sounds fairly practical. But I'm not entirely sure what you may have meant by overkill...maybe that all these things are more than I was expecting? If I have to go to some lengths to get this done I'm ok with that. In the end I think the effort will pay off.

Sorry I missed your post earlier, but, any more thoughts after my update here?

No more updates, i tried to cover all the options i know.

My last post had also an advise, to avoid wrong expectations.

some people think EZB is autonomous, so i raised the tablet/pc requirement.

you mentioned a mobile device (car), so i raised you will need a controlled/stable solution to be able to take with you and removed when needed.

finally i mentioned user interaction: stop, disconnect, connect, load application, start, etc.

Is doable, you will need to get familiar with the controller, software, sensors, actuators etc.

OK thanks for the push in the right direction. I'll start watching the tutorial videos you linked to. Will those videos be all I need to get familiar with the controller, software, sensors, etc you mentioned?

Is there anywhere else I should go here on the net to look for help developing this idea?

The only video link i sent is a demo of the capacitive touch control. I believe can be replicated also with an EZB i never tried, but the principle is toggle one digital pin on to off. If does not work you will need to find a tiny sensor or you can use a servo.

If everything is new to you, the Learn section is the best place to start.

then you can try to search the forum for previous posts or issues.

you will need to get the EZR hardware, you can go "a la carte" or go with the developer kit (a good bundle)

other topics:

I2C sensor you will need to know: how it works, how do you connect to EZB, and how you control (EZ-Script).

if you plan to use a servo, you will need a DC/DC converters from 12V to 7v one example: https://www.amazon.com/gp/product/B00RSOOBYC/

Seems to me that a software solution would be better. I'm not into programming for tablets, however I am into programming for PCs and Mainframes. And programming is programming, regardless of the platform. There should be methods available that would "look at" (monitor) a section of the screen and simulate a click on that section when it changes color. Such a scheme would be much easier to implement and may cost nothing using free tablet software. With luck, you could probably get someone to do it for you. Generally it should be a trivial thing to do for someone knowledgeable in some sort of tablet programming language. It would be in a PC language, at any rate. Plus it would be usable anywhere with no attachments required.

I can't recommend any software to do this for a tablet, I would only be able to help with a PC version of the tablet app. However, I would recommend going to a tablet software forum and asking your question there.

Wouldn't it be easier just to keep tapping the screen continually? That way you don't need Color detection or anything too fancy... As soon as the button became active and green it would register on the next tap.

Just a thought :)

ARC has a plug-in for mouse control. It can mouse click on any spot on the screen through a ez script. Of course you'll have to be running ARC on a windows computer or tablet.

I think you can accomplish this in a simpler way by using a sensor, a relay and a solenoid, no need to use a controller, servo, etc. As you only need to do one movement.