I use a software development system called "Delphi." It's quite popular in Europe but not as much in the US. Michelin USA uses it. I know because I introduced them to an early version of it while I worked for them in their central engineering dept. Last I heard they were using more up to date versions still. It's kind of like the Microsoft development systems, but it is not married to any one platform like the Microsoft stuff is. When I left Michelin and struck out on my own, I bought the Enterprise version to do independent software development. An investment of several thousand dollars. Ironically, one of my first jobs as a contractor was for "The Children's Place" chain of stores, using the very early version of the system again. They used it as their Point of Sales software and I was able to keep it running with new devices for a few more years.

Ok, that's more than you ever wanted to know, I'm sure. Suffice it to say, it's a rapid prototyping development system which allows me to develop programs fairly quickly. Though this one is taking a while because I can't devote as much time to it as I would like. Still, I expect to have it ready in a couple of days, along with something with the Yahoo site.

I've managed to come up with a method that will allow the ARC script to find out where the HTMLScrub program is and use that information to get to the HTMLScrub.ini file to find out the rest of what it needs to know to run in conjunction with it. There will be no need to enter the file path to the HTMLScrub program manually in the scripts. So the Scrub program can be placed anywhere at any time and the script will find it. That also allows strings to be stored in the HTMLScrub.ini file that can't be used in the scripts.

Additionally, the program now has a Setup and an Error mode. It can be called using a script so as to bring it up in Setup Mode to make changes to the options and the other things in it. In other words you don't have to access the HTMLScrub.ini file directly to make changes and additions to it.

The Error mode will halt the program if there is an error in the scrubbing process so that you can see what the problem is. That's optional, however.

Finally, I've managed to have it do it's thing without it popping up to be seen in the process as it does now. It still pops up but it is invisible so there will be no indication anything even happened. That is also optional but I think it will be best to leave it in invisible mode. Even if it were visible, it goes so fast now, even on large files, it's there and gone in a moment.

I look forward to testing out the new HTMLScrub file. I checked out Delphi, I think I will keep that for my phase 2 learning. Thanks.

Sorry it's taking so long for the update. I ran into a couple of snags. Overcame them, but then I decided to separate out the common parts of the code from the 3 websites I've been working with. Not too big a deal right now, but, as more and more sites of various sorts are added it will be best for the long run. In doing so, I've had to introduce the concepts of CommandControl being used to make "function calls" and the use of init scripts. The latter is probably something you are already familiar with at least.

In any event, the basic ideas are still the same and the code to doing it hasn't really changed. Just that what was done with local Gotos before is now being done with CommandControl instructions. I've made a project with it all in it and that is what I will upload to the cloud. Plus, of course, the usual update of the scrub program attached in the usual way.

Since the scrub program is a bit more involved now I will be sending the next version in an installer to simplify the process of putting it all in. I've put all the files I test with in the same directory as you use so you won't have to make any changes to the settings in that regard in the future. All in all, it should be pretty much plug and play.

How do you plan on handling different web sites, different data? What I have been doing is placing the scrub file in different directories and updating the ini with that data. Works great, have it working across a few sites. I used the script you uploaded and just modified it to handle the different sites, path names, and the varying data. No hurry on the update, I am using the other and learning as I go.

also, I cant find your HTML script in the cloud anymore. Did you move it or rename it? Need a clean copy so I can test out some things, I modified the one I originally downloaded.

I'm not quite sure what you are doing. From what I read it seems like you are placing a copy of the HTMLScrub.exe file and the HTMLScrub.ini in different directories. Then modifying the ini file to suit the needs for each website. Then using one script to access them all, calling each from it's own directory. Is that it? If not, please clarify.

What I'm doing basically is using a different script for each site, but always the same HTMLScrub.exe file and the HTMLScrub.ini files in the same place. All the website scripts are placed in one Script Manager I call "Main". If I want to call them from yet another script, I would simply call the various other website scripts via CommandControl instructions. Such as:

CC($Main,StartScriptWait,$Yahoo_SMB)
CC($Main,StartScriptWait,$Yahoo_MU)
CC($Main,StartScriptWait,$Marketwatch)

Each website script has different needs as far as getting the appropriate data is concerned and has a few lines at the beginning to determine which site to scrub.

Each Website script also sends a name to the scrub program indicating which site it is to scrub. This is done via a file called XFer.cfg. The scrub program uses that name to determine how to handle the raw file generated by the script concerning Starting Position and how much of the raw file to scrub. All that information for each website is in the one ini file.

I'm going on the assumption that fresh data from the website is desired every time it is accessed so there is no need to store data on a per-website basis. However, that could be done using the name sent in the XFer.cfg file. Currently, all the files related to the operation will be stored in the same directory as the Scrub program. They will also have more generic names now as well. The raw data from the website generated by the script by the HTTPGet command is called, appropriately enough, RawData.txt. and the scrubbed file, ModData.txt. instead of the VZ names since that is just a holdover from the past. You can even call them whatever you wish though, since the initialization script for the Main script manager gets those names from the ini file now. They no longer need be declared explicitly in the scripts. Nor do the paths to them (or anything else). That's all handled automatically.

Just saw your latest post. I'll put the old project back up for you to download. I took it down because it is outdated now.

@WBS, your third party apps are neat - have you considered creating plugins instead? It's much easier than you think, and since microsoft made .Net free, it doesn't cost you anything. I also publish all my plugins open-source which you can view for additional examples.

Here's a tutorial on step by step to create a plugin: https://synthiam.com/Tutorials/UserTutorials/146/1

@DJSures

Thank you for the kudos. I finally took a few to really look at the tutorial you suggested and found it to be very well detailed. Looks like I don't have to know all about C# to at least try the steps listed and see how it all comes together for myself. I've had the Community version for some months now and played around with it some, but only using VB. No doubt I can use C# as well, still the same components and the like after all. And an algorithm is an algorithm regardless of the language.

Anyway I'll give it a go when I finish my latest round of software stuff I'm doing now. Still, I wonder if I could twist the interface you have for Microsoft .NET and make it work with Delphi? A DLL is a DLL also, after all. Just need to get the calling conventions right. I've used C++ DLLs in my Delphi code from time to time. The newer versions can even work with .NET as well. Might look into that too.