Welcome to Synthiam!

Program robots using technologies created from industry experts. ARC is our free-to-use robot programming software that makes features like vision recognition, navigation and artificial intelligence easy.
Get Started

Asked
Resolved Resolved by ptp!

Filereadline Getting Error Data Is In Wrong Format

I have created a file by downloading data from the web.

this is the line of code

FileWrite($DataFile,HTTPGet($url1))

Where $DataFile is the file location on my harddrive and url1 is the where the data is on the web.

The data gets successfully wrote to the hard drive. 

Now my next line of code is $line = FileReadLine($DataFile). 

The first line of the Data file is:  <!DOCTYPE html>

This fails with the error Input string was not in a correct format.  

Now this used to work but with ARC this fails.

What format should the Datafile be in for this to continue?


Related Hardware EZ-Robot EZ-B v4
AI Support Bot
Related Content
Synthiam
#15  
[quote][/quote]I am able to read and write the data file.  But now I need to search through it and find a string.

I am using
               var linein = File.readLine(dataFile);

                var n = dow.search(linein);  (this is line 20 the error is referencing.)

but receive the following error:

Execution Error Line 20 Col 2 - Line 1: Invalid regular expression

another bit of help?
#16   — Edited
a working example:

Code:

var baseUrl = "https://synthiam.com";
var url1 = "https://synthiam.com/Community/Questions";

print("Downloading questions");
var content = Net.hTTPGet(url1);

print("Content length=" + content.length);

print("Questions found:");

var pos = 0;
do
{
//This method returns -1 if the value to search for never occurs.
pos = content.indexOf("/Community/Questions/", pos);

if (pos>=0)
{
beginPos = pos

//search for a double quote after finding the begin string
pos = content.indexOf("\"", beginPos);
if (pos>0)
{
endPos = pos;
var link = content.substring(beginPos, endPos);
print (baseUrl + link);
}
}
}
while(pos>=0);

the above code downloads Synthiam's front page questions and parses que questions links using a indexOf.

The method search uses regular expressions.

help with JS:
https://www.w3schools.com/jsref/jsref_indexof.asp


@DJ:
I believe there are some bugs with File class exported to Jint.

*** code edited ***
#18  
This is the html content:
User-inserted image


Red is the "begin text" and Blue is the end
#19  
special attention to substring method:
https://www.w3schools.com/jsref/jsref_substring.asp

Definition and Usage
The substring() method extracts the characters from a string, between two specified indices, and returns the new sub string.

This method extracts the characters in a string between "start" and "end", not including "end" itself.
Synthiam
#20  
What bugs are you noticing with the File class? And what version of ARC are you using? (latest?)
#21  
weird stuff going on, maybe is a consequence of a single bug.
I'll post simple code to  demo the issue.
#22  

Code:

function PrintLines(filename) 
{
var lineIx = 0;
while (!File.isReadEnd(filename))
{
var line = File.readLine(filename);
print(lineIx + " line:" + line);
}
}


var filename="c:\\temp\\lines.txt";

var now = new Date().toLocaleString();
if (!File.exists(filename))
{
print("file does not exist");
File.appendStringLine(filename, now + ">>" + "First time");
}
else
{
print("file exists");
File.appendStringLine(filename, now + ">>" + "foo");
}

print("reset read file");
File.readReset(filename);

print("PrintLines");
PrintLines(filename);

print("add bar");
File.appendStringLine(filename, now + ">>" + "bar");

print("PrintLines");
PrintLines(filename);

first time (file does not exist)
User-inserted image


Close ARC, Open ARC run the script:
User-inserted image


Q: without a close and explicit open how you control reads and writes ?

Code:

File.appendStringLine
Locks the file, and you need to close ARC.
#23  
Reading from a File locks the file too so you need to close ARC.
User-inserted image
#24  
PTP

In your answer in #16.  I think I follow but what I cant figure out is why doesn't your code download all the questions.  Why does it stop after 9.

I don't see anything in your code that says just show the first 9 questions.
#25   — Edited
That's a different question, the website content is dynamic, does not make sense to list all the questions when a page is loaded. 
Look to twitter when you scroll down to the last visible post, the page requests more content from the server same thing here.

synthiam page captures the document page scroll:

Code:

$(document).ready(function() {
if ($("#sidebarsearchsort").change(function() {
endOfResults = !1;
replaceSearchContent()
}),
$("#search-result").length)
$(document).on("scroll ready", function() {
element_in_scroll("#search-result .content-card:last-child") && (loading || addSearchContent())
})
});
and calls the function addSearchContent

Code:

function addSearchContent() {
searchContent(function(n) {
$("#search-result").append(n)
})
}


function searchContent(n) {
if (endOfResults !== !0 && loading !== !0) {
var t = typeof compact == "undefined" ? !1 : compact
, i = typeof subCat == "undefined" ? null : subCat
, r = typeof noLabel == "undefined" ? !1 : noLabel
, u = typeof sort == "undefined" ? $("#sidebarsearchsort").val() : sort
, f = typeof exSubCat == "undefined" ? null : exSubCat
, e = typeof searchQ == "undefined" ? null : searchQ;
loading = !0;
addLoadingCards(t);
$.ajax({
method: "POST",
data: {
__RequestVerificationToken: $("[name='__RequestVerificationToken']").val(),
pageNumber: pageNumber,
sort: u,
type: cat,
referenceId: $("#selected-behavior-control option:selected").val(),
compactView: t,
subCat: i,
noLabel: r,
excludeSubCat: f,
searchTerm: e
},
url: "/api/search/byType"
}).done(function(t) {
if (loading = !1,
removeLoadingCards(),
t === "") {
endOfResults = !0;
return
}
n(t);
pageNumber = pageNumber + 1;
var i = (JSON.stringify(t).match(/class=\\"content-card |class=\\"content-card\\"/g) || []).length;
i < 9 && (endOfResults = !0)
})
}
}
the function executes an ajax call to the server /api/search/byType and the server returns more html content (i.e. questions) that content will be appended to a specific document placeholder:

Code:

$("#search-result").append(n)

short story when you request the questions page a fixed number of questions are returned, after each scroll  more questions are retrieved from the server.

you can read more about "infinite scroll":
http://harshw.com/infinite-scroll-using-jquery-asp-net-mvc-mustache-entity-framework-paginate/
#26  
You lost me at infinite scroll.

But I do understand this (I think) the code only reads a page at a time.  It will take me some time to digest the above.

one last question for the day.  BTW.  I have it working now and getting the data from the page thanks to your help.

Here is the question.    this is what the data looks like:    >5:26 PM</td>   The data I want is the 5:26 PM.  But depending on the time of day it could be 10:26 PM.  Is there a one-liner function that can pull the 5:26 PM or 10:26 PM out from between those markups really easy without including the other stuff?
Synthiam
#27   — Edited
If the file is locked, you have to close the file from reading with the close command. Because reading the file keeps the file open while you read it. 

Code:


File.readClose( filename );
#28  
I missed that method, but i don't see an equivalent writeClose
Synthiam
#29  
Because there is no write. It's append. Appending just keeps appending to an existing file or creates a new one. There's no reason to keep a file open for appending. The only point to keeping the file open for reading is to know the position.

I've added two new methods for getting the current read position and getting the file length. It'll be in today's release.
#30  

Code:

File.appendStringLine
This commands locks the file and does not release the file. Is that an expected behavior ?
#31   — Edited
@dbeard:

User-inserted image


Example to capture the red part 

The forum break's the JavaScript code, posted code picture:

User-inserted image

 
@DJ 
Please read my previous post!
#32  
Here is my program for getting dow jones data.  I know it is a mess, but it works.

Code:

var url1 = "http://www.marketwatch.com/tools/marketsummary/indices/indices.asp?indexid=1&groupid=37";

print("Downloading Data");
var content = Net.hTTPGet(url1);

print("Content length=" + content.length);

print("Dow Jones Information Found:");

pos = 0;
do
{
//This method returns -1 if the value to search for never occurs.
var pos = content.indexOf("/investing/index/djia", pos);

if (pos>=0)
{
beginPos = pos

//search for a double quote after finding the begin string
pos = content.indexOf("\"", beginPos);
if (pos>0)
{
endPos = pos;
var link = content.substring(beginPos, endPos);
var volume = content.substring(endPos+144, endPos+153);
var tradedate = content.substring(endPos+193, endPos+205);
var tradetime = content.substring(endPos+245, endPos+255);
var amount = content.substring(endPos+297, endPos+309);
var tradepercent = content.substring(endPos+352, endPos+361);

place = 0
var startplace = volume.indexOf(">",place);
var endplace = volume.indexOf("<", place);
var fvolume = volume.substr(startplace+1, endplace-1);

place = 0
var startplace = tradedate.indexOf(">",place);
var endplace = tradedate.indexOf("<", place);
var ftdate = tradedate.substr(startplace+1, endplace-1);

place = 0
var startplace = tradetime.indexOf(">",place);
var endplace = tradetime.indexOf("<", place);
var fttime = tradetime.substr(startplace+1, endplace-1);

place = 0
var startplace = amount.indexOf(">",place);
var endplace = amount.indexOf("<", place);
var famount = amount.substr(startplace+1, endplace-1);

place = 0
var startplace = tradepercent.indexOf(">",place);
var endplace = tradepercent.indexOf("<", place);
var ftpercent = tradepercent.substr(startplace+1, endplace-1);

print("Date: " +ftdate);
print("Time: " +fttime);
print("Volume: "+fvolume);
print("Amount: "+famount);
print("Percent: "+ftpercent);

halt();
}
}
}
while(pos>=0);
#34  
Thanks everyone for the help.