FileReadLine getting error data is in wrong format

Question

I have created a file by downloading data from the web.

this is the line of code

FileWrite($DataFile,HTTPGet($url1))

Where $DataFile is the file location on my harddrive and url1 is the where the data is on the web.

The data gets successfully wrote to the hard drive.

Now my next line of code is $line = FileReadLine($DataFile).

The first line of the Data file is: <!DOCTYPE html>

This fails with the error Input string was not in a correct format.

Now this used to work but with ARC this fails.

What format should the Datafile be in for this to continue?

DJ Sures · Answer

If you're parsing data, i'd recommend using the javascript. The EZ-Script is quite terrible at parsing strings. While nothing has changed in the ezscript compiler, i imagine the webpage you're querying may have some changes that is breaking the parser. EZ-Script is not my favorite haha. Javascript is just so much more powerful (and 1000 times faster)

Dave Schulpius · Answer

My goodness, I gotta learn Javascript. Do you have any recommendations on educational literature I can read that would give me a good idea of how to learn it?

DJ Sures · Answer

Dave, it's so similar to ezscript... but easier i think. The commands are all categorized in their own sections. For example, all File commands start with File. And all Servo commands start with Servo. When you type the category, the drop down will display all commands in that category. So you can move your robot by typing...
 Movement.forward();

or you can create a variable like 
 var x = "some text"; print(x);

or you can make reusable functions like
 function MyFunction(var x) { &#160;x = x + 1; &#160;return&#160; x; } print(MyFunction(5));

Or you can get super crazy...
 var thisIsFun = Movement.forward(); thisIsFun;

IF conditions have open and closing braces...
 var x = 3; if (x == 3) { &#160; print("it matches"); } else { &#160; print("No match"); }

But... how ever you decide to learn javascript, the BLOCKLY in ARC generates javascript. So you can just use blockly and look at the JavaScript code it generates...

Dave Schulpius · Answer

Sweet. Thanks. I'll be looking closer at all this soon.

cem · Answer

DJ I get the impression that ez-script wouldn’t be supported for much long and removed?? Does that mean all my ez-scripts projects will have to be converted to JavaScripts?

dbeard · Answer

Question.  I don't know JavaScript.  But attempting something easy in JavaScript. This code is the code in Blockly  It returns an error object has no method. Can you get me past this hurdle.  Also any suggestions on a book or website to help me learn java script.

ptp · Answer

FileWrite($DataFile,HTTPGet($url1))

JS equivalent:
File.appendString(dataFile, Net.hTTPGet(url1));

test version:
var url1 = "https://synthiam.com/Community/Questions"; var dataFile = "c:\temp\questions.html"; var html = Net.hTTPGet(url1); File.appendString(dataFile, html);

ptp · Answer

Quote:DJ I get the impression that ez-script wouldn’t be supported for much long and removed?? Does that mean all my ez-scripts projects will have to be converted to JavaScripts?@DJ maybe a conversion tool EZ-Script to Javascript.

DJ Sures · Answer

Dbeard: ARC beta v2020.03.17.00 fixes the blockly issue with getVoltage and adds new File commands: https://synthiam.com/Products/Releases/ARC-Beta-2020-03-17-00-19052 Also, here's some code similar to PTP's but it deletes the file and reads from it...
 // assign the filename that we'll write and read from var filename = "c:\temp\doit.txt"; // If the file exists, delete it cause we're making a new one if (File.exists(filename)) File.delete(filename); // Download the contents of google's home page and put it in a file File.appendString(filename, Net.hTTPGet("https://www.google.ca")); // loop until the end of the line and print each line separated by a blank line while (!File.isReadEnd(filename)) { print(File.readLine(filename)); print(); }

PTP: EZ-Script won't be removed, as it'll continue to support legacy programs. We'll probably see it phase out over time. But it has always had weird bugs with reading files because of the way it handled quotes and other stuff. Future development on ezscript is super low on priority

dbeard · Answer

One more issue I am trying to play audio via a soundboard which is added to my project and it says there are no sound boards. Any thoughts.  I have followed the blockly tutorial on playing sound and when I use the pull down list I don't see my sound board either.

DJ Sures · Answer

Soundboard EZB or Soundboard PC? Only soundboard EZB are supported for "Play Audio" command. You want to use the Utility -> ControlCommand for the soundboard PC

dbeard · Answer

DJ.  Thanks.  I think I have everything I need, except one item.  In JavaScript I see the file. options.  But nothing about writing a file.   I see the ability to append, but not write a new file.  Or am I missing it.

ptp · Answer

Sometimes I almost lose hope... did you read my post ? I took time to open the ARC, test the code and post a valid example. Please check post #7  I've used File.appendString it creates a new file or appends to an existing one. If you don't want to append you can do a File.Delete before doing a File.appendString. To be honest, It's not your fault is mine.

dbeard · Answer

Ok, yes.  I was looking through the comments and missed your comment because it hides the top most and only shows the recent.  Started at #8 as the first visible post.  So sorry, didn't even realize that had happened until I saw your post and was looking for #7. Thanks.

dbeard · Answer

[quote][/quote]I am able to read and write the data file.  But now I need to search through it and find a string. I am using                var linein = File.readLine(dataFile);                 var n = dow.search(linein);  (this is line 20 the error is referencing.) but receive the following error: Execution Error Line 20 Col 2 - Line 1: Invalid regular expression another bit of help?

ptp · Answer

a working example:
var baseUrl = "https://synthiam.com"; var url1 = "https://synthiam.com/Community/Questions"; print("Downloading questions"); var content = Net.hTTPGet(url1); print("Content length=" + content.length); print("Questions found:"); var pos = 0; do { //This method returns -1 if the value to search for never occurs. pos = content.indexOf("/Community/Questions/", pos); if (pos>=0) { beginPos = pos //search for a double quote after finding the begin string pos = content.indexOf("\"", beginPos); if (pos>0) { endPos = pos; var link = content.substring(beginPos, endPos); print (baseUrl + link); } } } while(pos>=0);

the above code downloads Synthiam's front page questions and parses que questions links using a indexOf. The method search uses regular expressions. help with JS: https://www.w3schools.com/jsref/jsref_indexof.asp @DJ: I believe there are some bugs with File class exported to Jint. *** code edited ***

ptp · Answer

ptp · Answer

This is the html content:  Red is the "begin text" and Blue is the end

ptp · Answer

special attention to substring method: https://www.w3schools.com/jsref/jsref_substring.asp Definition and Usage The substring() method extracts the characters from a string, between two specified indices, and returns the new sub string. This method extracts the characters in a string between "start" and "end", not including "end" itself.

DJ Sures · Answer

What bugs are you noticing with the File class? And what version of arc are you using? (latest?)

ptp · Answer

weird stuff going on, maybe is a consequence of a single bug. I'll post simple code to  demo the issue.

ptp · Answer

first time (file does not exist)  Close ARC, Open ARC run the script:  Q: without a close and explicit open how you control reads and writes ?
File.appendStringLine

Locks the file, and you need to close ARC.

ptp · Answer

Reading from a File locks the file too so you need to close ARC.

dbeard · Answer

PTP In your answer in #16.  I think I follow but what I cant figure out is why doesn't your code download all the questions.  Why does it stop after 9. I don't see anything in your code that says just show the first 9 questions.

ptp · Answer

That's a different question, the website content is dynamic, does not make sense to list all the questions when a page is loaded.  Look to twitter when you scroll down to the last visible post, the page requests more content from the server same thing here. synthiam page captures the document page scroll:
$(document).ready(function() { if ($("#sidebarsearchsort").change(function() { endOfResults = !1; replaceSearchContent() }), $("#search-result").length) $(document).on("scroll ready", function() { element_in_scroll("#search-result .content-card:last-child") && (loading || addSearchContent()) }) });

and calls the function addSearchContent
function addSearchContent() { searchContent(function(n) { $("#search-result").append(n) }) } function searchContent(n) { if (endOfResults !== !0 && loading !== !0) { var t = typeof compact == "undefined" ? !1 : compact , i = typeof subCat == "undefined" ? null : subCat , r = typeof noLabel == "undefined" ? !1 : noLabel , u = typeof sort == "undefined" ? $("#sidebarsearchsort").val() : sort , f = typeof exSubCat == "undefined" ? null : exSubCat , e = typeof searchQ == "undefined" ? null : searchQ; loading = !0; addLoadingCards(t); $.ajax({ method: "POST", data: { __RequestVerificationToken: $("[name='__RequestVerificationToken']").val(), pageNumber: pageNumber, sort: u, type: cat, referenceId: $("#selected-behavior-control option:selected").val(), compactView: t, subCat: i, noLabel: r, excludeSubCat: f, searchTerm: e }, url: "/api/search/byType" }).done(function(t) { if (loading = !1, removeLoadingCards(), t === "") { endOfResults = !0; return } n(t); pageNumber = pageNumber + 1; var i = (JSON.stringify(t).match(/class=\"content-card |class=\"content-card\"/g) || []).length; i < 9 && (endOfResults = !0) }) } }

the function executes an ajax call to the server /api/search/byType and the server returns more html content (i.e. questions) that content will be appended to a specific document placeholder:
$("#search-result").append(n)

short story when you request the questions page a fixed number of questions are returned, after each scroll  more questions are retrieved from the server. you can read more about "infinite scroll": https://harshw.com/infinite-scroll-using-jquery-asp-net-mvc-mustache-entity-framework-paginate/

dbeard · Answer

You lost me at infinite scroll. But I do understand this (I think) the code only reads a page at a time.  It will take me some time to digest the above. one last question for the day.  BTW.  I have it working now and getting the data from the page thanks to your help. Here is the question.    this is what the data looks like:    >5:26 PM   The data I want is the 5:26 PM.  But depending on the time of day it could be 10:26 PM.  Is there a one-liner function that can pull the 5:26 PM or 10:26 PM out from between those markups really easy without including the other stuff?

DJ Sures · Answer

If the file is locked, you have to close the file from reading with the close command. Because reading the file keeps the file open while you read it. 
 File.readClose( filename );

ptp · Answer

I missed that method, but i don't see an equivalent writeClose

DJ Sures · Answer

Because there is no write. It's append. Appending just keeps appending to an existing file or creates a new one. There's no reason to keep a file open for appending. The only point to keeping the file open for reading is to know the position. I've added two new methods for getting the current read position and getting the file length. It'll be in today's release.

ptp · Answer

File.appendStringLine

This commands locks the file and does not release the file. Is that an expected behavior ?

ptp · Answer

@dbeard:  Example to capture the red part  The forum break's the JavaScript code, posted code picture:    @DJ  Please read my previous post!

dbeard · Answer

Here is my program for getting dow jones data.  I know it is a mess, but it works.
var url1 = "https://www.marketwatch.com/tools/marketsummary/indices/indices.asp?indexid=1&groupid=37"; print("Downloading Data"); var content = Net.hTTPGet(url1); print("Content length=" + content.length); print("Dow Jones Information Found:"); pos = 0; do { //This method returns -1 if the value to search for never occurs. var pos = content.indexOf("/investing/index/djia", pos); if (pos>=0) { beginPos = pos //search for a double quote after finding the begin string pos = content.indexOf("\"", beginPos); if (pos>0) { endPos = pos; var link = content.substring(beginPos, endPos); var volume = content.substring(endPos+144, endPos+153); var tradedate = content.substring(endPos+193, endPos+205); var tradetime = content.substring(endPos+245, endPos+255); var amount = content.substring(endPos+297, endPos+309); var tradepercent = content.substring(endPos+352, endPos+361); place = 0 var startplace = volume.indexOf(">",place); var endplace = volume.indexOf("",place); var endplace = tradedate.indexOf("",place); var endplace = tradetime.indexOf("",place); var endplace = amount.indexOf("",place); var endplace = tradepercent.indexOf("=0);