Asked
Resolved Resolved by ptp!

Filereadline Getting Error Data Is In Wrong Format

I have created a file by downloading data from the web.

this is the line of code

FileWrite($DataFile,HTTPGet($url1))

Where $DataFile is the file location on my harddrive and url1 is the where the data is on the web.

The data gets successfully wrote to the hard drive.

Now my next line of code is $line = FileReadLine($DataFile).

The first line of the Data file is:  <!DOCTYPE html>

This fails with the error Input string was not in a correct format.

Now this used to work but with ARC this fails.

What format should the Datafile be in for this to continue?


Related Hardware EZ-B v4

ARC Pro

Upgrade to ARC Pro

ARC Pro is your gateway to a community of like-minded robot enthusiasts and professionals, all united by a passion for advanced robot programming.

PRO
Synthiam
#1  

If you're parsing data, i'd recommend using the javascript. The EZ-Script is quite terrible at parsing strings. While nothing has changed in the ezscript compiler, i imagine the webpage you're querying may have some changes that is breaking the parser. EZ-Script is not my favorite haha. Javascript is just so much more powerful (and 1000 times faster)

#2  

My goodness, I gotta learn Javascript. Do you have any recommendations on educational literature I can read that would give me a good idea of how to learn it?

PRO
Synthiam
#3  

Dave, it's so similar to ezscript... but easier i think.

The commands are all categorized in their own sections. For example, all File commands start with File. And all servo commands start with Servo. When you type the category, the drop down will display all commands in that category.

So you can move your robot by typing...


Movement.forward();

or you can create a variable like


var x = "some text";

print(x);

or you can make reusable functions like


function MyFunction(var x) {

 x = x + 1;

 return  x;
}

print(MyFunction(5));

Or you can get super crazy...


var thisIsFun = Movement.forward();

thisIsFun;

IF conditions have open and closing braces...


var x = 3;

if (x == 3) {

  print("it matches");

} else {

  print("No match");
}

But... how ever you decide to learn javascript, the Blockly in ARC generates javascript. So you can just use Blockly and look at the JavaScript code it generates...

#4  

Sweet. Thanks. I'll be looking closer at all this soon.

United Kingdom
#5  

DJ I get the impression that ez-script wouldn’t be supported for much long and removed??

Does that mean all my ez-scripts projects will have to be converted to JavaScripts?

PRO
USA
#6  

Question.  I don't know JavaScript.  But attempting something easy in JavaScript.

This code is the code in Blockly

User-inserted image

It returns an error object has no method.

Can you get me past this hurdle.  Also any suggestions on a book or website to help me learn java script.

PRO
USA
#7  
FileWrite($DataFile,HTTPGet($url1))

JS equivalent:

File.appendString(dataFile, Net.hTTPGet(url1));

test version:

var url1 = "https://synthiam.com/Community/Questions";
var dataFile = "c:\\temp\\questions.html";

var html = Net.hTTPGet(url1);
File.appendString(dataFile, html);
PRO
USA
#8  

Quote:

DJ I get the impression that EZ-Script wouldn’t be supported for much long and removed??

Does that mean all my ez-scripts projects will have to be converted to JavaScripts?

@DJ maybe a conversion tool EZ-Script to Javascript.

PRO
Synthiam
#9   — Edited

Dbeard: ARC beta v2020.03.17.00 fixes the Blockly issue with getVoltage and adds new File commands: https://synthiam.com/Products/Releases/ARC-Beta-2020-03-17-00-19052

Also, here's some code similar to PTP's but it deletes the file and reads from it...


// assign the filename that we'll write and read from
var filename = "c:\\temp\\doit.txt";

// If the file exists, delete it cause we're making a new one
if (File.exists(filename))
  File.delete(filename);

// Download the contents of google's home page and put it in a file
File.appendString(filename, Net.hTTPGet("http://www.google.ca"));

// loop until the end of the line and print each line separated by a blank line
while (!File.isReadEnd(filename)) {

  print(File.readLine(filename));
  print();
}

PTP: EZ-Script won't be removed, as it'll continue to support legacy programs. We'll probably see it phase out over time. But it has always had weird bugs with reading files because of the way it handled quotes and other stuff. Future development on ezscript is super low on priority :)

PRO
USA
#10  

One more issue I am trying to play audio via a soundboard which is added to my project and it says there are no sound boards.

Any thoughts.  I have followed the Blockly tutorial on playing sound and when I use the pull down list I don't see my sound board either.

PRO
Synthiam
#11  

Soundboard EZB or Soundboard PC? Only soundboard EZB are supported for "Play Audio" command. You want to use the Utility -> ControlCommand for the soundboard PC

PRO
USA
#12  

DJ.  Thanks.  I think I have everything I need, except one item.  In JavaScript I see the file. options.  But nothing about writing a file.   I see the ability to append, but not write a new file.  Or am I missing it.

User-inserted image

PRO
USA
#13   — Edited

Sometimes I almost lose hope... did you read my post ? I took time to open the ARC, test the code and post a valid example.

Please check post #7

I've used File.appendString it creates a new file or appends to an existing one.

If you don't want to append you can do a File.Delete before doing a File.appendString.

To be honest, It's not your fault is mine.

PRO
USA
#14  

Ok, yes.  I was looking through the comments and missed your comment because it hides the top most and only shows the recent.  Started at #8 as the first visible post.  So sorry, didn't even realize that had happened until I saw your post and was looking for #7.

Thanks.

PRO
USA
#15  

[quote][/quote]I am able to read and write the data file.  But now I need to search through it and find a string.

I am using var linein = File.readLine(dataFile);

var n = dow.search(linein);  (this is line 20 the error is referencing.)

but receive the following error:

Execution Error Line 20 Col 2 - Line 1: Invalid regular expression

another bit of help?

PRO
USA
#16   — Edited

a working example:

var baseUrl = "https://synthiam.com";
var url1 = "https://synthiam.com/Community/Questions";

print("Downloading questions");
var content = Net.hTTPGet(url1);

print("Content length=" + content.length);

print("Questions found:");

var pos = 0;
do
{
  //This method returns -1 if the value to search for never occurs.
  pos = content.indexOf("/Community/Questions/", pos);
  
  if (pos>=0)
  {
      beginPos = pos

      //search for a double quote after finding the begin string
      pos = content.indexOf("\"", beginPos);
      if (pos>0)
      {
        endPos = pos;
        var link = content.substring(beginPos, endPos);
        print (baseUrl + link);
      }
  }
}
while(pos>=0);

   

the above code downloads Synthiam's front page questions and parses que questions links using a indexOf.

The method search uses regular expressions.

help with JS: https://www.w3schools.com/jsref/jsref_indexof.asp

@DJ: I believe there are some bugs with File class exported to Jint.

*** code edited ***

PRO
USA
#18  

This is the html content: User-inserted image

Red is the "begin text" and Blue is the end

PRO
USA
#19  

special attention to substring method: https://www.w3schools.com/jsref/jsref_substring.asp

Definition and Usage The substring() method extracts the characters from a string, between two specified indices, and returns the new sub string.

This method extracts the characters in a string between "start" and "end", not including "end" itself.

PRO
Synthiam
#20  

What bugs are you noticing with the File class? And what version of ARC are you using? (latest?)

PRO
USA
#21  

weird stuff going on, maybe is a consequence of a single bug. I'll post simple code to  demo the issue.

PRO
USA
#22  
function PrintLines(filename) 
{
  var lineIx = 0;
  while (!File.isReadEnd(filename)) 
  {
    var line = File.readLine(filename);
    print(lineIx + " line:" + line);
  }
}


var filename="c:\\temp\\lines.txt";

var now = new Date().toLocaleString();
if (!File.exists(filename))
{
  print("file does not exist");
  File.appendStringLine(filename, now + ">>" + "First time");
}
else
{
  print("file exists");
  File.appendStringLine(filename, now + ">>" + "foo");
}

print("reset read file");
File.readReset(filename);

print("PrintLines");
PrintLines(filename);

print("add bar");
File.appendStringLine(filename, now + ">>" + "bar");

print("PrintLines");
PrintLines(filename);

first time (file does not exist) User-inserted image

Close ARC, Open ARC run the script: User-inserted image

Q: without a close and explicit open how you control reads and writes ?

File.appendStringLine

Locks the file, and you need to close ARC.

PRO
USA
#23  

Reading from a File locks the file too so you need to close ARC. User-inserted image

PRO
USA
#24  

PTP

In your answer in #16.  I think I follow but what I cant figure out is why doesn't your code download all the questions.  Why does it stop after 9.

I don't see anything in your code that says just show the first 9 questions.

PRO
USA
#25   — Edited

That's a different question, the website content is dynamic, does not make sense to list all the questions when a page is loaded.  Look to twitter when you scroll down to the last visible post, the page requests more content from the server same thing here.

synthiam page captures the document page scroll:

$(document).ready(function() {
    if ($("#sidebarsearchsort").change(function() {
        endOfResults = !1;
        replaceSearchContent()
    }),
    $("#search-result").length)
        $(document).on("scroll ready", function() {
            element_in_scroll("#search-result .content-card:last-child") && (loading || addSearchContent())
        })
});

and calls the function addSearchContent

function addSearchContent() {
    searchContent(function(n) {
        $("#search-result").append(n)
    })
}


function searchContent(n) {
    if (endOfResults !== !0 && loading !== !0) {
        var t = typeof compact == "undefined" ? !1 : compact
          , i = typeof subCat == "undefined" ? null : subCat
          , r = typeof noLabel == "undefined" ? !1 : noLabel
          , u = typeof sort == "undefined" ? $("#sidebarsearchsort").val() : sort
          , f = typeof exSubCat == "undefined" ? null : exSubCat
          , e = typeof searchQ == "undefined" ? null : searchQ;
        loading = !0;
        addLoadingCards(t);
        $.ajax({
            method: "POST",
            data: {
                __RequestVerificationToken: $("[name='__RequestVerificationToken']").val(),
                pageNumber: pageNumber,
                sort: u,
                type: cat,
                referenceId: $("#selected-behavior-control option:selected").val(),
                compactView: t,
                subCat: i,
                noLabel: r,
                excludeSubCat: f,
                searchTerm: e
            },
            url: "/api/search/byType"
        }).done(function(t) {
            if (loading = !1,
            removeLoadingCards(),
            t === "") {
                endOfResults = !0;
                return
            }
            n(t);
            pageNumber = pageNumber + 1;
            var i = (JSON.stringify(t).match(/class=\\"content-card |class=\\"content-card\\"/g) || []).length;
            i < 9 && (endOfResults = !0)
        })
    }
}

the function executes an ajax call to the server /api/search/byType and the server returns more html content (i.e. questions) that content will be appended to a specific document placeholder:

$("#search-result").append(n)

short story when you request the questions page a fixed number of questions are returned, after each scroll  more questions are retrieved from the server.

you can read more about "infinite scroll": http://harshw.com/infinite-scroll-using-jquery-asp-net-mvc-mustache-entity-framework-paginate/

PRO
USA
#26  

You lost me at infinite scroll.

But I do understand this (I think) the code only reads a page at a time.  It will take me some time to digest the above.

one last question for the day.  BTW.  I have it working now and getting the data from the page thanks to your help.

Here is the question.    this is what the data looks like:    >5:26 PM</td>   The data I want is the 5:26 PM.  But depending on the time of day it could be 10:26 PM.  Is there a one-liner function that can pull the 5:26 PM or 10:26 PM out from between those markups really easy without including the other stuff?

PRO
Synthiam
#27   — Edited

If the file is locked, you have to close the file from reading with the close command. Because reading the file keeps the file open while you read it.


File.readClose( filename );

PRO
USA
#28  

I missed that method, but i don't see an equivalent writeClose

PRO
Synthiam
#29  

Because there is no write. It's append. Appending just keeps appending to an existing file or creates a new one. There's no reason to keep a file open for appending. The only point to keeping the file open for reading is to know the position.

I've added two new methods for getting the current read position and getting the file length. It'll be in today's release.

PRO
USA
#30  
File.appendStringLine

This commands locks the file and does not release the file. Is that an expected behavior ?

PRO
USA
#31   — Edited

@dbeard:

User-inserted image

Example to capture the red part

The forum break's the JavaScript code, posted code picture:

User-inserted image @DJ  Please read my previous post!

PRO
USA
#32  

Here is my program for getting dow jones data.  I know it is a mess, but it works.

var url1 = "http://www.marketwatch.com/tools/marketsummary/indices/indices.asp?indexid=1&groupid=37";

print("Downloading Data");
var content = Net.hTTPGet(url1);

print("Content length=" + content.length);

print("Dow Jones Information Found:");

pos = 0;
do
{
  //This method returns -1 if the value to search for never occurs.
  var pos = content.indexOf("/investing/index/djia", pos);
  
  if (pos>=0)
  {
      beginPos = pos

      //search for a double quote after finding the begin string
      pos = content.indexOf("\"", beginPos);
      if (pos>0)
      {
        endPos = pos;
        var link = content.substring(beginPos, endPos);
        var volume = content.substring(endPos+144, endPos+153);
        var tradedate = content.substring(endPos+193, endPos+205);
        var tradetime = content.substring(endPos+245, endPos+255);
        var amount = content.substring(endPos+297, endPos+309);
        var tradepercent = content.substring(endPos+352, endPos+361);
                
        place = 0
        var startplace = volume.indexOf(">",place);
        var endplace = volume.indexOf("<", place);
        var fvolume = volume.substr(startplace+1, endplace-1);
        
        place = 0
        var startplace = tradedate.indexOf(">",place);
        var endplace = tradedate.indexOf("<", place);
        var ftdate = tradedate.substr(startplace+1, endplace-1);
        
        place = 0
        var startplace = tradetime.indexOf(">",place);
        var endplace = tradetime.indexOf("<", place);
        var fttime = tradetime.substr(startplace+1, endplace-1);
        
        place = 0
        var startplace = amount.indexOf(">",place);
        var endplace = amount.indexOf("<", place);
        var famount = amount.substr(startplace+1, endplace-1);
        
        place = 0
        var startplace = tradepercent.indexOf(">",place);
        var endplace = tradepercent.indexOf("<", place);
        var ftpercent = tradepercent.substr(startplace+1, endplace-1);
        
        print("Date:    " +ftdate);
        print("Time:    " +fttime);
        print("Volume:  "+fvolume);
        print("Amount:  "+famount);
        print("Percent: "+ftpercent);
        
        halt();
      }
  }
}
while(pos>=0);
PRO
Synthiam
#33  

Use this to see what files you have locked: https://synthiam.com/Products/Controls/General/File-Information-16183

PRO
USA
#34  

Thanks everyone for the help.