Asked
Resolved Resolved by ptp!

Filereadline Getting Error Data Is In Wrong Format

I have created a file by downloading data from the web.

this is the line of code

FileWrite($DataFile,HTTPGet($url1))

Where $DataFile is the file location on my harddrive and url1 is the where the data is on the web.

The data gets successfully wrote to the hard drive.

Now my next line of code is $line = FileReadLine($DataFile).

The first line of the Data file is:  <!DOCTYPE html>

This fails with the error Input string was not in a correct format.

Now this used to work but with ARC this fails.

What format should the Datafile be in for this to continue?


Related Hardware EZ-B v4

ARC Pro

Upgrade to ARC Pro

Unleash your robot's full potential with the cutting-edge features and intuitive programming offered by Synthiam ARC Pro.

PRO
USA
#25   — Edited

That's a different question, the website content is dynamic, does not make sense to list all the questions when a page is loaded.  Look to twitter when you scroll down to the last visible post, the page requests more content from the server same thing here.

synthiam page captures the document page scroll:

$(document).ready(function() {
    if ($("#sidebarsearchsort").change(function() {
        endOfResults = !1;
        replaceSearchContent()
    }),
    $("#search-result").length)
        $(document).on("scroll ready", function() {
            element_in_scroll("#search-result .content-card:last-child") && (loading || addSearchContent())
        })
});

and calls the function addSearchContent

function addSearchContent() {
    searchContent(function(n) {
        $("#search-result").append(n)
    })
}


function searchContent(n) {
    if (endOfResults !== !0 && loading !== !0) {
        var t = typeof compact == "undefined" ? !1 : compact
          , i = typeof subCat == "undefined" ? null : subCat
          , r = typeof noLabel == "undefined" ? !1 : noLabel
          , u = typeof sort == "undefined" ? $("#sidebarsearchsort").val() : sort
          , f = typeof exSubCat == "undefined" ? null : exSubCat
          , e = typeof searchQ == "undefined" ? null : searchQ;
        loading = !0;
        addLoadingCards(t);
        $.ajax({
            method: "POST",
            data: {
                __RequestVerificationToken: $("[name='__RequestVerificationToken']").val(),
                pageNumber: pageNumber,
                sort: u,
                type: cat,
                referenceId: $("#selected-behavior-control option:selected").val(),
                compactView: t,
                subCat: i,
                noLabel: r,
                excludeSubCat: f,
                searchTerm: e
            },
            url: "/api/search/byType"
        }).done(function(t) {
            if (loading = !1,
            removeLoadingCards(),
            t === "") {
                endOfResults = !0;
                return
            }
            n(t);
            pageNumber = pageNumber + 1;
            var i = (JSON.stringify(t).match(/class=\\"content-card |class=\\"content-card\\"/g) || []).length;
            i < 9 && (endOfResults = !0)
        })
    }
}

the function executes an ajax call to the server /api/search/byType and the server returns more html content (i.e. questions) that content will be appended to a specific document placeholder:

$("#search-result").append(n)

short story when you request the questions page a fixed number of questions are returned, after each scroll  more questions are retrieved from the server.

you can read more about "infinite scroll": http://harshw.com/infinite-scroll-using-jquery-asp-net-mvc-mustache-entity-framework-paginate/

PRO
USA
#26  

You lost me at infinite scroll.

But I do understand this (I think) the code only reads a page at a time.  It will take me some time to digest the above.

one last question for the day.  BTW.  I have it working now and getting the data from the page thanks to your help.

Here is the question.    this is what the data looks like:    >5:26 PM</td>   The data I want is the 5:26 PM.  But depending on the time of day it could be 10:26 PM.  Is there a one-liner function that can pull the 5:26 PM or 10:26 PM out from between those markups really easy without including the other stuff?

PRO
Synthiam
#27   — Edited

If the file is locked, you have to close the file from reading with the close command. Because reading the file keeps the file open while you read it.


File.readClose( filename );

PRO
USA
#28  

I missed that method, but i don't see an equivalent writeClose

PRO
Synthiam
#29  

Because there is no write. It's append. Appending just keeps appending to an existing file or creates a new one. There's no reason to keep a file open for appending. The only point to keeping the file open for reading is to know the position.

I've added two new methods for getting the current read position and getting the file length. It'll be in today's release.

PRO
USA
#30  
File.appendStringLine

This commands locks the file and does not release the file. Is that an expected behavior ?

PRO
USA
#31   — Edited

@dbeard:

User-inserted image

Example to capture the red part

The forum break's the JavaScript code, posted code picture:

User-inserted image

@DJ  Please read my previous post!

PRO
USA
#32  

Here is my program for getting dow jones data.  I know it is a mess, but it works.

var url1 = "http://www.marketwatch.com/tools/marketsummary/indices/indices.asp?indexid=1&groupid=37";;

print("Downloading Data");
var content = Net.hTTPGet(url1);

print("Content length=" + content.length);

print("Dow Jones Information Found:");

pos = 0;
do
{
  //This method returns -1 if the value to search for never occurs.
  var pos = content.indexOf("/investing/index/djia", pos);
  
  if (pos>=0)
  {
      beginPos = pos

      //search for a double quote after finding the begin string
      pos = content.indexOf("\"", beginPos);
      if (pos>0)
      {
        endPos = pos;
        var link = content.substring(beginPos, endPos);
        var volume = content.substring(endPos+144, endPos+153);
        var tradedate = content.substring(endPos+193, endPos+205);
        var tradetime = content.substring(endPos+245, endPos+255);
        var amount = content.substring(endPos+297, endPos+309);
        var tradepercent = content.substring(endPos+352, endPos+361);
                
        place = 0
        var startplace = volume.indexOf(">",place);
        var endplace = volume.indexOf("<", place);
        var fvolume = volume.substr(startplace+1, endplace-1);
        
        place = 0
        var startplace = tradedate.indexOf(">",place);
        var endplace = tradedate.indexOf("<", place);
        var ftdate = tradedate.substr(startplace+1, endplace-1);
        
        place = 0
        var startplace = tradetime.indexOf(">",place);
        var endplace = tradetime.indexOf("<", place);
        var fttime = tradetime.substr(startplace+1, endplace-1);
        
        place = 0
        var startplace = amount.indexOf(">",place);
        var endplace = amount.indexOf("<", place);
        var famount = amount.substr(startplace+1, endplace-1);
        
        place = 0
        var startplace = tradepercent.indexOf(">",place);
        var endplace = tradepercent.indexOf("<", place);
        var ftpercent = tradepercent.substr(startplace+1, endplace-1);
        
        print("Date:    " +ftdate);
        print("Time:    " +fttime);
        print("Volume:  "+fvolume);
        print("Amount:  "+famount);
        print("Percent: "+ftpercent);
        
        halt();
      }
  }
}
while(pos>=0);