United Kingdom
Asked — Edited

Flaky Serial Link With V4

DJ I hope you can help? I am experiencing major problems with the reliability of the serial link that concerns me a lot, as it stands the problem is bad enough to undermine all my work (over the last year) on utilizing the v4 for the ALTAIR robots.

The new ALTAIR heads have multiple PICs that all report back to a Master PIC which itself communicates with the v4. On testing I keep noticing the v4 drops out randomly and also breaks all serial comms at random times. In an attempt to discover the problem, I shut down all the interrupts buzzing around the PIC network, so that the Master was just (simply) communicating with the v4 with no external interrupts. I then ran some simple code that just sends 3 bytes to the v4 (every 100mS) then increments the bytes and so on. At the v4 end it just looks for the 3 bytes in the buffer reads them and increment a $times counter. What happens is that the v4 just breaks serial comms or has byte read errors at random times.

When the fault occurs serial comms completely stop and it is necessary to stop the script and start it again. I have found this happens when I put a static "3" in the UartAvailable line (for example if UartAvailable(0,0) = 3), but this should not be a problem as the v4 is only ever going to get an isolated packet of 3 bytes? To get around this if I change the static "3" to ">= 2" (for example if UartAvailable(0,0) >= 2) then the comms does not stop or error (so much), but of course this means that a packet misread has occurred and the buffer has accepted a 2 byte packet which means we have lost a byte.

From my tests, in general it looks like the bytes available error occurs once in roughly 100 packet sends and a byte error (data byte read incorrectly) occurs around once every 250 packet sends. Below is a screen dump from one test which shows the errors from just over 1600 packet sends.

User-inserted image

Unreliable serial is a major setback for me as all my robots use serial linked subsystem PIC modes so these dropouts where serial links can also just stop is a complete disaster for my robot designs!

Tony


ARC Pro

Upgrade to ARC Pro

Your robot can be more than a simple automated machine with the power of ARC Pro!

PRO
United Kingdom
#9  

I am now continually getting "nagged" to post that my forum thread "Flaky serial link with v4" has been resolved when it has not. Some things are probably not resolvable and the fact that DJ and Jeramie have never come back with a solution maybe confirms that the serial link can be flakey?

I think its pretty poor to have to pretend that someone has resolved my help request just to stop (what in my opinion are) these unnecessary emails, this will be the last time I will post for assistance because of this.

EZ-Robot, can I suggest that you allow for the situation that a help request cannot be resolved, so it stops these continuously annoying emails being sent in these cases. Thanks

Tony

PRO
USA
#10  

I hate that nag and stopped using the assistance option because of it. I'm in line with Tony on this absolutely unnecessary and annoying. Seems to work on days unresolved. Please remove this nag EZRobot.

Sorry now back to Tony's thread.....

#11  

@Toymaker Yeah, it is annoying. I started getting them even no one responded to one of my posts at all. Just my original post in the thread, nothing else. Steve G was kind enough to "bump" the thread to try to get some response. But the fact remains it is still unresolved, and I will continue to get notices of that fact what seems like every other day. Obviously these notices are automated, but maybe there needs to be humans looking at the situations to keep them from going out unnecessarily. Especially considering that you can be banned for not marking the thread resolved within some amorphous time period. In some cases it just takes time to get an issue resolved. Perhaps a lot of time. Getting unending notices so soon and so often does not help the situation. Worse, it can lead to simply giving up on the product altogether.

PRO
Canada
#12  

While I can't do anything about the email notifications, that's @DJ's department:) I can mention that I haven't had a chance to look at the serial link yet but I should have time in the next little bit.

Tony quick question for you, are you using internal RC oscillators with your PICs or external crystals?

PRO
United Kingdom
#13  

Thanks Will and WBS for your comments, I am relieved that it is not just me that gets wound up on this!

I now get a "nag" email every day, with the usual mandatory threats.

I am quite offended by the threat that we can be banned if we do not accept that the thread has been resolved within a certain time period, when obviously on some occasions it does not! To EZ-Robot I am not going to pretend this thread has been resolved when it has not - so to all my forum friends if I suddenly disappear from the group you know why!

I will not conform to this treatment, (and being treated like some sort of errant child) so I may well get banned!

DJ and EZ-Robot, you respectfully need to seriously look at this urgently else you will lose valuable users and future customers.

My recommendation to forum members is to never use "assistance required", I am sure help will still be given from this great community.

Tony

PRO
United Kingdom
#14  

Jeremie, I am using internal oscillators but am 100% sure this is not the problem - with the latest generation of PICs, I have never seen timing issues using RC. Also the head PIC network are all serially linked at the same baud rate and there are no issues there. The problem has to be at the v4 buffer end, what I have done now is to get the master_PIC to wait for the v4 (to flag) before any data transactions are attempted, obviously this is not optimum if very important data needs to be sent to the v4 as there can be appreciable delays before it receives the data, but this is the only way I can get the data packets to transfer reliably. Ideally it would be good to be able send urgent data over to the v4 instantly without have to wait for another v4 instigated cycle which is obviously a delay.

More info on this, the master_PIC serial link to v4 has a "high priority" interrupt, so when the v4 sends a dummy (or command) byte the master_PIC stops whatever it is doing (jumps to the interrupt) and sends the latest data packet (containing sensory data etc from the networked PICs) back to the v4. The v4 has to instigate the transaction, not having interrupts on the v4 is one reason for these limitations.

Also the random disconnections are a real nuisance, the longest I have got is a run of 4 hours, but mostly there is a disconnection every 2 hours or so.

Tony

Ireland
#15  

Tony, like many others I wish I could jump in and resolve the serial communication issue, but personally I have little knowledge in this field. May I make a suggestion.

Use two Ezb's connected by serial ( no other devices connected ) and see can you replicate the fault,if that is possible it may identify where the issue lies.Then you will find a way with you skills to resolve it.

Pat

PRO
Synthiam
#16  

This is a logic oversight and not a bug with the ez-b or flaky serial. The IF condition which checks for the available bytes not equaling 3 is the doing the job correctly. The reason for the behavior which you have identified is that at the exact moment the UartAvailble() returns, there is indeed 0, 1 or 2 bytes in the buffer and not 3 as you were hoping. This is because the transmission from the other serial device is transmitting at a baud rate which takes time.

The logical challenge is a result of what you're not used to with micros. The EZ-B v4 isn't a micro, it's a program on a micro that does a bunch of stuff and communicates over WiFi. So, Tony's past experience as a micro assembler programmer is to check the UART Receive bit on the micro to instantly (within a microsecond loop) see if there is data to be read. However, on the EZ-B v4 you can only check to see if data is available at a much much slower interval (milliseconds vs microseconds). This dramatically increases the chance of checking for data while the data is still being transmitted from the serial master device.

You will always run into a situation where the number of expected bytes has not been transmitted, yet. And that's expected behavior for this design - delay, loop and the next time the data will be available.

My advice is...

  1. ignore the error condition that you had created because it will occur when ever the condition executes while data is in the middle of being transmitted.

  2. treat the condition with the same logic as my original example that i had provided to you when the question was first asked. Reference this thread: https://synthiam.com/Community/Questions/8067

You will notice that in my original response to you, I had written a loop which checks for the availability to be EQUAL or GREATER than the byte packet size. From that, i pull the data in increments of the packet. Here is an example and merging your most recent code...


# This is a demo on how to wait for a specific UART packet size
# and process the data when it has been received

# Initialize the UART (this also resets the ez-b's input buffer)
UARTInit( 0, 0, 9600 )

:loop

# Only process data if the packet size is available (packet size is 3 bytes)
# If the expected data size is not available, assume we're in the
# middle of transmission, delay and loop.
IF (UartAvailable(0, 0) >= 3)

  # A valid packet is ready. Read the packet
  UARTReadBinary( 0, 0, 3, $inputData )

  $byte1_in  = $inputdata[0]
  $byte2_in  = $inputdata[1]
  $byte3_in  = $inputdata[2]  # 8 thermal pixel data
  $times++
  $new_data = 0

  print($byte1_in +":"+ $byte2_in +":"+ $byte3_in)

  IF ($last_byte1 != $byte1_in or $last_byte2 != $byte2_in or $last_byte3 != $byte3_in)
    # latest data packet is different to last
    $new_data = 1
    print("new data")
    $last_byte1 = $byte1_in
    $last_byte2 = $byte2_in
    $last_byte3 = $byte3_in
  ENDIF

ELSE

  # no valid packet is available
  # pause before we check again
  # to be friendly on the communication channel

  sleep(100)

ENDIF

goto(loop)

To provide additional information on how the EZ-B v4 serial works may help set your mind at ease. The STM32 ARM micro for the EZ-B v4 has DMA for the UART receive and transmit. I use only the receive DMA and it is enabled when the UartInit() method is called which configures the DMA and UART parameters. The DMA has a large array in memory which it stores the incoming bytes automatically. This is a fantastic hardware configuration and requires only configuration code and no "program" code. The data is stored automatically in a buffer and the DMA knows its position in the array by a CPU register.

When ARC calls UartAvailable(), the EZ-B v4 returns the data in the register which is the number of bytes currently populated.

When you read the Uart, the register is not reset. As you would expect, the number of bytes you read are subtracted from the register. This means it maintains the bytes which you haven't read. The only way to fully clear the register and reset it to 0 is to read all available byte data.

The EZ-B v4 micro has a hardware configuration, not a software program that handles the UART. In my very extensive tests, the hardware of the STM32 micro has not had any issues.