Asked — Edited

Bug Report: Ez-B In Client Mode Still Running Dhcp

EDIT EDIT EDIT: See post 16 for explanation of the cause, and a possible solution

I have mentioned this in a few conversations about network issues, but I don't know if DJ saw it since the issues were marked a resolved before he commented, so I am opening an explicit thread for this bug report.

While troubleshooting some strange issues on my 802.11g and n network when running multiple EZ-B's (specifically, EZ-B's and the few other G or N devices I have refusing their reserved IP addresses and getting other addresses, not showing up in the DHCP client list of my router, and some strange error messages in the router log) I ran wireshark on a computer connected to my 802.11n network with a display filter looking at bootp messages then powering on EZ-B's.

The first EZ-B come up normally and connects to my network with normal messaging.

When I power up the second one, and it sends out a discover, both my router, and the first EZ-B send out an offer. The EZ-B accepts one of the offers (usually the one from the first EZ-B), and rejects the other (and the router sends out a reject and logs an error because it is seeing another DHCP server on the network).

When I power up a 3rd EZ-B, the first 2 send offers.

4th EZ-B, the first 3 send offers.

Any other device also get DHCP offers from the EZ-B's as well as the router.

Thankfully most of my network is on 802.11ac or wired and not impacted by this, but the few 802.11n devices behave very strangely when the EZ-Bs are on (losing their internet connection, or restarting their network connections).

Hopefully this will be an easy firmware fix, because it is actually fairly serious and unfortunately validates the few network administrators who have told users here that they can't put the EZ-Bs on their networks. Having more than one DHCP server on a network is a major networking policy violation.

Alan


ARC Pro

Upgrade to ARC Pro

Elevate your robot's capabilities to the next level with Synthiam ARC Pro, unlocking a world of possibilities in robot programming.

PRO
Synthiam
#1  

We have many ez-b's on the same network with dhcp without similar issues - including schools. Perhaps your router interface is not ideal for debugging and helping you understand how dhcp works - specifically at the broadcast level.

#2  

Maybe you should run wirshark on the network and see the same thing instead of dismissing my troubleshooting. A lot of people have complained about unusual situations, particularly when using reserved IP addresses on their EZ-B's. I took the time to figure it out rather than blaming their routers. Wirshark distinctly shows that the EZ-B's are responding to DHCP requests when in client mode. They should only do this in AP mode.

Alan

PRO
Synthiam
#3  

There's also a lot of strangeness on routers as well - the funny thing we have noticed with networking protocols is the lack of standardization between manufacturers. Remember the HP issue we had with the first bunch of EZ-B's and DHCP? That was due to the driver being used by HP not using DHCP correctly - and not following the dhcp guideline. We had to update the next release with a fix, just for HP computers because they refused to fix their mistake. It ended up costing EZ-Robot many thousands of dollars in R&D and reprogramming modules.

We follow the guideline for DHCP - and test it on our ASUS router at the office, as well as the router at each of our homes.

#4  

The EZ-B's are sending DHCP offers to any device that attaches to the network. That is not a router issue other than that sometimes the router sends it's offer a few millisecond after the EZ-B's do. Another router may respond faster and therefore mask the issue, but the fact remains that the EZ-B's are sending messages that they should not be sending when in client mode.

After work tomorrow I'll charge up the batteries and duplicate it with three or four different brand routers and I can send you the capture files if that is what you need.

Alan

#5  

OK, here are some screen shots of what Wireshark sees happening. I'll do one post per shot. So far, I have tested with two routers, both by TP-Link, but very different models. I can also test with several different D-Links, a Tenda, and an Actiontech, but I will need to spend some time setting them up and I think I have enough evidence here that there is an issue and it isn't with my routers. But, if you disagree, I'll do the additional tests. I'll also set up a Windows DHCP server and disable my router DHCP server, and after the first EZ-B gets an address, I'll disable the Windows one and you will see that the EZ-B is still handing out addresses.

Here is the screen shot, and I will explain what it is showing below:

User-inserted image

OK. Packet 1060 and 1213 are all the first EZ-B coming online. It successfully gets its reserved IP address, 192.168.1.4 from my router.

Packet 1961 was when I booted the second EZ-B. ou can see that in packet 1963, there is a DHCP Offer coming from 192.168.1.4 (the first EZ-B). The second EZ-B then sends a request in packet 1964 which the first EZ-B sends an ACK and the router sends a NAK. (Router log showed errors about Server ID not found and request an invalid IP. That was in response to the Offer sent from the first EZ-B and is the reason you don't see an offer from the router). This EZ-B received IP address 192.168.1.2 even though the router has that address reserved for a different EZ-B's MAC address.

The second EZ-B does not appear in the router's DHCP client list, but I can router packets to it (not shown since I am filtering on bootp).

OK, now with packet 2185 I boot up the third EZ-B and it sends a discover. The second one (192.168.1.2) sends an offer, the EZ-B sends a request, both original EZ-B's send ACKs, the 3rd EZ-B declines one of those. The router is unhappy and sends a NAK. This bounces around for a while before the EZ-B finally settles in with an address (happens to be 192.168.1.3, the next available according to whichever EZ-B won the DHCP fight, but not the address that EZ-B should have receive from my router based on its MAC reservation which was 192.168.1.2..... Obviously could not have received that because there was already one on the network.

Next post will have the screen shot from the same test but with a different router.

Alan

Packet

#6  

OK. Next screenshot, description below.

User-inserted image

This one is a little simpler, I think because I only have one of the EZ-B's defined with a reserved IP in this router.

Packets 29, 32 are a normal transaction. This EZ-B does not have a reserved address in this router, so the router assigned it 192.168.1.5.

Packet 83 the second EZ-B comes online. This one also does not have a reserved IP in the router. The first EZ-B offers it an IP address which it accepts. 192.168.1.2 this all happens before the router even thinks about responding.... Again, the second EZ-B does not show in the router's client list, and in fact, the EZ-B has the IP address that is reserved for a different MAC address. However, ARC can see it.

Packet 138 the 3rd EZ-B comes online and both the first and second send offers. One of them wins and finally the router gets involved and sends a NAK because it is totally lost now that there are multiple DHCP servers on its network......

The 3rd EZ-B again get IP address 192.168.1.3 when by its router reservation it should have received 192.168.1.2.

Clearly, the EZ-B's are still running DHCP server in client mode.

I did one final test as well. I rebooted a computer on the network with 2 EZ-Bs running, and it was also assigned its IP address by the EZ-B and not the router (in fact, it again was given 192.168.1.2 rather than its reserved IP address of 192.168.1.103. It was also assigned a single DNS server of 192.168.1.1. My router specifies DNS servers in its DHCP offers of 208.67.222.222 and 208.67.220.220. I turned off the EZ-B's and restarted the NIC and the computer connected back to my network correctly.

Let me know what other tests you need to prove that you have left DHCP server running on the EZ-B's when they are in client mode.

If you need instructions on running Wireshark, contact me.

Alan

#7  

thetechguru

Thank you for the detailed information.

After reading your posts I did some investigation my self as i have been having some DHCP problems with a few mobile network devices and gave up on them.

After looking at your post and doing some investigation , I see this happening when my wife reconnects her network on her moblie device.

So i changed it to not search for a wifi but hardwired the wifi name in the device and all is good now.

It never really affected my EZB 4 boards and I ran them together all the time and never though that could be an issue with the mobile device.

Thanks again.

#8  

If you aren't trying to have reserved addresses, or you reserve the lowest octets in your range and start them in low to high order, you won't see a problem with the EZ-B's, but any other device on the G or N network could see an issue.

My network is a mix of G, N, and AC devices. The AC devices are fine, but the G and N ones are affected if they are started after the EZ-B's.

Alan

#9  

DJ, Have you looked into this issue yet? It is still causing irritating issues if I don't start my EZ-B's in the same order every time, and I believe it is causing the issue I am helping @oldbotbuilder troubleshoot where his wife's PC disconnects from the network whenever he connects his EZ-B to his router.

Alan

PRO
Synthiam
#10  

I've contacted the manufacturer of the tcp stack that is licensed for our wifi modules - we're at the mercy of their response.

#11  

Thanks DJ.

I am hoping this will be a simple firmware update.

Alan

PRO
Synthiam
#12  

It won't be - there is no way to update firmware on the wifi modules because they're static. Best we can get from them is an answer on a configuration setting or to acknowledge the issue.

#13  

As I suspected, the reason not everyone sees this is that some routers are faster at responding to DHCP requests and mask the issue.

I just got a new FiOS Quantum Gateway (Greenwave Systems) router for my Verizon FiOS service, and when I set up the robots to connect with it, they each get their assigned IP addresses regardless of which starts first, and other wireless devices get assignments from the router even when EZ-Bs are running. On the other hand, it is not a very configurable device compared to others I have tested. For instance, I can not specify the DNS provider I want my devices to use.

I experienced the issue on 3 different TP-Link routers and an ActionTech router (my older Fios Router), but the TP-Link at least are much more sophisticated devices, so I will just use the FiOS device for the robots on their own network, and continue using the TP-Link for the rest of my network bridged to the FiOS router to get internet access.

Alan

United Kingdom
#14  

I don't know if this is effecting the issues I'm currently having, but was there any word from the manufacturer of the tcp stack, as it has been 4 months now?

PRO
Synthiam
#15  

The manufacturer of the wifi chip used in the EZ-B v4 had communicated with additional detail about the DHCP. The DHCP on the wifi chip is delayed to "wait and see" if there is another dhcp server on the network. This is a feature used for portable networks where there is no dhcp and only a common access-point. Because this wifi chip is optimized for use in robotics, plc, iot and other portable network applications, it is common for these networks to not have dhcp servers. So the wifi IC delays and responds accordingly. Due to the slow processing speed of some older home routers, their dhcp server may respond slower and therefore trigger the dhcp server of the ez-b v4 wifi chip.

This is why ez-robot has been unable to reproduce the experience which alan had introduced in his thread - in both my home and office network. This instance has only arisen less than a handful of times on the forum, and has been resolved with router upgrades.

The DHCP delayed server does not affect communication performance. DHCP is the protocol used to issue IP addresses to devices and computers on a physical layered network. This means a network not separated by routers, such as your home network.

If the DHCP delayed server is responding due to the delayed response of the home network router, report your router model and version here to me with a description of the experience and i will compile a trouble ticket with the vendor for resolution.

Something to note, the wifi ic on the ez-b is being discontinued by the manufacturer, which may mean no further upgrades. Even though there will be plenty of surplus inventory of the wifi chip, ez-robot does not manufacture hardware with discontinued components - so ez-robot is changing the wifi chip for 2016. Due to the modular design of the ez-b v4, the communication board of the ez-b v4 can be swapped out with the new module when it is available - if the issue with delayed dhcp continues to cause issues.

If you want the DHCP completely disabled - and control over the wifi module code directly, you can re-flash the wifi module. All you need is one of these: http://www.ebay.ca/sch/i.html?_from=R40&_sacat=0&_sop=15&_nkw=pickit+3+clone&rt=nc&LH_BIN=1

If you purchase one of those PicKit 3 Clone Programmers, i can create a quick tutorial and provide the firmware source code which entirely disables DHCP on the chip. Alan, this applies to you or anyone as well. Unrelated to Alan's dhcp server experience, ez-robot has been thinking about releasing the webserver firmware open-source for customizing web UI and additional custom features. Perhaps this is a good time to do it. Get one of those pickit 3 clone programmers (the clone ones work great and are cheaper) and i'll get on pushing the wifi code on this website.

With the PicKit 3, the ez-b v4 wifi source code and webserver of the ez-b v4 can be modified for custom logo, html and branding. It'll be pretty awesome:D - I guess you can even change the TCP ports of the camera and ez-b protocol as well. Lots of power in that code release - including disabling dhcp entirely for station mode.

Lastly, if you believe the performance is far too unusable - there may be a defect in the manufacturing, which is unlikely but possible. That being said, feel free to Contact Us and send the ez-b v4 directly to our facility in Calgary, Alberta for review.

#16  

I just ordered one of the programmers. I did not expedite the shipping, so expecting in a few weeks. I'll be happy to be your "beta tester" for the tutorial.

Alan

PRO
Synthiam
#17  

Awesome - I'm nerding out and putting the package together tonight. While I wait for for my custom bb8 to print:)

Didn't like the bb8 builders version so I'm making my own. And i received a not-very-friendly message from one of their members. They seem to have InMoov fever by also not welcoming contributors... Anyway, my bb8 will be entirely 3D printable, unlike all the others which require purchasing a pile of hardware. Apologies for the off topic:)

PRO
Synthiam
#19  

Did you get the programmer? How's your new dhcp and open iot firmware running?:)

#20  

I got the programmer, but haven't had a chance to try it out yet. Maybe Sunday.

Alan

PRO
Synthiam
#21  

geez whiz! I figured you'd be alllllllllll over programming the v4!

#22  

I will be. This summer has been rough. My basement was flooded by a plumbing leak and it has been undergoing mold remediation and reconstruction for 5 months. All that is left is carpet install and the return of belongings that were not destroyed fron the Servpro warehouse (including virtually all of my tools and many robot parts).

Thankfully my Roli and boxbot were upstairs at the time, but I don't have a good workspace upstairs. When the work is done I'll have a fabulous workshop amd will finally get started on my big Steampunk Dogbot I have been talking about for 3 years.

Alan

United Kingdom
#23  

@Alan.

Hurray, I've been hoping you would get around to the steampunk build. I'm really pleased you have finally got your basement almost sorted. What a pain in the butt with that flood, but at least you'll have a shiny new workspace to play about in now.

#24  

For anyone who has been following this thread, a discussion started in https://synthiam.com/Community/Questions/8685&page=5 with post #47.

The upshot of it is that the PIK programmer that DJ discusses above is pretty easy to use, and the latest version of the firmware that you can load with it resolves the issue.

If there are members in the US who are nervous about flashing their EZ-Bs or just don't want to buy the PIK programmer, I would be happy to flash them for you for just the cost of return shipping the EZ-B back to you (although the programmer is so cheap, it would probably cost you less just to buy one).

Only tricky part is that the header pins that came with the PIK are not a tight fit in the EZ-B, so you need to hold it tightly in place while programming.

Alan

#26  

Thanks for the update. Very good instructions.

All users, be sure to read the instructions.txt in the archive as well as the instructions in the Open WiFi IoT tutorial.

Alan