HomeGenie Forum
General Category => Troubleshooting and Support => Topic started by: Wibo on January 26, 2015, 09:29:55 PM
-
Hi Gene,
could use some help here.
When I started reliability tests all kinds off issue came by.
Would like to start with the sending HG box.
Nmqtt is throwing exceptions as soon as I generate more then one sensor event per second.
When I let 4 sensors generate an event every second it's party time.
It's not always happening, it happened that I had to wait 30 min.
The log file is not showing the exception, you need the console.
I have tested this on RPi and win7, Rpi from command line and win7 as debugging session.
They both have the problem, although RPi is more sensitive.
CPU load on RPi and even Win7 can raise above 50% when Nmqtt is not happy.
Despite all the problems above no events seem to get lost when I'm monitoring Mosquitto broker.
Setup:
Rpi B+ based on image 431, /usr/local/bin/homegenie replaced by linux build r460 bin/debug
Win7 running r460 from VS
Mosquitto mqtt broker running on win7
Everything was down except one broker and one HG box only sending mqtt with FakeTemperatureCluster.
Already saw you're on r466 now and have made new images and installs.
As opposed to the last fix you made for me, I can now test a new revision very fast.
Attached the hgx, console and log output from a RPi session.
As a bonus an interesting one that could indicate multithreading issues.
Could start researching Nmqtt unless you have a better idea.
best regards,
Wibo.
-
Hi Wibo,
yes it could be a multithreading issue. If so add a "lock" instruction in the MQTT client app at line 140 like shown below:
// We want to do further processing whenever a module changes
When.ModuleParameterChanged( (module, property) => {
if (module.HasFeature("MQTT.SensorPublish") && (property.Name.StartsWith("Sensor.") || property.Name.StartsWith("Status.") || property.Name.StartsWith("Meter.")) && !module.Instance.Domain.StartsWith("MQTT:"))
{
lock(this)
try{
Net.Publish(clientid + "/" + module.Instance.Domain + "/" + module.Instance.Address + "/event", Newtonsoft.Json.JsonConvert.SerializeObject(property));
} catch (Exception e){ Console.WriteLine("ERR:"+e.Message + "\n" + e.StackTrace); }
}
// returning true, will route event to other listeners
return true;
});
then click "compile" to makes changes effective.
Cheers,
g.
-
Hi Gene,
some additional info:
I use the hgx only to generate events in the standard way by RaiseEvent.
All the rest is HG.
Activated mqtt
I publish the modules to mqtt and that's it.
You really want me to bypass the HG module publishing system and send mqtt message myself like your example is suggesting?
best regards,
Wibo.
-
Nmqtt seems to be a dead project, last activity in 2013...
-
Wibo,
I'm suggesting to modify the MQTT client program in HG automation section.
Not to write a new one.
- Go to Configure->Automation->Interconnections
- Click on the MQTT Network program
- Click Edit and go to Program Code tab
- insert a lock(this) instruction at line 140 (like shown in the snippet of my previous message)
- click Compile
g.
-
Hi Gene,
sorry I misunderstood.
I followed instructions and replaced the When.ModuleParameterChanged snippet.
Recompiled the MQTT client app in Configure->Automation->Interconnections.
Restarted HG, checked if the code change was still there.
Same results in the Console for RPi, duplicate key exception is still thrown and RPi is really slow.
Won't have more time this evening for testing.
Regards,
Wibo.
-
Hi Gene,
FYI:
Made a windows forms test app for Nmqtt
No fancy code, just trying to reproduce the exception.
and yes, it's reproducible!!!
This is a Nmqtt problem, going to test some more...
Think I'll postpone my update to r466.
Regards,
Wibo.
-
Thanks for looking into this, let me know your findings about possible nmqtt fixes.
There's r471 now available, in case you want to help testing or use latest source code for debugging.
Cheers,
g.
-
Hi Gene,
hope to post my testing report tomorrow.
Nmqtt is a very weird library.
It only seems to work in one mode and that's not the QOS guaranteed delivery.
I can accomplish very high bandwidth in the MqttQos.AtMostOnce.
The library throws exceptions on Dispose and on much more it should handle itself.
I'm not looking forward to download the source and fix it, since it's a dead project since 2013.
Is it an option to switch to another living DotNet MQTT implementation?
Regards,
Wibo.
-
Hi Wibo,
any other working MQTT implementation is welcome.
Cheers,
g.
-
Hi Gene,
good to hear there's no sentimental value on nmqtt.
The library is very buggy and your retry mechanism is making it worse. (Retrying = burst)
There's nothing wrong with your retry mechnism, it's nmqtt that's crappy.
We all want guaranteed delivery for our messages without eating all CPU cycles.
nmqtt just can't do that without being fixed.
The receiving code has simular problems, but I didn't test as thoroughly.
Test report is attached, going to do research on alternatives...
Regards,
Wibo.
-
Hi all,
found this untill now...
Nmqtt:
Last activity 2013 and buggy
MqttDotNet: http://www.stevenlovegrove.com/?id=37 (http://www.stevenlovegrove.com/?id=37)
Source: https://github.com/stevenlovegrove/MqttDotNet (https://github.com/stevenlovegrove/MqttDotNet)
Last activity july 2014
M2MQTT: https://m2mqtt.codeplex.com/ (https://m2mqtt.codeplex.com/)
Last activity january 2015 (forum)
Last release december 2014
It has an Eclipse Public License (EPL)
KittyHawkMQ: http://www.kittyhawkmq.com/ (http://www.kittyhawkmq.com/)
Not open source
=================================
M2MQTT seems interesting, going to look at the source.
If there's anyone who used/researched any of the above projects, please let me know.
Regards,
Wibo.
-
M2MQTT Source seems to be ok.
support for mqtt QOS
support for windows phone, mono and lots of other things
Code is full of ifdefs though like this:
#if !(MF_FRAMEWORK_VERSION_V4_2 || MF_FRAMEWORK_VERSION_V4_3 || COMPACT_FRAMEWORK || WINDOWS_APP || WINDOWS_PHONE_APP)
Going to test it...
Regards,
Wibo.
-
Hi Gene,
first tests with M2MQTT are very promising all QOS modes are working.
It's very stable when stressing it, but I need better testing abilities at high bandwidth.
Have to make a testprog now that works on RPi and win7.
Also have to make a testprog for the receiving side, so I can test if any messages are lost at high bandwidth.
Will be busy for a while to see how good this library is...
Regards,
Wibo.
-
Hi Wibo,
thanks for checking this out. If your tests give an happy result, I will change the NetHelper class so to use m2mqtt that will then replace nmqtt.
Cheers,
g.
-
Good to hear!
I'm using this line to send messages rigth now:
_mqttClient.Publish(topic, Encoding.UTF8.GetBytes(message), MqttMsgBase.QOS_LEVEL_AT_LEAST_ONCE, true);
Can you confirm you want Encoding.Unicode here?
Regards,
Wibo.
-
Yes this is right.
g.
-
FYI:
I have a working Windows.Forms testapp on RPi now, some creative programming was needed since I couldn't use the textbox.
The currently used Nmqtt was definitely not sending messages in Unicode.
The Mosquitto monitoring tool I use inserted spaces for every char when I started to use Unicode.
I'll investigate this too, it could become a backwards compatibility issue
Going to make the testapp more mature and add receiving features...
-
Perhaps Encoding.UTF8 is a better choice then. I've already had troubles with the web server and unicode encoding. So go for UTF8.
_mqttClient.Publish(topic, Encoding.UTF8.GetBytes(message), MqttMsgBase.QOS_LEVEL_AT_LEAST_ONCE, true);
g.
-
reading this (the UPDATE remark) I also wonder what is the real deal between the two? ;D
http://stackoverflow.com/questions/3951722/whats-the-difference-between-unicode-and-utf8 (http://stackoverflow.com/questions/3951722/whats-the-difference-between-unicode-and-utf8)
Are we going to lose accented/extended characters or so? These are most likely not going to be used in module's parameters.
ADD:
http://www.rrn.dk/the-difference-between-utf-8-and-unicode (http://www.rrn.dk/the-difference-between-utf-8-and-unicode)
g.
-
Hi Gene,
I'll have to do some reading what's the best for HG.
At work I use Unicode, it supports Poland, Greece, Hungary, Denmark, etc...
To be continued...
Regards,
Wibo
-
Bad news about M2Mqtt...
It can send really fast.
All messages are arriving at the broker.
All messages can be logged with the mosquitto_sub tool
M2Mqtt receiving is totally different, it starts loosing messages when I stress it a little.
100 messages in burst per second is the limit, without noticeable CPU load, network is fine too.
Tested a lot of setups, they all break, unless you go to snail speed.
Going to do tests with MqttDotnet.
Fortunately I already have a nice test program.
Regards,
Wibo.
-
keeping finger crossed :)
-
Hi Gene,
Lets start with the bad news:
ExactlyOnce mode blows up in your face (That would be the guaranteed delivery)
Last commit in svn was a target change from framework 4.5 to 2.0
It took some time to improve my test app, so it wouldn't have influence on performance.
Now the good news:
AtMostOnce mode was so stable and fast I had to improve my test app.
All messages arrived no matter what.
When stressing the only thing was CPU load.
19000 messages per second between two win7 boxes running in vmware.
440 messages per second between RPi B+ and vmware win7 box.
Details follow in the report.
Gene need to know a few things...
I'm going to test against at least framework 4.0, since the image has no standard 2.0 support.
HG is targetted for 4.0 do you have any plans to target 4.5?
What mode do you prefer since ExactlyOnce is not working?
Testing right now with AtMostOnce (no retries)
HG uses AtLeastOnce (build in retries)
Do you want homegenie to control the retry or a library?
Order of arrival will be disturbed when the library is doing it too.
Still have a lot of testing to do, but at least I'm getting good results.
Gave up on the guaranteed delivery, TCP is good enough for me.
Think this is a heritage from other protocols like RS232 and alikes.
Regards,
Wibo.
-
Hi Wibo,
HG will keep targetting 4.0 for now. But this may change at some point.
The library should take care of the message delivery and retrying.
g.
-
Thx,
Already started testing against 4.0 yesterday without having trouble, speed is a bit lower.
I'll switch to AtLeastOnce for the testing when I have some measuring data with 4.0 to compare.
Hope I can finish testing this weekend...
Regards,
Wibo.
-
Hi Gene,
started testing with AtLeastOnce and again I have an unreliable connection.
I think BestEfforst(AtMostOnce) is the only way to go.
None of these libraries seem to have reliable multithreaded code when stressed.
Except this library when in AtMostOnce mode (no retries)
Using it like this means handling retries in HG.(But it never looses a message)
We probably also have to do a reconnect after lost connection.(Haven't investigated this yet)
Test report so far attached.
Penny for your thoughts.
Regards,
Wibo.
-
I went back testing a few HG things.
I you use the program "Sensor.* events forwarding" for communication it's self healing.
unplug network, boot device, it doesn't matter just fix the problem and everything start working again.
This is a very powerful feature!
Using the MQTT interconnection is totally different.
Disrupt anything and the system dies until restarted.
I'm playing with the idea to write some code on top of MqttDotnet.
It could use the non retry mode of MqttDotnet.
Have a simple retry (once or twice non queued)
Have an automatic reconnect making it self healing.
All those libraries seem to have forgotten where the Q stands for.
I think MQTT is a really nice protocol and could be very useful without the Q working.
Please let me know what your ideas are about this issue...
Regards,
Wibo.
-
Hi Wibo,
I'm thinking to add a TcpServer helper class for hg automation programs. So perhaps we could try writing an automation program that implement a custom solution with message queue/retry mechanism.
The code of the TcpServer class is already "hidden" in MIG project, so adding the helper class should be straight forward.
g.
-
Hi Gene,
writing a queue/retry/auto reconnect mechanism means at least one worker thread.
Doing this in a scripted automation program seems like a no go to me.
The scripts are very hard to debug, resource management is hard, I would do only simple things.
Implementing such a system should really be in the HomeGenie code base in my opinion.
I don't have enough knowledge of the MIG code yet, so hope you can come up with something that works.
Right now I really need some successes (Spend 3 weeks testing 3 mqtt client libraries without getting anywhere).
Going to focus on getting my hardware online and start logging.
I have to do this with my own software anyway, for several reasons.
If I can do some testing with a new approach just ask.
Mqtt was very nice because you could test it isolated from HomeGenie.
The client/broker architecture was also a very nice feature and Mosquitto is stable as a rock.
I could even have used the same broker for my own communication requirements if HG wasn't using a wildcard topic.
I need some time to think this over.
Regards,
Wibo.
-
Thinking is good.
Btw what do you mean hg is using a wildcard topic? Topic can be configured.
g.
-
Hi Gene,
when HG subscribes it subscribes with #, it wants to see every message.
I know the current design needs it this way.
Let's say I want to use the same broker with an unique topic and send 5000 msg/s.
Every subscribed RPi and other weaker brothers will die.
It's not that I want to send so much data, it's just that I can remotely kill an RPi running HG just by sending a lot of messages to an mqtt broker.
Regards,
Wibo.
-
I understand that HG or whatever application subscribing # would receive lot of messages and so subject to flooding.
That's why I'm telling you that you can change the topic ;)
Cheers,
g.
-
Hi Gene,
just a status update...
Writing my own software to accomplish my goal.
Later on I can start communicating with HG.
I'm still using MqttDotnet and it has surprising features.
I'm still in a early alpha phase, let you now when I have results.
Regards,
Wibo.
-
Hi Gene,
have the results of the Dutch jury.
In the beginning the results were amazing, I had MqttDotnet running in my own project.
After some serious testing on win7 and writing some extensions like auto reconnect, I could disconnect network cables stop and start mqtt broker. Disconnecting network cables for not to long (30s) didn't result in lost messages, killing the broker did result in lost messages, but I can live with that.
The whole setup was self healing, auto reconnecting, etc.
I started testing on RPi and the results where dramatic.
After getting my own bugs out (thrown exceptions differ under mono) the MqttDotnet library acted terribly when disconnected. disconnecting network cables long enough or taking down the broker resulted in:
Not able to reconnect, unless the process was restarted.
Having a workerthread deep in MqttDotnet going nuclear eating all CPU cycles from one core untill the process was restarted.
While investigating the possible cause I saw to many TODO: in the multithreading area and other disturbing code.
Given up on MqttDotnet...
Spent 2 days to see if Microsoft Azure could be something, but I have a feeling that won't be a winner on RPi
Investigating ZeroMQ right now...
http://zeromq.org/intro:read-the-manual (http://zeromq.org/intro:read-the-manual)
Let you know the results...
Regards,
Wibo.
-
Hi Gene,
I dropped ZeroMQ after playing with it for a week.
In fact there were 2 dotNet libraries and one of them was unstable.
The other one clrzmq was very stable, very modular, very mono/RPi compatible.
The problem was that I had to write to much code and needed a very good understanding of the library design. I didn't want to spend so much time in writing a simple broker and implementing heart-beat and reconnect code.
So past week I researched RabbitMQ.
This one seems promising!
I already have more features and more stability in a week then with any of the 4 other libraries I tested.
Last night it sent 50 million messages at a rate of 1000msg/s without loosing any message.
It's still premature to say RabbitMQ is the best message queue solution for dotNet that also works on RPi.
I'm going to use the Rabbit dotNet client in my software, until now I only used it from a test program. Really hope this will be the one, already spend to much time on communication.
I'll let you know the results...
Summary of RabbitMQ attached
Regards,
Wibo.
-
Wondering how this turned out with RabbitMQ, Started using MQTT for my hg setup to communicate between hg system but after adding a couple of devices it started to throw exceptions "An element with the same key already exists in the dictionary." So the NMQTT isn't working very well and it restarts the connection a couple of times each minute.