publishing mqtt message issue

[Wibo]

Good to hear!

I'm using this line to send messages rigth now:
_mqttClient.Publish(topic, Encoding.UTF8.GetBytes(message), MqttMsgBase.QOS_LEVEL_AT_LEAST_ONCE, true);

Can you confirm you want Encoding.Unicode here?

Regards,
Wibo.

[Gene]

Yes this is right.

g.

[Wibo]

FYI:

I have a working Windows.Forms testapp on RPi now, some creative programming was needed since I couldn't use the textbox.

The currently used Nmqtt was definitely not sending messages in Unicode.
The Mosquitto monitoring tool I use inserted spaces for every char when I started to use Unicode.
I'll investigate this too, it could become a backwards compatibility issue

Going to make the testapp more mature and add receiving features...

[Gene]

Perhaps Encoding.UTF8 is a better choice then. I've already had troubles with the web server and unicode encoding. So go for UTF8.

Code: [Select]

_mqttClient.Publish(topic, Encoding.UTF8.GetBytes(message), MqttMsgBase.QOS_LEVEL_AT_LEAST_ONCE, true);

g.

[Gene]

reading this (the UPDATE remark) I also wonder what is the real deal between the two? 
http://stackoverflow.com/questions/3951722/whats-the-difference-between-unicode-and-utf8

Are we going to lose accented/extended characters or so? These are most likely not going to be used in module's parameters.

ADD:
http://www.rrn.dk/the-difference-between-utf-8-and-unicode

g.

[Wibo]

Hi Gene,

I'll have to do some reading what's the best for HG.
At work I use Unicode, it supports Poland, Greece, Hungary, Denmark, etc...

To be continued...

Regards,
Wibo

[Wibo]

Bad news about M2Mqtt...

It can send really fast.
All messages are arriving at the broker.
All messages can be logged with the mosquitto_sub tool

M2Mqtt receiving is totally different, it starts loosing messages when I stress it a little.
100 messages in burst per second is the limit, without noticeable CPU load, network is fine too.
Tested a lot of setups, they all break, unless you go to snail speed.

Going to do tests with MqttDotnet.
Fortunately I already have a nice test program.

Regards,
Wibo.

[Gene]

keeping finger crossed 

[Wibo]

Hi Gene,

Lets start with the bad news:
ExactlyOnce mode blows up in your face (That would be the guaranteed delivery)
Last commit in svn was a target change from framework 4.5 to 2.0
It took some time to improve my test app, so it wouldn't have influence on performance.

Now the good news:
AtMostOnce mode was so stable and fast I had to improve my test app.
All messages arrived no matter what.
When stressing the only thing was CPU load.
19000 messages per second between two win7 boxes running in vmware.
440 messages per second between RPi B+ and vmware win7 box.
Details follow in the report.

Gene need to know a few things...
I'm going to test against at least framework 4.0, since the image has no standard 2.0 support.

HG is targetted for 4.0 do you have any plans to target 4.5?

What mode do you prefer since ExactlyOnce is not working?
Testing right now with AtMostOnce (no retries)
HG uses AtLeastOnce (build in retries)

Do you want homegenie to control the retry or a library?
Order of arrival will be disturbed when the library is doing it too.

Still have a lot of testing to do, but at least I'm getting good results.
Gave up on the guaranteed delivery, TCP is good enough for me.
Think this is a heritage from other protocols like RS232 and alikes.

Regards,
Wibo.

[Gene]

Hi Wibo,

HG will keep targetting 4.0 for now. But this may change at some point.
The library should take care of the message delivery and retrying.

g.

[Wibo]

Thx,

Already started testing against 4.0 yesterday without having trouble, speed is a bit lower.
I'll switch to AtLeastOnce for the testing when I have some measuring data with 4.0 to compare.
Hope I can finish testing this weekend...

Regards,
Wibo.

[Wibo]

Hi Gene,

started testing with AtLeastOnce and again I have an unreliable connection.

I think BestEfforst(AtMostOnce) is the only way to go.
None of these libraries seem to have reliable multithreaded code when stressed.
Except this library when in AtMostOnce mode (no retries)

Using it like this means handling retries in HG.(But it never looses a message)
We probably also have to do a reconnect after lost connection.(Haven't investigated this yet)

Test report so far attached.
Penny for your thoughts.

Regards,
Wibo.

[Wibo]

I went back testing a few HG things.

I you use the program "Sensor.* events forwarding" for communication it's self healing.
unplug network, boot device, it doesn't matter just fix the problem and everything start working again.
This is a very powerful feature!

Using the MQTT interconnection is totally different.
Disrupt anything and the system dies until restarted.

I'm playing with the idea to write some code on top of MqttDotnet.
It could use the non retry mode of MqttDotnet.
Have a simple retry (once or twice non queued)
Have an automatic reconnect making it self healing.

All those libraries seem to have forgotten where the Q stands for.

I think MQTT is a really nice protocol and could be very useful without the Q working.

Please let me know what your ideas are about this issue...

Regards,
Wibo.

[Gene]

Hi Wibo,

I'm thinking to add a TcpServer helper class for hg automation programs. So perhaps we could try writing an automation program that implement a custom solution with message queue/retry mechanism.
The code of the TcpServer class is already "hidden" in MIG project, so adding the helper class should be straight forward.

g.

[Wibo]

Hi Gene,

writing a queue/retry/auto reconnect mechanism means at least one worker thread.
Doing this in a scripted automation program seems like a no go to me.
The scripts are very hard to debug, resource management is hard, I would do only simple things.
Implementing such a system should really be in the HomeGenie code base in my opinion.

I don't have enough knowledge of the MIG code yet, so hope you can come up with something that works.

Right now I really need some successes (Spend 3 weeks testing 3 mqtt client libraries without getting anywhere).
Going to focus on getting my hardware online and start logging.
I have to do this with my own software anyway, for several reasons.

If I can do some testing with a new approach just ask.
Mqtt was very nice because you could test it isolated from HomeGenie.
The client/broker architecture was also a very nice feature and Mosquitto is stable as a rock.
I could even have used the same broker for my own communication requirements if HG wasn't using a wildcard topic.

I need some time to think this over.

Regards,
Wibo.