more G-Labs products

Author Topic: Possible Memory Leak ?  (Read 1572 times)

April 03, 2015, 11:02:37 AM
Read 1572 times

Void

  • *
  • Information
  • Newbie
  • Posts: 18
Hi !

I've installed 484 to check if an issue with my instalation gets solved.

After 482 installation, i saw hg's proccess being killed by OS due an out of memory error (Server is a VM with 1Gb of RAM). I've created a watchdog script to detect if hg is not running, and start it again.

I've added an snmp monitor to the server and cacti to recover info and create graphs.

After a few weeks of monitoring i'm getting this charts (Attached).

This looks like a memory leak bug in my installation (Or in some of my automation programs), HG memory increases until an out of memory kills him, the watchdog restarts HG, CPU is most of the time over 90% and mono's memory consumption increases until the next kill.

I've tryied disabling all my programs in HG, reseting DB, etc, without success. The mono's proc increases memory usage until the OS kills him due an out of memory.

The server is running Debian 7... i will try a fresh install with Ubuntu (But i will preserve the VM for testing pourposes if somebody needs to check something)

Cheers!
« Last Edit: April 03, 2015, 11:05:14 AM by Void »

April 03, 2015, 04:59:45 PM
Reply #1

bkenobi

  • *****
  • Information
  • Global Moderator
  • Posts: 1525
I haven't noticed any issues on RPi r484 so far.  This type of monitoring was what I was hoping to do a while back but couldn't figure out an approach.  If I get a chance, I'll try to generate a similar chart.

Looking at Cacti, it indicates that several other applications are needed to run to tool.
Quote
Cacti requires MySQL, PHP, RRDTool, net-snmp, and a webserver that supports PHP such as Apache or IIS.
Does the web server HG uses qualify for this (not sure which HG uses)?  I don't want to run multiple web servers, so I'm hoping that the one HG already has installed would work here.  Another thought I had was to see if we could harvest the CPU/memory data and plot it within HG's plotting tools already available.  This actually seems like a really nice feature that should be included anyway to know the health of the HA server.
« Last Edit: April 03, 2015, 05:14:10 PM by bkenobi »

April 04, 2015, 01:04:51 PM
Reply #2

Void

  • *
  • Information
  • Newbie
  • Posts: 18
Cacti was installed in another server to monitor all my home's systems. The problem with HG's web server is the lack of mysql and php support (But you can use php-cli with mysql via a wrapper in HG's WS).

Another aproach is to create an automation program that reads values from /proc and stores data in the HG's DB (Or shows it in a widget).

Something like this could be very interesting in HG for testing pourposes. :D

Another interesting feature could be a mono monitor, to detect threads with huge resource consumption or other bugs.

After server's reinstall, mono's proc consumes low cpu and memory resources, but after a restore of my backup, mono starts to consume a lot of resources again.

Some automation program could be the responsible of this. I will investigate in deep disabling again everything.

Cheers!

May 30, 2015, 05:40:27 AM
Reply #3

mikeH

  • **
  • Information
  • Jr. Member
  • Posts: 27
I'm currently running on a 14.04 VM with 1.5GB of memory allocated.  I run out of memory (mono seems to be grumpy) about once every two days.  Prior to this on an RPi, I never experienced an OOM error in roughly 2 weeks of use.  I'll keep an eye on this as well and post follow ups.

May 30, 2015, 09:42:10 AM
Reply #4

Gene

  • *****
  • Information
  • Administrator
  • Posts: 1472
  • Tangible is the future!
    • Yet Another Programmer
always ensure to be using latest mono runtime:

http://www.mono-project.com/docs/getting-started/install/linux/

also, it could depend on the code used in your custom apps.

Cheers.
g.

January 05, 2016, 11:31:07 PM
Reply #5

Void

  • *
  • Information
  • Newbie
  • Posts: 18
Solved (Sorry for the delay)

After upgrading mono and HG to r510 everything is working like a charm  ;)

(Upgrade was at 20:00 and used memory is constant)