more G-Labs products

Author Topic: Analyse and backup/restore  (Read 1736 times)

March 14, 2015, 09:56:51 PM
Read 1736 times

nolio

  • *****
  • Information
  • Global Moderator
  • Posts: 544
Hi,
After a lot of install, reinstall of HG, i can easily backup and restore my configuration. But on the statistic/analyse part, it's not so easy.

The most time i don't care about save my statistic or the "udapte" method permit to keep. I think, i already try to replace the new "homegenie_stats.db" by my old one and it works. But on r482, i try my r480 db and my HG bug ... (i need to reinstall).

So now my question, do you have a clean method to backup and restore this db file ? (without the risk of crashing).
Bye

March 17, 2015, 07:05:00 PM
Reply #1

hybridview

  • **
  • Information
  • Jr. Member
  • Posts: 37
Large amounts of historical data are not very useful in the current stats UI in HG, and it can also impact performance. Therefore, stats data is treated more like "temporary" or disposable data. However, it would be pretty useful to keep this data for use in an application better suited for data analysis, such as Excel.

I was personally thinking of building a quick method to export the DB to some format (like csv, json, etc) so it can be analyzed in another application. Maybe something like this could be leveraged to provide export + import functionality.  In the end, it all comes down to how much free time the developers have, the difficulty of the feature to implement, and the user demand of said feature.

For import, here are some possible issues:
* You might need to implement de-duplication functionality in case user uploads duplicate or overlapping data.
* Change of DB schema across HG update: If schema changes, data will no longer be compatible. At work, my solution to a similar problem was to embed the data schema version in the export file. During import, I call custom-built "transformations" based on the version found in the file. These transformations convert the imported data to a compatible format (if possible). In my code, any time I change the DB schema, I add a new transform function.
* Impact of large import on running HG app: Due to the demands of such a process and the potential issues of importing huge datasets while the app is running, it might be a safer idea to build this kind of function into the HG service launch/manager application. You would need shell access to the HG server, but at least the service could be halted during import and started back up after.
* Historical data less useful in current stats UI: It might not make much sense to even support import at this time. In the future, if the stats UI is changed to support more analysis and/or going back in time to see previous days consumption (not just averages), import would be much more significant.

For export, here are the issues that must be addressed:
* Memory consumption: I'm not 100% sure how SQLLite handles concurrency (DB writes during long DB read), but I assume it just loads the query result entirely into memory where it can be read later without keeping DB cursor open. If you are exporting a 50MB database, this could cause a memory issue, especially on devices like RaspberryPi.

March 17, 2015, 10:01:55 PM
Reply #2

nolio

  • *****
  • Information
  • Global Moderator
  • Posts: 544
I don't think about it, but you are probably right about the disposable of this data for now.
I just think it can be useful to see the evolution of temperature in a room this year and the year before ...
About the export, i just try to do
  • a copy of the "homegenie_stats.db" file
  • try to open with "SQLLiteBrowser"
  • Export in CSV
But the data are like this :
Code: [Select]
TimeStart TimeEnd Domain Address Parameter AverageValue
2015-03-17 xx:48:54.230211 2015-03-17 20:03:53.917944 HomeAutomation.ZWave xx Sensor.Humidity xx.x
2015-03-17 xx:48:54.230211 2015-03-17 20:03:53.917944 HomeAutomation.ZWave xx Sensor.Luminance x.x
2015-03-17 xx:48:54.230211 2015-03-17 20:03:53.917944 HomeAutomation.ZWave xx Sensor.Temperature xx.x
2015-03-17 xx:03:53.916617 2015-03-17 20:03:53.917944 HomeAutomation.ZWave xx Sensor.Power x.x
2015-03-17 xx:03:53.916617 2015-03-17 20:03:53.917944 HomeAutomation.ZWave xx Sensor.Power x.x
2015-03-17 xx:03:53.916617 2015-03-17 20:03:53.917944 HomeAutomation.ZWave xx Meter.KilowattHour x.x
2015-03-17 xx:03:53.916617 2015-03-17 20:03:53.917944 HomeAutomation.ZWave xx Sensor.Power x.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.FeelsLikeC xx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.FeelsLikeF xx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.1.Day xx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.1.TemperatureC.High xx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.1.TemperatureC.Low x.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.1.TemperatureF.High xx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.1.TemperatureF.Low xx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.1.Year xxxx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.2.Day xx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.2.TemperatureC.High xx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.2.TemperatureC.Low x.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.2.TemperatureF.High xx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.2.TemperatureF.Low xx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.2.Year xxxx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.3.Day xx.x
2015-03-17 xx:38:53.917516 2015-03-17 20:08:53.9182xx HomeAutomation.HomeGenie.Automation xx Conditions.Forecast.3.TemperatureC.High xx.x

So it can be hard to use in excel for doing graph ...

And last SQLLiteBrowser permit to import CSV or SQL.

March 18, 2015, 09:31:02 PM
Reply #3

nolio

  • *****
  • Information
  • Global Moderator
  • Posts: 544
I try to manipulate data by command line sqlite3.
Export
Code: [Select]
sqlite3 homegenie_stats.db
SQLite version 3.8.6 2014-08-15 11:46:33
Enter ".help" for usage hints.
sqlite> .mode list
sqlite> .separator ,
sqlite> .output export.txt
sqlite> select * from ValuesHist;
sqlite> .exit
Data
Code: [Select]
2015-03-17 19:48:54.230211,2015-03-17 20:03:53.917944,HomeAutomation.ZWave,xx,Sensor.Humidity,xx.x
2015-03-17 19:48:54.230211,2015-03-17 20:03:53.917944,HomeAutomation.ZWave,xx,Sensor.Luminance,x.x
2015-03-17 19:48:54.230211,2015-03-17 20:03:53.917944,HomeAutomation.ZWave,xx,Sensor.Temperature,xx.x
Import
Code: [Select]
sqlite3 homegenie_stats.db
SQLite version 3.8.6 2014-08-15 11:46:33
Enter ".help" for usage hints.
sqlite> .mode csv
sqlite> .import export.txt ValuesHist
sqlite> .exit

The data are impoted despite the value are already in the db .... But it can be a way to insert some old data if HomeGenie manage the data which are imported ....

March 23, 2015, 02:18:58 PM
Reply #4

hybridview

  • **
  • Information
  • Jr. Member
  • Posts: 37
Thanks for the info about import/export with SQLite. Very useful!

I forgot to mention that I currently have a backup/restore process that I follow before making any upgrades to the HG system. I simply shut down the service, make a copy of the stats DB to another folder, then turn on the HG service and perform my updates. That way, if I have a problem with the DB, I can turn off the service and copy the backup back to the HG folder.