Hi all,
is there a way to implement a linear regression on the logger CR1000X? We have tried to implement the explicit formula of the linear regression, but this has lead to a memory overflow. The memory overflow occured where the sums (totalrun) were calculated of one variable times another variable.
Has anyone faced this issue before or has an idea?
Thanks in advance!
Have you tried to reduce data accumulation period? Using a moving window can limit the number of data points used in the regression calculation to a manageable amount. This means that only the most recent points are used, reducing the required memory while maintaining responsiveness to recent data changes.
you can find more help here: https://stackoverflow.com/friday night funkin'
Hello, I am also having the same problem as you and have not found a solution yet, I am also here waiting for everyone's answer.
Hi,
I have further worked on this (colleague of the original post author).
Have tried the following approach:
-abandoned the idea of doing a "TotalRun" on the variable in the buffer. Instead, reading the 10min resolution values directly from the 10min data table with the "TableName.FieldName" instruction
- introduced buffer array with the number of fields needed (432 for 3 days backwards; 3 days * 24 hours * 6 intervals per hour = 432).
- every 10 min, the array is filled with the 432 most recent values from the 10min table
- in the same run, the products and squares of values in this buffer are calculated and written into another array reserved for these, so that I get two more arrays for the square and the product (the "sum of x*y" and "sum of x²" needed for the linear regression)
Until here, this works fine. The arrays are filled every 10 minutes. I also tested it before with 1min values and only 20 entries.
Next step, I want to build the sums over the entire arrays. This returns "7999" for the summation of products and squares, which is the overflow error code. These variables are declared as double and should therefore be sufficient to sum up 432 values.
One important info to understand this problem: We are working with solar radiation (Global Horizontal Irradiance) values here, trying to automatically calibrate one sensor of lesser quality to another one of better quality in real time by using a linear regression based on their deviation. Hence, the values of the variables we are working with here are values of up to ~1300 (W/m²), and their product/square hence is up to 1.7 Mio.
432 * 1.7 Mio is less than ~1.7*10^308 which is the range of a double, so I do not quite understand why the variable overflows.
Is there a limitation to how large the numbers the CR1000x can handle, other than the theoretical range of a double?
Are you storing your data as FP2 in the table? Try Long or IEEE4/8.
https://help.campbellsci.com/crbasic/cr6/Content/Instructions/sample.htm