Out of memory during build of large Computed Table
We're running out of memory when building large computed tables. Changing the Data Import Chunk Size (https://support.sisense.com/hc/en-us/articles/230646588-Out-of-Memory-Error-During-Build) doesn't appear to be making a difference. We're currently using 7.0.2.10036.
Is there any way to get extremely large computed tables to build successfully, without using extremely large amounts of server memory?
Any and all help would be greatly appreciated.
Thanks,
John
-
Some further details...
The largest of these tables currently contains about 120 GB of data, split between about 60 Million rows @ ~2 KB per row. Our single Sisense server has (Edit, memory mis-quoted as 14 GB before) 28 GB of memory.This table is expected to grow much larger over time.
-
Hi Malinda,
Unfortunately, I am unable to do this. The Custom table in Sisense is already a workaround to allow for accumulative build of the non-static source data.
Moving it back to SQL Server defeats the purpose of the custom table, which is to calculate the latest version of the accumulated source data using rowversion and the primary key on the source table data, with the intention of filtering invalid data.
Oddly enough, the source data is already a view...
Thanks,
John
-
John,
If you are going to use this as your ElastiCube server, you'll probably want to increase the amount of memory anyway for queries. According to the minimum requirements (https://documentation.sisense.com/minimum-requirements-and-supported-platforms/), depending on the number of users you have, you will want 32-64G of memory. For our high availability deployment, we have 2 cube servers, each with 486G of memory. We also have a build server (actually resides on our SQL Server for our warehouse). Our largest cube has been as large as 450G and we have around 100 concurrent users every day with more than 4,000 users overall.
Malinda
-
Not sure if this could help, but check this:
https://documentation.sisense.com/latest/managing-data/connectors/out-of-memory.htm#gsc.tab=0
Please sign in to leave a comment.
Comments
10 comments