Accumulation: Imported Elasticube?

Comments

7 comments

  • Avatar
    Michael Guidone

    I experienced the same sort of dilemma and I tried the same approach. In out case it 70 million+ rows. At the time the elasticube data connector did not support accumulative builds (and I do not think that has changed). We found that a more sane approach to reloading that data in multiple cubes is to schedule an earlier cube build that uses that data, export the needed tables from that cube to CSV. Then import from CSV in the subsequent cubes.  With this approach you can do the accumulative build on the first cube, then only do CSV imports on the others. 

     

    https://support.sisense.com/hc/en-us/community/posts/115008222267-Export-Tables-To-CSV-Post-Plugin

     

    There is a newer version of the plugin for 7.1

     

     

     

    1
    Comment actions Permalink
  • Avatar
    Roopa Raghavan

    @Michael - Have a question for you. EC2EC still can't accumulate data and we are dealing with 40mil+ rows and I was looking for a way to accumulate from one cube to other. SiSense support confirmed that this still isn't a feature, so I'm exploring other possibilities. I have tried using the plugin(exposttoCSVpostplugin) in our dev environment and it works.

    Now question is - we have an automated process set up right from loading data into our sql tables, triggering the first cube build(accumulated build) and second cube build which takes longer as data source is the first elasticube.

    If I were to use this plugin in our production and each day it stores a new (updated) CSV file in the path given, how do you set up elasticube build for the (second) cube to pick up the latest CSV file from the file path? I have seen that the plugin creates a new file timestamped each time cube builds.

    How did you solve this for your purposes?

    Thanks for your inputs and help!

    Roopa

    0
    Comment actions Permalink
  • Avatar
    Michael Guidone

    The post build plugin has bugs. It is destructive to certain data types in the export. We have abandoned using Sisense for any complex data processing in the backend. 

    We were using the csv import feature of being able to import all CSV from a specific directory. We were dropping the exported csv files to the import folder used by another table in another cube build.

     

     

    1
    Comment actions Permalink
  • Avatar
    Roopa Raghavan

    @Michael, now you've piqued my curiosity. What kind of bugs did you see and what datatypes were affected?

    Anotehr ques - Are you saying you are able to set a folder/directory as input data source for an elasticube?I didn't find a way to do this. All I see is an option to set the specific file - excel or CSV file as input source. Please let me know what I'm missing? We are using Sisense V7.2.1

    Thanks!

    0
    Comment actions Permalink
  • Avatar
    Michael Guidone

    I'll just quote from the support ticket we reported at the time:

    ___

    We started using the Table to CSV Post Plugin today and found that it is destructive to the data it exports. The python code is manipulating the data so that it doesn't have to deal with any special characters like commas ,newlines and any word with the string 'None' in it, even though it goes on to use the python CSV writer that knows how to support those characters and quote everything.

    For Example: "Nonex Limited, LLC" get replaced with "x Limited LLC"

    We figured that this would be an easy python fix and correct this ourselves by using the csv writer QUOTEALL option and removing the offending destructive code. (See attached file)"

    also:

    We experienced some odd behavior with the CSV import between the desktop and web modelers. When adding a CSV file datasource to the desktop modeler and then building the cube there were warning messages stating that some rows were ignored due to being empty/invalid. When we did the same process in the web modeler (adding a CSV datasource for a table and then building), it didn't ignore any rows and everything was imported to the cube.

    My question is, is the web modeler building differently? Why would the desktop modeler ignore rows in the file when the web modeler did not? The only explanation I can gather is that the desktop manager doesn't handle escaped double quotes ( " ) properly. If you notice the first row above in the second column it says "Rock ""N"" Roll Auto Sales, LLC" the name we would expect to be imported would have double quotes around the "N". ( Rock "N" Roll Auto Sales, LLC ) The rest of the rows have similar syntax in their names. It is how quotes in text are escaped in CSV files.

    ___

     

    Sisense responded:

    This is not an official plugin of Sisense and was developed by our community members (I just posted this in the community I didn't actually develop it). In the next few months, we will work on bringing our community back to life and will be more than happy for you to help.

    We followed up and asked if and when they would have an update, and they never replied. 

     

     

     

    0
    Comment actions Permalink
  • Avatar
    Michael Guidone

    It can be done using the WIndows ECM on CSV import. 

    0
    Comment actions Permalink
  • Avatar
    Roopa Raghavan

    Michael - thank you so much for your retailed response. I see the select folder option and thanks for pointing me in the right direction. I will check out the differences between the desktop and web modelers, I have only been using desktop ECM and not the web version at all.

    Thanks again.

    Roopa

    0
    Comment actions Permalink

Please sign in to leave a comment.