Blog

Over my 17 years of working in Business Intelligence and Analytics tools, one of the great white whales has often been achieving a perfect balance of data storage and data freshness. Data warehouses allow us to store and analyze billions of rows of data, but the processes used to load those warehouse tables takes time.

Long data loads that populate data warehouse tables are extremely common across different tools, and NetSuite Analytics Warehouse is no exception. When the tool was originally released in 2022, users were limited to refreshing the data warehouse once per day.

Since then, NetSuite has enhanced the capabilities available in the tool to allow what they call “Frequent Data Refreshes.” This means that you can select portions of the application to refresh on a more frequent basis. Since it doesn’t force a refresh of the entire warehouse structure, the refreshes are quicker and can be scheduled more often.

This definition can be controlled by functional area or specific warehouse table, so the recommendation is to only apply this setting where there is a tangible business benefit or value to doing so. Having data refreshes run unnecessarily will only cause more issues with maintenance and administration if they run long or are interrupted.

Options for the frequent data loads include running them as often as once per hour, although you will need to ensure your selection completes well within that window to allow it to routinely fire a new load so often.

You can define frequent refreshes by functional area…

...or by specific warehouse table.

NetSuite has also included the option to prioritize certain tables to be refreshed over others. This applies to the nightly refresh and ensures that these tables are loaded and made available first, which provides some flexibility for administrators with more active user groups that want the latest data as early as possible each day.

Finally, NetSuite has also added the ability to schedule full warehouse refreshes. These loads completely wipe the data warehouse of NetSuite ERP data and reloads with the current values from the ERP. These loads take longer than the default incremental loads that only load records that have been changed or created recently, so you should schedule them with care. They can be beneficial to run in certain cases, though, if data inconsistency has been found, or a new subsidiary has been added, as examples.

All of these new features provide a greater flexibility with customizations for how NSAW handles loading your ERP data. As the NSAW user base continues to expands, I only expect more and more flexibility with these and other aspects of the tool to become standard in the coming months and years.