Guide To Evolving Your Enterprise Data Warehouse

The Inefficiency of Siloed Data

Most enterprise data resides in silos. Your finance system, enterprise resource planning (ERP) system, and customer relationship management (CRM) system all store data in their separate platforms. While this data is useful to the people with access to the system, many decisions require analysis using data across the organization. For instance, people trying to make decisions on staffing, inventory, and project deadlines may not have access to critical data stored in the finance system.

The easiest way to circumvent this problem is by creating reports. Data-driven individuals send these reports to various stakeholders on a daily, weekly, or monthly basis. This, however, only shifts the problem to the individual business units, who then must spend time collating finance or customer data into their own data. This approach is not ideal, as it distracts business units from doing their actual jobs and increases the likelihood of errors in the data collation process.

One way to overcome the challenge of data silos and integrate data from multiple sources is to build an enterprise-level Data Lake. The data lake acts as a centralized repository for data from different systems within the organization, including finance, HR, sales, and operations. You will load your data in its original format into the data lake on a regular basis, usually daily, making the creation of the data lake relatively simple.

Interpreting the data in a data lake can be challenging, however, since the data exists in several separate data models. There may be no direct mapping across your data sources, making it difficult to link, for instance, a client in the CRM system to the same client in the billing system.

To make the data in the data lake more accessible to users, you will want to set up an Analytics Database. This can be a relational/SQL database, such as MS SQL Server, a schema-less database, such as MongoDB, or a graph database, such as neo4j. Unlike operational databases, an analytics database is optimized for large data reads rather than individual record updates. Performance tuning, such as indexing, is handled automatically in an analytics database.

One other benefit of an analytics database is simplifying control over who can access your enterprise data. By funneling all data requests through the analytics database, you can provide access to your enterprise data through a single data platform, instead of dealing with access to individual systems. You can also control access at the table or row level, aligned with standard business practices.

By joining all your data from various systems in a single analytics database, users who are familiar with SQL or JSON can query the data they need and join it with other data sources (with some effort and sufficient expertise). However, differences in entity naming, or primary keys, may prevent the easy comparison of data across data models. Similarly, nuances such as data granularity or how changes are handled in each data model may prevent an apples-to-apples comparison between data sets. For this, your enterprise will need to implement a Cloud Data Warehouse.