Unify your business data with OneLake and Microsoft Fabric

Microsoft launched Fabric, its all-in-one data and analytics platform, a few months ago, covering everything from data movement and ETL processes to data science, real-time analytics, business intelligence, and visualization.

Strong data governance is essential to a well-functioning enterprise and OneLake offers a unified and cohesive datalake solution for all Fabric experiences. We take a look at all its benefits, use cases, and key features.  

What is OneLake

OneLake is a unified data lake platform that centralizes all of a company’s data in one place (similar to how OneDrive centralizes documents). This centralized infrastructure facilitates cross-team collaboration, eliminates information silos, and improves operational efficiency by avoiding the dispersion of data across multiple systems.

Its intuitive interface and advanced capabilities make Microsoft OneLake an essential tool for those looking to optimize the management of their digital assets.

Unlike other products such as OneDrive (designed for personal and team file and document storage), OneLake specializes in enterprise-scale data management and analysis. This includes advanced capabilities for processing large volumes of data and support for Machine Learning operations and predictive analytics. In addition, OneLake is built on Azure Data Lake infrastructure, which ensures high compatibility and performance with other Azure tools dedicated to analytics and AI.  

OneLake DataHub

OneLake has emerged as the solution to the challenges of the previous numerous lakes for different business groups or the overhead of managing multiple resources, improving collaboration.  

Default governance with distributed ownership

Knowing where a client’s collaboration begins and ends provides a natural boundary of governance and compliance, which is under the control of a tenant manager.

Within a tenant, OneLake can create any number of workspaces, which allow different parts of the organization to distribute ownership and access policies.

Within a workspace, data elements can be created and all data in OneLake can be accessed through them. Fabric stores lakehouses, warehouses, and other elements in OneLake.  

Open to all levels

OneLake is built on Azure Data Lake Storage Gen2 and can support any file type, structured or unstructured. All Fabric data elements, such as data warehouses and lakehouses, automatically store their data in OneLake in Delta Parquet format.

OneLake supports the same ADLS Gen2 APIs and SDKs to be compatible with existing ADLS Gen2 applications, including Azure Databricks. You can approach data in OneLake as one large ADLS storage account for the entire organization. Each workspace appears as a container within that storage account, and the different data items appear as folders within those containers. 

File Explorer for Windows

Allows you to easily browse OneLake data from Windows using the OneLake file explorer. This makes navigating through all workspaces and data elements easy, and easily upload, download, or modify files.

This explorer simplifies working with data lakes, allowing even non-technical business users to use it.  

Secure data backup

OneLake provides more value from a single copy of data with no data movement or duplication. With this tool, you no longer need to copy data only for use with another engine or to eliminate silos for analysis with data from other sources.

In addition, shortcuts make it easy to share data between users and applications without having to move or duplicate information unnecessarily. When teams work independently in separate workspaces, these shortcuts allow them to combine data from different business groups and domains into a virtual data product to meet a user’s specific needs.

In fact, with Fabric, the different analytics engines store data in Delta Parquet’s open format to allow the same data to be used across multiple engines. Business users can create Power BI reports directly on OneLake using the new Direct Lake mode in the Analysis Services engine. The Analysis Services engine is what drives Power BI semantic models and has always offered two ways to access data: import and direct query. 

Benefits of using OneLake

Microsoft OneLake offers a great scalable storage solution, as well as security or integration with other Microsoft tools, but brings many more benefits: 

• Efficiency in data management: OneLake can centralize and organize large volumes of information in one place, facilitating more effective access and management. In addition, thanks to the use of optimized algorithms for index creation and fast information search, queries can be made and data retrieved in much less time.
• Lower cost and greater benefits: it stands out for its highly satisfactory cost-benefit ratio for companies, since, through its consumption-based pricing model, companies can scale their storage needs without incurring large upfront investments. Compression technologies reduce the investment required by reducing the space needed to store large amounts of data. In addition, recovery and high availability capabilities ensure that operations continue without interruption, minimizing the costs associated with downtime.  
•  Scalability and flexibility: it can adapt to the changing needs of organizations, so storage size can be increased or decreased as needed without affecting performance. This flexibility allows companies to adjust their resources efficiently and cost-effectively. In addition, compatibility with various third-party tools facilitates integration and customization, ensuring that it can adapt to different environments and business requirements.  

Fabric and OneLake

As the heart of Microsoft Fabric and the foundational data layer, OneLake offers a unified and cohesive datalake solution for all Fabric experiences. Designed as SaaS, the solution combines different areas into an integrated analytics system with OneLake as the foundational data layer.

A company’s data is often distributed or spread across different areas or departments, yet they may need access to the same or similar data sources. If data sources are copied across domains, it may not be clear who owns the initial source, which can lead to a lack of trust in data products and teams.

The new updates bring value to Fabric and OneLake technology in several ways: 

• Multi-cloud shortcuts improve accessibility and the ability to integrate new data sources while minimizing data redundancy.
• Database Mirroring facilitates seamless near real-time analysis use cases and more centralized data management.  

By leveraging these two capabilities, organizations can effortlessly establish virtualized lakes, breaking down silos between various entities, eliminating the waiting period for IT to establish and maintain pipelines, and reducing resource allocation and time for data migration. 

Multi-cloud shortcuts

By acting as a unified data lake, with shortcuts, data can be referenced in different locations while still being logically represented within the same lake.

These shortcuts allow links to be created that point to a particular storage location. This storage location can be internal or external.

On the other hand, an important aspect of reducing output costs is the caching of shortcuts between clouds. These shortcuts can be created from lakehouses or KQL databases and managed programmatically through designated APIs.

This feature is a major step towards distributed data ownership in OneLake, helping to unify the data landscape and a breakthrough for the Fabric community. 

Database Mirroring

Another feature is database replication. This offers the ability to maintain data synchronization between two databases without the need to create complex ETL pipelines. This allows you to replicate a database or a subset of database tables in Fabric’s OneLake, keeping it up to date in near real-time. In addition, there are no further storage costs associated with mirroring, as this feature is implemented by reading transaction log changes from the source database and updating the relevant data.

One of the main advantages is that it protects operational databases from analytical queries. With many join and grouping operations, they can be quite demanding for operational DBs, and Mirroring helps reduce the workload. Another advantage is the ability to cross-query between databases directly in Fabric. In addition, it supports table schema evolution and supported data type changes.

However, the most significant advantage is the potential for real-time analytics combined with reduced time to value and the ability to quickly create powerful data products. Power BI can access mirrored database tables with Direct Lake mode, providing the ability to easily create dashboards and reports for near real-time monitoring. 

In a rapidly changing business environment, the ability to analyze data instantly has become a necessity, and thanks to it, companies gain the ability to monitor events in real-time.

This allows you to react quickly to changes and solve potential problems. And at Plain Concepts we help you get the most out of it.

We offer you a Microsoft Fabric Adoption Framework with which we will evaluate the technological and business solutions, we will make a clear roadmap for the data strategy, we visualize the use cases that make the difference in your company, we take into account the sizing of equipment, time and costs, we study the compatibility with existing data platforms and we migrate Power BI, Synapse and Datawarehouse solutions to Fabric.