Azure Data Factory and Security: Protect your data today

In a landscape led by data and where companies often feel overwhelmed by the amount of data they handle, we find a business situation that needs an urgent change.

A few months ago, we shared with you an article about all the details of Azure Data Factory, but in this one, we are going to focus on its security considerations. Take note!

Azure Data Factory Security

Azure Data Factory is a cloud data solution that allows you to ingest, prepare, and transform data on at large scale. It facilitates its use for a wide variety of use cases, such as data engineering, migration of on-premises packages to Azure, operational data integration, analytics, data ingestion into warehouses, etc.

Data Factory’s management resources are integrated into Azure’s security infrastructure and apply all the security measures offered by Azure. This is because, in a Data Factory solution, one or more data pipelines are created, which are a logical grouping of activities that perform a task.

Although Data Factory is available in some regions, the data movement service is available globally to ensure data compliance, efficiency, and lower network exit costs.

Although it includes Azure Integration Runtime and the self-hosted integration runtime environment, it does not store any temporary data, cache data, or logs, except for linked service credentials from cloud data stores, which are encrypted using certificates.

With this solution, data-driven workflows can be created to orchestrate data movement between supported data stores and orchestrate data processing by process services in other regions or local environments.

Security considerations

The security considerations to be taken into account in the two data movement scenarios are: cloud scenario (source and destination are publicly accessible via the Internet) and hybrid scenario (source or destination are behind a firewall or within a local corporate network).

Cloud scenarios

It applies to cloud services such as Azure Storage, Azure Synapse Analytics, Azure SQL Database, or Azure Data Lake Store, among others.

For the protection of data store credentials, we have two options:

• Storing encrypted credentials in a managed Azure Data Factory vault: Helps protect data store credentials by encrypting them with Microsoft-managed certificates.
• Storing credentials in Azure Key Vault: You can also store the data store credential there, as Data Factory retrieves the credential during the execution of an activity.

Centralizing the storage of application secrets allows you to control their distribution. Key Vault reduces the chances of secrets being accidentally leaked. Applications can securely access the information they need through URIs. These URIs allow applications to retrieve specific versions of a secret.

In turn, if the cloud data store supports HTTPS or TLS, all data transfers between Data Factory data movement services and a cloud data store are done over the HTTPS or TLS secure channel.

Hybrid scenarios

Hybrid scenarios require the self-hosted integration runtime environment to be installed on a local or virtual network (Azure) or within a virtual private cloud (Amazon). The self-hosted integration runtime environment must be able to access local data stores.

The command channel enables communication between Data Factory data movement services and the self-hosted integration runtime environment. The communication contains activity-related information. The data channel is used to transfer data between local data stores and cloud data stores.

Best practices for securing the movement of data in Azure Data Factory

Ensuring the secure movement of data is critical, especially when data is sensitive, to protect confidentiality, integrity, and regulatory compliance.

Some of the steps and best practices to follow include:

1. Use secure connections: To ensure the security of data transfer, it is recommended to always use protocols such as HTTPS or SSL/TLS. This allows data to be encrypted during transmission and protects it from unwanted access or tampering.
2. Implement encryption: With built-in encryption capabilities, sensitive data can be protected while at rest. To preserve confidentiality, Key Vault helps in encrypting data pipelines and storage systems.
3. Role-based access control (RBAC): This solution can be used to enforce the principle of least privilege, putting RBAC into practice. Based on each user’s data access requirements and service principle, responsibilities are defined and assigned.
4. Store credentials securely: Keep credentials out of Azure Data Factory pipelines by encrypting them. In addition, to maintain secure authentication during data movement operations, it is best to restrict who has access to stored secrets using managed identities and access controls.
5. Enable data lineage and monitoring: Enable data lineage tracking and monitoring to follow the flow of data through pipelines and operations. This can record information about data transport processes, such as source, destination, and transformation activities, including logging and auditing functions.
6. Implement data masking and redaction: To selectively expose or hide data based on user responsibilities and permissions, dynamic data masking (DDM) can be used. To replace or remove sensitive data fragments before transmission or storage, redaction rules are recommended.
7. Secure networks and VNet integration: Configure firewall rules and network security groups (NSGs) to limit inbound and outbound traffic to trusted IP addresses and virtual networks.
8. Continuous security and compliance monitoring: adopting security monitoring procedures is critical to quickly identify and address any security threats. Azure Sentinel and Security Center can help monitor data movement activities and proactively reduce threats through audits and compliance assessments.
9. Update and patch: It is very important to update all ADF resources with the latest news and fixes (linked services, pipelines, and integration runtimes). Automatic deployment pipelines should be set up to ensure smooth maintenance and upgrades.
10. Train users: Security awareness and training are critical throughout this process. Sharing best practices for moving data securely, such as incident response plans, data handling techniques, and authentication systems, will foster a culture of accountability and security awareness in your organization.

Secure Azure Data Factory Deployment

Organizations can improve their Azure Data Factory security posture and ensure secure data flow throughout the data integration process by adhering to the best practices and tips shared above.

In the face of a rapidly changing business environment, the ability to analyze data instantly has become a necessity, and thanks to it, companies gain the ability to monitor events in real time.

This allows you to react quickly to changes and solve potential problems. At Plain Concepts, we help you get the most out of it.

We propose a data strategy in which you can get value and get the most out of your data.

We help you discover how to get value from your data, control and analyze all your data sources, and use data to make intelligent decisions and accelerate your business:

• Data analytics and strategy assessment: We evaluate data technology for architecture synthesis and implementation planning.
• Modern analytics and data warehouse assessment: We provide you with a clear view of the modern data warehousing model through understanding best practices on how to prepare data for analysis.
• Exploratory data analysis assessment: We look at the data before making assumptions so you get a better understanding of the available data sets.
• Digital Twin and Smart Factory Accelerator: We create a framework to deliver integrated digital twin manufacturing and supply chain solutions in the cloud.

In addition, we offer you a Microsoft Fabric Adoption Framework with which we will evaluate the technological and business solutions, we will make a clear roadmap for the data strategy, we visualize the use cases that make the difference in your company, we take into account the sizing of equipment, time and costs, we study the compatibility with existing data platforms and we migrate Power BI, Synapse and Datawarehouse solutions to Fabric.