Remove edition tag call-for-code
article thumbnail

Automated data governance with AWS Glue Data Quality, sensitive data detection, and AWS Lake Formation

AWS Big Data

In this post, we showcase how to use AWS Glue with AWS Glue Data Quality , sensitive data detection transforms , and AWS Lake Formation tag-based access control to automate data governance. Currently, these requirements are hard-coded and managed manually for each set of users. It’s required to ensure the governance is met as defined.

article thumbnail

Introducing shared VPC support on Amazon MWAA

AWS Big Data

In this post, we demonstrate automating deployment of Amazon Managed Workflows for Apache Airflow (Amazon MWAA) using customer-managed endpoints in a VPC, providing compatibility with shared, or otherwise restricted, VPCs. In this example, we create the VPC and Amazon MWAA environment in the same account.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Query your Apache Hive metastore with AWS Lake Formation permissions

AWS Big Data

We illustrate a cross-account sharing use case, where a Lake Formation steward in producer account A shares a federated Hive database and tables using LF-Tags to consumer account B. The admin continues to set up Lake Formation tag-based access control (LF-TBAC) on the federated Hive database and share it to account B.

article thumbnail

Configure ADFS Identity Federation with Amazon QuickSight

AWS Big Data

Amazon QuickSight Enterprise edition can integrate with your existing Microsoft Active Directory (AD), providing federated access using Security Assertion Markup Language (SAML) to dashboards. You can control which users have access to QuickSight using your existing IdP. The end-user initiates authentication from the sign-in page of the IdP.

article thumbnail

End-to-end development lifecycle for data engineers to build a data integration pipeline using AWS Glue

AWS Big Data

End-to-end development lifecycle for a data integration pipeline Today, it’s common to define not only data integration jobs but also all the data components in code. Implement In the implementation phase, data engineers code the data integration pipeline. Data is a key enabler for your business.

article thumbnail

Implement tag-based access control for your data lake and Amazon Redshift data sharing with AWS Lake Formation

AWS Big Data

With this feature, Amazon Redshift customers can now manage sharing, apply access policies centrally, and effectively scale the permission using LF-Tags. Lake Formation also provides tag-based access control (TBAC), which can be used to simplify and scale governance of data catalog objects such as databases and tables.

article thumbnail

Orchestrate an end-to-end ETL pipeline using Amazon S3, AWS Glue, and Amazon Redshift Serverless with Amazon MWAA

AWS Big Data

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed orchestration service for Apache Airflow that you can use to set up and operate data pipelines in the cloud at scale. Apache Airflow is an open source tool used to programmatically author, schedule, and monitor sequences of processes and tasks, referred to as workflows.

Metadata 108