Remove edition tag call-for-code
article thumbnail

Automated data governance with AWS Glue Data Quality, sensitive data detection, and AWS Lake Formation

AWS Big Data

In this post, we showcase how to use AWS Glue with AWS Glue Data Quality , sensitive data detection transforms , and AWS Lake Formation tag-based access control to automate data governance. Currently, these requirements are hard-coded and managed manually for each set of users. It’s required to ensure the governance is met as defined.

article thumbnail

Query your Apache Hive metastore with AWS Lake Formation permissions

AWS Big Data

We illustrate a cross-account sharing use case, where a Lake Formation steward in producer account A shares a federated Hive database and tables using LF-Tags to consumer account B. The admin continues to set up Lake Formation tag-based access control (LF-TBAC) on the federated Hive database and share it to account B.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

End-to-end development lifecycle for data engineers to build a data integration pipeline using AWS Glue

AWS Big Data

End-to-end development lifecycle for a data integration pipeline Today, it’s common to define not only data integration jobs but also all the data components in code. Implement In the implementation phase, data engineers code the data integration pipeline. Data is a key enabler for your business.

article thumbnail

Orchestrate an end-to-end ETL pipeline using Amazon S3, AWS Glue, and Amazon Redshift Serverless with Amazon MWAA

AWS Big Data

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed orchestration service for Apache Airflow that you can use to set up and operate data pipelines in the cloud at scale. Apache Airflow is an open source tool used to programmatically author, schedule, and monitor sequences of processes and tasks, referred to as workflows.

Metadata 104
article thumbnail

Prioritizing AI? Don’t shortchange IT fundamentals

CIO Business Intelligence

Generative AI continues to dominate IT projects for many organizations, with two thirds of business leaders telling a Harris Poll they’ve already deployed generative AI tools internally, and IDC predicting spend on gen AI will more than double in 2024. But the usual laundry list of priorities for IT hasn’t gone away.

IT 140
article thumbnail

Synchronize your Salesforce and Snowflake data to speed up your time to insight with Amazon AppFlow

AWS Big Data

Developers need to understand the application APIs, write implementation and test code, and maintain the code for future API changes. Amazon AppFlow , which is a low-code/no-code AWS service, addresses this challenge. Scheduled – Amazon AppFlow can run schedule-triggered flows based on a pre-defined schedule rule.

article thumbnail

Stack Overflow announces OverflowAI

CIO Business Intelligence

Today marks the beginning of a new and exciting era for Stack Overflow. Let’s highlight the new features and products we announced today from the stage of WeAreDevelopers. After that, I’ll provide more detail on the guiding principles we’re putting in place to align our use of AI with the core values of Stack Overflow and our community.

Testing 72