article thumbnail

Orchestrate an end-to-end ETL pipeline using Amazon S3, AWS Glue, and Amazon Redshift Serverless with Amazon MWAA

AWS Big Data

VPC endpoints are created for Amazon S3 and Secrets Manager to interact with other resources. The Amazon provider is used to interact with AWS services like Amazon S3, Amazon Redshift Serverless, AWS Glue, and more. Secrets like user name, password, DB port, and AWS Region for Redshift Serverless are stored in Secrets Manager.

article thumbnail

Federate Amazon QuickSight access with open-source identity provider Keycloak

AWS Big Data

Download the SAML metadata file. In the navigation pane under Clients , import the SAML metadata file. Download the Keycloak IdP SAML metadata file from that URL location. For Metadata document , upload the Keycloak IdP SAML metadata XML file you downloaded and saved to your local machine earlier. Choose Browse.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Accelerate HiveQL with Oozie to Spark SQL migration on Amazon EMR

AWS Big Data

We split the solution into two primary components: generating Spark job metadata and running the SQL on Amazon EMR. The first component (metadata setup) consumes existing Hive job configurations and generates metadata such as number of parameters, number of actions (steps), and file formats. X Python 3.8 Amazon EMR 6.1

article thumbnail

Build streaming data pipelines with Amazon MSK Serverless and IAM authentication

AWS Big Data

or higher Appropriate AWS credentials for interacting with resources in your AWS account. The following software installed on your development machine, or use an AWS Cloud9 environment, which comes with all requirements preinstalled: Java Development Kit 17 or higher (for example, Amazon Corretto 17 , OpenJDK 17 ) Python version 3.11

Testing 95
article thumbnail

Gain insights from historical location data using Amazon Location Service and AWS analytics services

AWS Big Data

The Data Catalog provides metadata that allows analytics applications using Athena to find, read, and process the location data stored in Amazon S3. The crawlers will automatically classify the data into JSON format, group the records into tables and partitions, and commit associated metadata to the AWS Glue Data Catalog. Choose Run.

article thumbnail

Process and analyze highly nested and large XML files using AWS Glue and Amazon Athena

AWS Big Data

To analyze XML files stored in Amazon S3 using AWS Glue and Athena, we complete the following high-level steps: Create an AWS Glue crawler to extract XML metadata and create a table in the AWS Glue Data Catalog. We use the AWS Glue crawler to extract XML file metadata. We also use a custom XML classifier in this solution.

article thumbnail

Amazon DataZone now integrates with AWS Glue Data Quality and external data quality solutions

AWS Big Data

By selecting the corresponding asset, you can understand its content through the readme, glossary terms , and technical and business metadata. We use this data source to import metadata information related to our datasets. Use Amazon DataZone APIs through Boto3 to push custom data quality metadata.