AWS Big Data

MARCH 11, 2024

Amazon Redshift is a fully managed, scalable cloud data warehouse that accelerates your time to insights with fast, straightforward, and secure analytics at scale. Tens of thousands of customers rely on Amazon Redshift to analyze exabytes of data and run complex analytical queries, making it the most widely used cloud data warehouse.

Analytics 65

Analytics Data Warehouse Optimization Metrics 65

AWS Big Data

AUGUST 31, 2023

Ingesting a high volume of streaming data has been a defining characteristic of operational analytics workloads with Amazon OpenSearch Service. Many of these workloads involve either self-managed Apache Kafka or Amazon Managed Streaming for Apache Kafka (Amazon MSK) to satisfy their data streaming needs.

Testing 105

Testing Data Processing Dashboards Management 105

AWS Big Data

DECEMBER 13, 2023

This post showcases how to use streaming ingestion to bring data to Amazon Redshift. It’s simple to set up, and directly ingests streaming data into your data warehouse from Amazon Kinesis Data Streams and Amazon Managed Streaming for Kafka ( Amazon MSK ) without the need to stage in Amazon Simple Storage Service (Amazon S3).

Data Warehouse 96

Data Warehouse Snapshot Data Processing Management 96

AWS Big Data

NOVEMBER 13, 2023

The AWS Identity and Access Management (IAM) authentication feature in Amazon Managed Streaming for Apache Kafka (Amazon MSK) now supports all programming languages. Both Amazon MSK provisioned and serverless cluster types support the new Amazon MSK IAM expansion to all programming languages.

Testing 92

Testing Management Consulting IT 92

AWS Big Data

SEPTEMBER 22, 2023

In this post, we will describe how and why we decided to migrate from self-managed Kafka to Amazon Managed Streaming for Apache Kafka ( Amazon MSK ). We’ll start with an overview of our self-managed Kafka, why we chose to migrate to Amazon MSK, and ultimately how we did it. in “newkafka.”

Management 101

Management Metrics Cost-Benefit Data Lake 101

AWS Big Data

MAY 3, 2023

In this post, we provide a detailed overview of streaming messages with Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Amazon ElastiCache for Redis , covering technical aspects and design considerations that are essential for achieving optimal results. The following figure shows an example of the data flow at SOCAR.

IoT 103

IoT Internet of Things Data Transformation Management 103

AWS Big Data

OCTOBER 31, 2023

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed workflow orchestration service for Apache Airflow that you can use to set up and operate end-to-end data pipelines in the cloud at scale. We define the steps needed to set up the integration between Amazon MWAA and Snowflake.

Data Processing 88

Data Processing Management Publishing Testing 88

AWS Big Data

APRIL 18, 2024

In this post, we discuss how the Salesforce TIP team optimized their architecture using Amazon Web Services (AWS) managed services to achieve better scalability, cost, and operational efficiency. The data lake consumers then use Apache Presto running on Amazon EMR cluster to perform one-time queries.

Optimization 85

Optimization Data Lake Management Key Performance Indicator 85

AWS Big Data

NOVEMBER 9, 2023

dbt enables you to write SQL select statements, and then it manages turning these select statements into tables or views in Amazon Redshift. Queues and topics – Queues and topics come from various integration applications that generate data in real time.

Data Warehouse 103

Data Warehouse Testing Data Quality Reporting 103

AWS Big Data

MARCH 25, 2024

We discuss the value of AWS data streaming services such as Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon Kinesis Data Streams , Amazon Managed Service for Apache Flink , and Amazon Kinesis Data Firehose in building generative AI applications.

Data Lake 88

Data Lake Unstructured Data Management Modeling 88

AWS Big Data

SEPTEMBER 6, 2023

Amazon’s serverless Apache Kafka offering, Amazon Managed Streaming for Apache Kafka (Amazon MSK) Serverless , is attracting a lot of interest. At the time of writing, the Amazon MSK library for IAM is exclusive to Kafka libraries in Java, creating a challenge for users of other programming languages.

Testing 95

Testing Metadata Cost-Benefit Management 95

AWS Big Data

SEPTEMBER 13, 2023

To address these issues effectively, we propose using Amazon Managed Streaming for Apache Kafka (Amazon MSK), a fully managed Apache Kafka service that offers a seamless way to ingest and process streaming data. Following the data processing, the streaming job stores data in Amazon S3 and generates a Data Catalog table.

Management 89

Management Data Processing Interactive Metadata 89

AWS Big Data

JANUARY 10, 2024

Amazon Managed Service for Apache Flink is a fully managed service that reduces the complexity of building and managing Apache Flink applications. Amazon Managed Service for Apache Flink manages the underlying Apache Flink components that provide durable application state, metrics, logs, and more.

Metrics 97

Metrics Management Snapshot IT 97

AWS Big Data

SEPTEMBER 19, 2023

Managing configurations for Amazon MSK Connect , a feature of Amazon Managed Streaming for Apache Kafka (Amazon MSK), can become challenging, especially as the number of topics and configurations grows. The challenges lie in the overhead of managing configurations, as well as dealing with patching and upgrades.

Management 83

Management Data-driven Optimization Data Processing 83

AWS Big Data

SEPTEMBER 28, 2023

In EDAs, modern event brokers, such as Amazon EventBridge and Apache Kafka, play a key role to publish and subscribe to events. There are two ways to send events from Apache Kafka to EventBridge: the preferred method using Amazon EventBridge Pipes or the EventBridge sink connector for Kafka Connect.

Data-driven 122

Data-driven Metrics Management Publishing 122

AWS Big Data

JANUARY 30, 2023

In April 2022, Amazon Managed Streaming for Apache Kafka (Amazon MSK) launched an exciting new capability, Amazon MSK Serverless. Amazon MSK is a fully managed service for Apache Kafka that makes it easier for developers to build and run highly available, secure, and scalable applications based on Apache Kafka.

Management 86

Management Optimization Data Processing IT 86

AWS Big Data

AUGUST 31, 2023

Amazon Redshift is a fast, fully managed, cloud data warehouse that allows you to process and run your complex SQL analytics workloads on structured and semi-structured data. Amazon S3 Event Notifications is an Amazon S3 feature that you can enable in order to receive notifications when specific events occur in your S3 bucket.

Data Warehouse 81

Data Warehouse Data-driven Testing Business Intelligence 81

AWS Big Data

JUNE 6, 2023

This post explains how the underlying infrastructure affects Kafka performance when you use Amazon Managed Streaming for Apache Kafka (Amazon MSK) tiered storage. We delve deep into the core components of Amazon MSK tiered storage and address questions such as: How does read and write work in a tiered storage-enabled cluster?

Metadata 75

Metadata Optimization Management Metrics 75

AWS Big Data

MAY 17, 2023

Customers are adopting Amazon Managed Service for Apache Kafka (Amazon MSK) as a fast and reliable streaming platform to build their enterprise data hub. In addition to streaming capabilities, setting up Amazon MSK enables organizations to use a pub/sub model for data distribution with loosely coupled and independent components.

Management 88

Management Data-driven Publishing Data Processing 88

AWS Big Data

AUGUST 9, 2023

Similarly, in a serverless paradigm, application logs in Amazon CloudWatch are sourced from a variety of participating services, and traversing the lineage across logs can also present challenges. We have used an SNS topic for routing both success and failure states for the Lambda-based tasks.

Data Lake 86

Data Lake Metrics Testing Cost-Benefit 86

AWS Big Data

NOVEMBER 27, 2023

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data. Today, we’re excited to bring the benefits of Graviton3 to Kafka workloads, with Amazon MSK now offering M7g instances for new MSK provisioned clusters.

Cost-Benefit 95

Cost-Benefit Optimization Management Testing 95

AWS Big Data

APRIL 3, 2024

Transactional data lakes built on AWS and Snowflake Snowflake provides various integrations for Iceberg tables with multiple storage options, including Amazon S3 , and multiple catalog options, including AWS Glue Data Catalog and Snowflake. Advantages of Apache Iceberg Apache Iceberg is a distributed, community-driven, Apache 2.0-licensed,

Data Lake 93

Data Lake Snapshot Metadata Data Architecture 93

AWS Big Data

MARCH 13, 2023

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is an AWS streaming data service that manages Apache Kafka infrastructure and operations, making it easy for developers and DevOps managers to run Apache Kafka applications and Kafka Connect connectors on AWS, without the need to become experts in operating Apache Kafka.

Optimization 76

Optimization Management Metrics Metadata 76

AWS Big Data

MARCH 8, 2024

Amazon OpenSearch Serverless focuses on delivering seamless scalability and management of search workloads; Amazon OpenSearch Ingestion complements this by providing a robust solution for anomaly detection on indexed data. If validation fails, refer to Troubleshooting Amazon OpenSearch Service for troubleshooting and guidance.

Data Processing 97

Data Processing Risk Management Machine Learning Big Data 97

AWS Big Data

APRIL 3, 2024

Today, we are pleased to announce that Amazon DataZone is now able to present data quality information for data assets. Amazon DataZone now integrates directly with AWS Glue to display data quality scores for AWS Glue Data Catalog assets. Amazon DataZone data assets can be updated at varying frequencies.

Data Quality 87

Data Quality Visualization Metadata Metrics 87

AWS Big Data

JUNE 19, 2023

In Part 1 of this two-part series, we discussed the concepts of how to enforce Kafka quotas in Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters while using AWS Identity and Access Management (IAM) access control. Create Topic-B on the MSK cluster. Create IAM roles for the client applications to access Topic-B.

Metrics 73

Metrics Management Testing Publishing 73

AWS Big Data

NOVEMBER 13, 2023

Amazon MSK Serverless is a cluster type for Amazon Managed Streaming for Apache Kafka (Amazon MSK) that is the most straightforward way to run Apache Kafka clusters without having to manage compute and storage capacity. In this post, we show how you can make these code changes by using the provided code libraries and examples.

Cost-Benefit 76

Cost-Benefit Management Optimization Consulting 76

CIO Business Intelligence

OCTOBER 20, 2023

The certification covers high-level topics such as the information systems auditing process, governance and management of IT, operations and business resilience, and IS acquisition, development, and implementation. According to PayScale, the average annual salary for CISA certified IT pros is $114,000 per year.

IT 136

IT Consulting Testing IoT 136

AWS Big Data

MARCH 7, 2024

For example, application logs and traces from web applications can be collected directly in a data lake, and a portion of that data can be moved out to a log analytics store like Amazon OpenSearch Service for daily analysis. UltraWarm nodes use Amazon S3 with related caching solutions to improve performance.

Data Lake 111

Data Lake Analytics Dashboards Metrics 111

AWS Big Data

JULY 25, 2023

Amazon Redshift Serverless makes it simple to run and scale analytics in seconds. Just load your data and start querying right away in the Amazon Redshift Query Editor or in your favorite business intelligence (BI) tool. The Amazon Redshift CloudWatch metrics are data points for operational monitoring.

Metrics 78

Metrics Data Warehouse Dashboards Snapshot 78

AWS Big Data

OCTOBER 20, 2023

Amazon MSK Connect is a feature of Amazon Managed Streaming for Apache Kafka (Amazon MSK) that offers a fully managed Apache Kafka Connect environment on AWS. Amazon Relational Database Service (Amazon RDS) or Amazon Aurora servers have DNS names that are publicly resolvable but not accessible.

Data Processing 85

Data Processing Snapshot Data Warehouse Management 85

AWS Big Data

FEBRUARY 28, 2024

Today, we are launching Terraform support for Amazon OpenSearch Ingestion. OpenSearch Ingestion is a fully managed, serverless data collector that delivers real-time log, metric, and trace data to Amazon OpenSearch Service domains and Amazon OpenSearch Serverless collections. Terraform installed on your local machine.

Metrics 65

Metrics Management Software Testing 65

AWS Big Data

NOVEMBER 21, 2023

Amazon OpenSearch Ingestion is a fully managed, serverless pipeline that delivers real-time log, metric, and trace data to Amazon OpenSearch Service domains and OpenSearch Serverless collections. Customers use Amazon OpenSearch Ingestion pipelines to ingest data from a variety of data sources, both pull-based and push-based.

Metrics 92

Metrics Management Technology IT 92

AWS Big Data

APRIL 4, 2024

Amazon Redshift is a recommended service for online analytical processing (OLAP) workloads such as cloud data warehouses, data marts, and other analytical data stores. Amazon Redshift Serverless makes it straightforward to run and scale analytics in seconds without the need to set up and manage data warehouse clusters.

Big Data 65

Big Data Data Warehouse Advertising OLAP 65

AWS Big Data

FEBRUARY 27, 2024

Amazon OpenSearch Serverless is an on-demand auto scaling configuration for Amazon OpenSearch Service. We recently introduced a feature with Amazon OpenSearch Ingestion (OSI) to make this migration even more effortless. About the Authors Muthu Pitchaimani is a Search Specialist with Amazon OpenSearch Service.

Metadata 93

Metadata Data Processing Dashboards IoT 93

AWS Big Data

JUNE 12, 2023

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed Apache Kafka service that offers a seamless way to ingest and process streaming data. This job extracts data from the Kafka topics, deserializes it using the schema information from the Data Catalog table, and loads it into Amazon S3.

Management 77

Management Metadata Testing Internet of Things 77

AWS Big Data

MARCH 18, 2024

Amazon Managed Service for Apache Flink , which offers a fully managed, serverless experience in running Apache Flink applications, now supports Apache Flink 1.18.1 , the latest version of Apache Flink at the time of writing. This connector is compatible with Amazon OpenSearch Service provisioned and OpenSearch Service Serverless.

Management 98

Management Snapshot Broadcasting Optimization 98