AWS Big Data

MARCH 14, 2024

In this post, we discuss how the VMware Tanzu CloudHealth DevOps team migrated their self-managed Apache Kafka workloads (running version 2.0) to Amazon Managed Streaming for Apache Kafka (Amazon MSK) running version 2.6.2. The unwavering reliability of Kafka aligns with our commitment to data integrity.

Management 106

Management Insurance Optimization Strategy 106

AWS Big Data

MARCH 21, 2024

They have to reliably capture, process, analyze, and load the data into a myriad of data stores, all in real time. Apache Kafka is a popular choice for these real-time streaming needs. Your clients that gather the data onsite are written in Python, and they can send all the data as Apache Kafka topics to Amazon MSK.

Data Lake 94

Data Lake Management Modeling Optimization 94

IBM Big Data Hub

NOVEMBER 3, 2023

It allows your business to ingest continuous data streams as they happen and bring them to the forefront for analysis, enabling you to keep up with constant changes. Apache Kafka boasts many strong capabilities, such as delivering a high throughput and maintaining a high fault tolerance in the case of application failure.

Cost-Benefit 78

Cost-Benefit Experimentation Testing Digital Transformation 78

IBM Big Data Hub

FEBRUARY 12, 2024

At the forefront of this event-driven revolution is Apache Kafka, the widely recognized and dominant open-source technology for event streaming. While most enterprises have already recognized how Apache Kafka provides a strong foundation for EDA, they often fall behind in unlocking its true potential.

Data-driven 84

Data-driven Cost-Benefit Uncertainty Technology 84

AWS Big Data

NOVEMBER 13, 2023

Amazon MSK Serverless is a cluster type for Amazon Managed Streaming for Apache Kafka (Amazon MSK) that is the most straightforward way to run Apache Kafka clusters without having to manage compute and storage capacity. With OAUTHBEARER support, you can build clients that can work across both Amazon MSK and other Kafka environments.

Cost-Benefit 76

Cost-Benefit Management Optimization Consulting 76

Cloudera

SEPTEMBER 26, 2023

Organizations increasingly rely on streaming data sources not only to bring data into the enterprise but also to perform streaming analytics that accelerate the process of being able to get value from the data early in its lifecycle.

Technology 90

Technology IT Data-driven Data Architecture 90

Cloudera

OCTOBER 4, 2022

This blog post will provide guidance to administrators currently using or interested in using Kafka nodes to maintain cluster changes as they scale up or down to balance performance and cloud costs in production deployments. Kafka brokers contained within host groups enable the administrators to more easily add and remove nodes.

Data Processing 77

Data Processing Management Big Data Reporting 77

AWS Big Data

APRIL 18, 2024

The message bus in TIP’s existing architecture mainly uses Apache Kafka for ingesting different log types coming from the upstream systems. Kafka had a single topic for all the log types before they were consumed by different downstream applications including Splunk, Streaming Search, and Log Normalizer.

Optimization 85

Optimization Data Lake Management Key Performance Indicator 85

Analytics Vidhya

OCTOBER 23, 2020

Overview Hadoop is among the most popular tools in the data engineering and Big Data space Here’s an introduction to everything you need to. The post Introduction to the Hadoop Ecosystem for Big Data and Data Engineering appeared first on Analytics Vidhya.

Big Data 248

Big Data Analytics 248

AWS Big Data

APRIL 28, 2023

You can now use Amazon Managed Streaming for Apache Kafka (Amazon MSK) multi-VPC private connectivity (powered by AWS PrivateLink ) and cluster policy support for MSK clusters to simplify connectivity of your Kafka clients to your brokers. It also shows how to define a cluster policy for your MSK clusters.

Management 91

Management Cost-Benefit Data Processing Consulting 91

Analytics Vidhya

SEPTEMBER 22, 2022

Dale Carnegie” Apache Kafka is a Software Framework for storing, reading, and analyzing streaming data. The post Build a Simple Realtime Data Pipeline appeared first on Analytics Vidhya. Only knowledge that is used sticks in your mind.- The Internet of Things(IoT) devices can generate a large […].

Internet of Things 244

Internet of Things IoT Data Science Publishing 244

AWS Big Data

MARCH 25, 2024

Amazon MSK Connect is a fully managed service for Apache Kafka Connect. With a few clicks, MSK Connect allows you to deploy connectors that move data between Apache Kafka and external systems. About the Authors Chinmayi Narasimhadevara is a is a Solutions Architect focused on Big Data and Analytics at Amazon Web Services.

Management 81

Management Metadata Reporting Big Data 81

AWS Big Data

APRIL 4, 2024

Amazon Redshift is a recommended service for online analytical processing (OLAP) workloads such as cloud data warehouses, data marts, and other analytical data stores. These campaigns are optimized by using an AI-based bid process that requires running hundreds of analytical queries per campaign.

Big Data 65

Big Data Data Warehouse Advertising OLAP 65

AWS Big Data

JULY 20, 2023

One key component that plays a central role in modern data architectures is the data lake, which allows organizations to store and analyze large amounts of data in a cost-effective manner and run advanced analytics and machine learning (ML) at scale. To overcome these issues, Orca decided to build a data lake.

Data Lake 86

Data Lake Analytics Snapshot Optimization 86

AWS Big Data

MARCH 29, 2024

AWS offers a broad selection of managed real-time data streaming services to effortlessly run these workloads at any scale. In this post, Nexthink shares how Amazon Managed Streaming for Apache Kafka (Amazon MSK) empowered them to achieve massive scale in event processing.

Cost-Benefit 89

Cost-Benefit Data-driven Metrics Management 89

AWS Big Data

FEBRUARY 21, 2024

Towards the end of 2022, AWS announced the general availability of real-time streaming ingestion to Amazon Redshift for Amazon Kinesis Data Streams and Amazon Managed Streaming for Apache Kafka (Amazon MSK) , eliminating the need to stage streaming data in Amazon Simple Storage Service (Amazon S3) before ingesting it into Amazon Redshift.

Analytics 105

Analytics Management Data-driven Data Integration 105

AWS Big Data

MARCH 11, 2024

Amazon Redshift is a fully managed, scalable cloud data warehouse that accelerates your time to insights with fast, straightforward, and secure analytics at scale. Tens of thousands of customers rely on Amazon Redshift to analyze exabytes of data and run complex analytical queries, making it the most widely used cloud data warehouse.

Analytics 65

Analytics Data Warehouse Optimization Metrics 65

AWS Big Data

SEPTEMBER 28, 2023

In EDAs, modern event brokers, such as Amazon EventBridge and Apache Kafka, play a key role to publish and subscribe to events. EventBridge is a serverless event bus that ingests data from your own apps, software as a service (SaaS) apps, and AWS services and routes that data to targets.

Data-driven 122

Data-driven Metrics Management Publishing 122

AWS Big Data

NOVEMBER 13, 2023

Join the AWS Analytics team at AWS re:Invent this year, where new ideas and exciting innovations come together. For those in the data world, this post provides a curated guide for all analytics sessions that you can use to quickly schedule and build your itinerary. A shapeshifting guardian and protector of data like Data Lynx?

Analytics 110

Analytics Data Lake Data Warehouse Data-driven 110

AWS Big Data

NOVEMBER 13, 2023

The AWS Identity and Access Management (IAM) authentication feature in Amazon Managed Streaming for Apache Kafka (Amazon MSK) now supports all programming languages. Administrators can simplify and standardize access control to Kafka resources using IAM.

Testing 92

Testing Management Consulting IT 92

AWS Big Data

DECEMBER 13, 2023

Traditionally, customers used batch-based approaches for data movement from operational systems to analytical systems. A batch-based approach can introduce latency in data movement and reduce the value of data for analytics. usually a data warehouse) needs to reflect those changes in near real-time.

Data Warehouse 96

Data Warehouse Snapshot Data Processing Management 96

AWS Big Data

NOVEMBER 27, 2023

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that enables you to build and run applications that use Apache Kafka to process streaming data. Today, we’re excited to bring the benefits of Graviton3 to Kafka workloads, with Amazon MSK now offering M7g instances for new MSK provisioned clusters.

Cost-Benefit 95

Cost-Benefit Optimization Management Testing 95

AWS Big Data

AUGUST 31, 2023

Ingesting a high volume of streaming data has been a defining characteristic of operational analytics workloads with Amazon OpenSearch Service. Many of these workloads involve either self-managed Apache Kafka or Amazon Managed Streaming for Apache Kafka (Amazon MSK) to satisfy their data streaming needs.

Testing 105

Testing Data Processing Dashboards Management 105

AWS Big Data

MAY 17, 2023

Customers are adopting Amazon Managed Service for Apache Kafka (Amazon MSK) as a fast and reliable streaming platform to build their enterprise data hub. In addition to streaming capabilities, setting up Amazon MSK enables organizations to use a pub/sub model for data distribution with loosely coupled and independent components.

Management 88

Management Data-driven Publishing Data Processing 88

AWS Big Data

MARCH 25, 2024

In this post, we discuss why data streaming is a crucial component of generative AI applications due to its real-time nature. In-context learning LLMs are trained with point-in-time data and have no inherent ability to access fresh data at inference time.

Data Lake 88

Data Lake Unstructured Data Management Modeling 88

AWS Big Data

SEPTEMBER 6, 2023

They are being applied in numerous ways, including monitoring website traffic, tracking industrial Internet of Things (IoT) devices, analyzing video game player behavior, and managing data for cutting-edge analytics systems. Apache Kafka, a top-tier open-source tool, is making waves in this domain.

Testing 95

Testing Metadata Cost-Benefit Management 95

AWS Big Data

SEPTEMBER 13, 2023

Streaming data has become an indispensable resource for organizations worldwide because it offers real-time insights that are crucial for data analytics. The escalating velocity and magnitude of collected data has created a demand for real-time analytics. For this post, we use the kafka-console-producer.sh

Management 89

Management Data Processing Interactive Metadata 89

AWS Big Data

JANUARY 8, 2024

This is the first post to a blog series that offers common architectural patterns in building real-time data streaming infrastructures using Kinesis Data Streams for a wide range of use cases. In this post, we will review the common architectural patterns of two use cases: Time Series Data Analysis and Event Driven Microservices.

Analytics 111

Analytics IoT Data-driven Snapshot 111

AWS Big Data

MARCH 20, 2023

In this post, we share how Poshmark improved CX and accelerated revenue growth by using a real-time analytics solution. High-level challenge: The need for real-time analytics Previous efforts at Poshmark for improving CX through analytics were based on batch processing of analytics data and using it on a daily basis to improve CX.

Analytics 52

Analytics Slice and Dice Data Processing Data Lake 52

IBM Big Data Hub

JANUARY 8, 2024

They often use Apache Kafka as an open technology and the de facto standard for accessing events from a various core systems and applications. IBM provides an Event Streams capability build on Apache Kafka that makes events manageable across an entire enterprise. This approach then enables them to build new responsive applications.

Data-driven 78

Data-driven Management Advertising Enterprise 78

Cloudera

JANUARY 31, 2024

Streams Replication Manager (SRM) is an enterprise-grade replication solution that enables fault tolerant, scalable, and robust cross-cluster Kafka topic replication. SRM replicates data at high performance and keeps topic properties in sync across clusters. Starting with Cloudera Data Platform (CDP) Private Cloud Base 7.1.9,

Management 69

Management Metrics Big Data Machine Learning 69

AWS Big Data

MARCH 8, 2023

Change data capture (CDC) refers to the process of identifying and capturing changes made to data in a database and then delivering those changes in real time to a downstream system. This is especially important if you’re making time-sensitive decisions in a high-velocity data environment.

Data-driven 74

Data-driven Testing Data Processing Management 74

AWS Big Data

FEBRUARY 13, 2024

Amazon MSK Serverless is a cluster type of Amazon Managed Streaming for Apache Kafka (Amazon MSK) that makes it straightforward for you to run Apache Kafka without having to manage and scale cluster capacity. With MSK Serverless, you can use Apache Kafka on demand and pay for the data you stream and retain on a usage basis.

Data Processing 81

Data Processing Management IT Software 81

AWS Big Data

NOVEMBER 8, 2023

Learn how to utilize your datasets using Amazon SageMaker and Amazon Bedrock and popular frameworks like PyTorch with AWS compute, storage, and analytics. Also hear different architectural patterns that customers use today to harness their business data for customized generative AI solutions. Reserve your seat now!

Data-driven 87

Data-driven Data Lake Machine Learning Cost-Benefit 87

AWS Big Data

MAY 3, 2023

This post is a continuation of How SOCAR built a streaming data pipeline to process IoT data for real-time analytics and control. The key requirements for SOCAR included achieving maximum performance for real-time data analytics, which required storing data in an in-memory data store.

IoT 103

IoT Internet of Things Data Transformation Management 103

AWS Big Data

JUNE 13, 2023

This post presents a reference architecture for real-time queries and decision-making on AWS using Amazon Kinesis Data Analytics for Apache Flink. In addition, we explain why the Klarna Decision Tooling team selected Kinesis Data Analytics for Apache Flink for their first real-time decision query service.

Data Analytics 52

Data Analytics Analytics Risk Snapshot 52

AWS Big Data

JUNE 6, 2023

In the first post of the series, we described some core concepts of Apache Kafka cluster sizing, the best practices for optimizing the performance, and the cost of your Kafka workload. A logical stream of data in Kafka is referred to as a topic. It’s marked on all the Kafka records. log 00000000000000000000.index

Metadata 75

Metadata Optimization Management Metrics 75