article thumbnail

Enhance monitoring and debugging for AWS Glue jobs using new job observability metrics, Part 3: Visualization and trend analysis using Amazon QuickSight

AWS Big Data

In Part 2 of this series, we discussed how to enable AWS Glue job observability metrics and integrate them with Grafana for real-time monitoring. In this post, we explore how to connect QuickSight to Amazon CloudWatch metrics and build graphs to uncover trends in AWS Glue job observability metrics.

Metrics 104
article thumbnail

Try semantic search with the Amazon OpenSearch Service vector engine

AWS Big Data

We’ve put together two demos on the public OpenSearch Playground to show you the strengths and weaknesses of the different techniques: one comparing textual vector search to lexical search, the other comparing cross-modal textual and image search to textual vector search. In the text box at the top, enter the query tennis clothes.

Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Run Kinesis Agent on Amazon ECS

AWS Big Data

It also emits Amazon CloudWatch metrics to help you better monitor and troubleshoot the streaming process. log and publish them to a Kinesis Data Firehose delivery stream called kinesis-agent-demo : { "firehose.endpoint": "firehose.ap-southeast-2.amazonaws.com", The agent handles file rotation, checkpointing, and retry upon failures.

Testing 93
article thumbnail

Bringing More AI to Snowflake, the Data Cloud

DataRobot Blog

This includes: Supporting Snowflake External OAuth configuration Leveraging Snowpark for exploratory data analysis with DataRobot-hosted Notebooks and model scoring. We recently announced DataRobot’s new Hosted Notebooks capability. Learn more about DataRobot hosted notebooks. Learn more at DataRobot.com/Snowflake.

article thumbnail

Monitor Apache Spark applications on Amazon EMR with Amazon Cloudwatch

AWS Big Data

In this post, we demonstrate how to publish detailed Spark metrics from Amazon EMR to Amazon CloudWatch. By default, Amazon EMR sends basic metrics to CloudWatch to track the activity and health of a cluster. Solution overview This solution includes Spark configuration to send metrics to a custom sink.

Metrics 93
article thumbnail

Build a serverless log analytics pipeline using Amazon OpenSearch Ingestion with managed Amazon OpenSearch Service

AWS Big Data

In the demo, you use the AWS Cloud9 EC2 instance profile’s credentials to sign requests sent to OpenSearch Ingestion. In this demo, the OpenSearch Service domain uses fine-grained access control for authentication, so you need to map the OpenSearch Ingestion pipeline role to the OpenSearch backend role all_access.

article thumbnail

What you need to know about product management for AI

O'Reilly on Data

But there’s a host of new challenges when it comes to managing AI projects: more unknowns, non-deterministic outcomes, new infrastructures, new processes and new tools. An AI pilot project, even one that sounds simple, probably won’t be something you can demo quickly. AI doesn’t fit that model.