AWS Big Data

JANUARY 17, 2024

With Amazon EMR 6.15, we launched AWS Lake Formation based fine-grained access controls (FGAC) on Open Table Formats (OTFs), including Apache Hudi, Apache Iceberg, and Delta lake. The Amazon EMR record server component supports table-, column-, row-, cell-, and nested attribute-level data filtering functionality.

Data Lake 89

Data Lake Snapshot Big Data Data-driven 89

AWS Big Data

AUGUST 31, 2023

Amazon Redshift supports querying a wide variety of data formats, such as CSV, JSON, Parquet, and ORC, and table formats like Apache Hudi and Delta. Apache Iceberg is the latest table format that is supported now in preview by Amazon Redshift. Iceberg stores the metadata pointer for all the metadata files.

Data Lake 89

Data Lake Data Warehouse Metadata Data Architecture 89

AWS Big Data

JULY 3, 2023

Backtesting is a process used in quantitative finance to evaluate trading strategies using historical data. This helps traders determine the potential profitability of a strategy and identify any risks associated with it, enabling them to optimize it for better performance.

Snapshot 80

Snapshot Data Lake Testing Strategy 80

Cloudera

FEBRUARY 22, 2022

The data explosion has to be met with new solutions, that’s why we are excited to introduce the next generation table format for large scale analytic datasets within Cloudera Data Platform (CDP) – Apache Iceberg. Apache Iceberg is a new open table format targeted for petabyte-scale analytic datasets. Key Design Goals .

Snapshot 106

Snapshot Metadata Cost-Benefit Data Architecture 106

AWS Big Data

JUNE 9, 2023

Across all use cases, permissions, data governance, and data protection are table stakes, and customers require a high level of control over data security, encryption, and lifecycle management. We also provide insights into the features and capabilities of the most common open table formats available to support various use cases.

Data Lake 109

Data Lake Metadata Optimization Statistics 109

Cloudera

JULY 13, 2023

Iceberg is an emerging open-table format designed for large analytic workloads. The Apache Iceberg project continues developing an implementation of Iceberg specification in the form of Java Library. Different query engines such as Impala, Hive, and Spark can immediately benefit from using Apache Iceberg Java Library.

Metadata 70

Metadata Snapshot Data Warehouse Statistics 70

AWS Big Data

MAY 24, 2023

Apache Iceberg is an open table format for large datasets in Amazon Simple Storage Service (Amazon S3) and provides fast query performance over large tables, atomic commits, concurrent writes, and SQL-compatible table evolution. Expiration actions – These actions define when objects expire. With the s3.delete.tags

Data Lake 80

Data Lake Snapshot Metadata Optimization 80

AWS Big Data

NOVEMBER 29, 2023

With these new AI optimizations, Amazon Redshift Serverless scales proactively and automatically with workload changes across all key dimensions —such as data volume, concurrent users, and query complexity. From price-performance improvements to zero-ETL, to generative AI capabilities, we have something for everyone.

Data Warehouse 97

Data Warehouse Data Lake Analytics Machine Learning 97

AWS Big Data

SEPTEMBER 13, 2023

AWS has invested in native service integration with Apache Hudi and published technical contents to enable you to use Apache Hudi with AWS Glue (for example, refer to Introducing native support for Apache Hudi, Delta Lake, and Apache Iceberg on AWS Glue for Apache Spark, Part 1: Getting Started ).

Data Lake 87

Data Lake Data Processing Metadata Snapshot 87

AWS Big Data

JULY 20, 2023

In this post, we describe Orca’s journey building a transactional data lake using Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and AWS Analytics. We explore why Orca chose to build a transactional data lake and examine the key considerations that guided the selection of Apache Iceberg as the preferred table format.

Data Lake 84

Data Lake Analytics Snapshot Optimization 84

AWS Big Data

AUGUST 16, 2023

Apache Iceberg is an open table format for very large analytic datasets. Iceberg manages large collections of files as tables, and it supports modern analytical data lake operations such as record-level insert, update, delete, and time travel queries. The following diagram illustrates this workflow.

Data Lake 68

Data Lake Metadata Testing Snapshot 68

IBM Big Data Hub

OCTOBER 4, 2023

It’s no longer a nice-to-have, but an integral part of a successful data strategy. By leveraging multiple fit-for-purpose query engines, organizations can optimize costly warehouse workloads, and will no longer need to keep multiple copies of data for various workloads or across repositories for analytics and AI use cases.

Data Lake 85

Data Lake Metadata Cost-Benefit Data Warehouse 85

IBM Big Data Hub

MAY 9, 2023

Through workload optimization an organization can reduce data warehouse costs by up to 50 percent by augmenting with this solution. [1] This proliferation of data spans every industry, and organizations have an opportunity to turn it into actionable insights that can inform revenue strategies and enhance operational efficiencies.

Data Warehouse 78

Data Warehouse Cost-Benefit Recreation/Entertainment Unstructured Data 78

AWS Big Data

APRIL 28, 2023

One of the most common questions we get from customers is how to effectively monitor and optimize costs on AWS Glue for Spark. One of the most common questions we get from customers is how to effectively monitor and optimize costs on AWS Glue for Spark. This section describes a way to monitor overall costs on AWS Glue for Apache Spark.

Optimization 90

Optimization Metrics Interactive Data Integration 90

IBM Big Data Hub

MARCH 14, 2024

By using fit-for-purpose databases, customers can efficiently run workloads, using the appropriate engine at the optimal cost to optimize analytics for the best price-performance.   IBM and AWS have partnered to accelerate customers’ cloud-based data modernization strategies.

Cost-Benefit 83

Cost-Benefit Metadata Optimization Management 83

IBM Big Data Hub

JULY 17, 2023

” Sean Im, CEO, Samsung SDS America “In the field of generative AI and foundation models, watsonx is a platform that will enable us to meet our customers’ requirements in terms of optimization and security, while allowing them to benefit from the dynamism and innovations of the open-source community.” IBM watsonx.ai

Machine Learning 66

Machine Learning Data Warehouse Modeling Cost-Benefit 66

IBM Big Data Hub

JUNE 28, 2023

To solve this challenge, IBM Consulting is working with partners like Amazon Web Services (AWS), who are focused on providing a platform and tool set for processing omics data at in a secure, scalable and cost optimized manner. What is Amazon Omics? This process alone saves hundreds of hours of productive time.

Informatics 67

Informatics Consulting Cost-Benefit Data Architecture 67

Sisense

MAY 11, 2021

Building the right data model is an important part of your data strategy. But this was only the tip of the analytics iceberg. You can’t talk about data analytics without talking about data modeling. Discover why. What is data modeling? When it comes to data modeling, function determines form. Working the (data modeling) process.

Modeling 84

Modeling Big Data IoT Data Warehouse 84

AWS Big Data

NOVEMBER 29, 2023

They ingest application logs into raw Parquet tables in an Amazon Simple Storage Service (Amazon S3) data lake. The team uses dbt-glue to build a transformed gold model optimized for business intelligence (BI). From the launch of the adapter, AWS has continued investing into dbt-glue to cover more requirements.

Data Lake 99

Data Lake Management Metrics Data Warehouse 99

AWS Big Data

NOVEMBER 8, 2023

Join us as we delve into the world of real-time streaming data at re:Invent 2023 and discover how you can use real-time streaming data to build new use cases, optimize existing projects and processes, and reimagine what’s possible. re:Invent is a learning conference organized by AWS for the global cloud computing community. Register now!

Data-driven 85

Data-driven Data Lake Machine Learning Cost-Benefit 85

AWS Big Data

NOVEMBER 13, 2023

Join G2 Krishnamoorthy, Vice President of AWS Analytics, to discuss strategies for embedding analytics into your applications and ideas for building a data foundation that supports your business initiatives. Join the AWS Analytics team at AWS re:Invent this year, where new ideas and exciting innovations come together.

Analytics 109

Analytics Data Lake Data Warehouse Data-driven 109

CIO Business Intelligence

JUNE 21, 2023

According to a recent poll of senior executives conducted by The Harris Poll on behalf of Insight Enterprises, 39% of companies have already established policies or strategies around generative AI and 42% are in the process of developing them. The ones who say it’s not on the table, that’s a bad mistake,” he adds.

Cost-Benefit 118

Cost-Benefit Modeling Risk Enterprise 118

Cloudera

APRIL 1, 2024

This recognition underscores Cloudera’s commitment to continuous customer innovation and validates our ability to foresee future data and AI trends, and our strategy in shaping the future of data management. Cloudera, a leader in big data analytics, provides a unified Data Platform for data management, AI, and analytics.

Unstructured Data 102

Unstructured Data Cost-Benefit Metadata Machine Learning 102