AWS Big Data

MAY 16, 2024

With the new REST API, you can now invoke DAG runs, manage datasets, or get the status of Airflow’s metadata database, trigger, and scheduler—all without relying on the Airflow web UI or CLI. This comes at the cost of running more web server instances, which is a trade-off between cost-optimization and responsiveness.

Testing 93

Testing Interactive Metrics Management 93

Cloudera

FEBRUARY 11, 2019

Cloudera provides a unified platform with multiple data apps and tools, big data management, hybrid cloud deployment flexibility, admin tools for platform provisioning and control, and a shared data experience for centralized security, governance, and metadata management.

Management 43

Management Metadata Analytics Machine Learning 43

Cloudera

AUGUST 31, 2021

Running on CDW is fully integrated with streaming, data engineering, and machine learning analytics. It has a consistent framework that secures and provides governance for all data and metadata on private clouds, multiple public clouds, or hybrid clouds. Consideration of both data & metadata in the migration.

Data Warehouse 81

Data Warehouse Cost-Benefit Metadata Data-driven 81

IBM Big Data Hub

SEPTEMBER 20, 2022

Often data scientists aren’t thrilled with the prospect of generating all the documentation necessary to meet ethical and regulatory standards. This is where technology such as IBM FactSheets , can help by reducing the manual labor needed to capture metadata and other facts about a model across stages of the AI lifecycle.

Metadata 56

Metadata Machine Learning Data-driven Modeling 56

AWS Big Data

JUNE 7, 2023

Through their unique position in ports, at sea, and on roads, they optimize global cargo flows and create sustainable customer value. Cargotec captures terabytes of IoT telemetry data from their machinery operated by numerous customers across the globe. An AWS Glue job (metadata exporter) runs daily on the source account.

Metadata 56

Metadata Data Lake Machine Learning Big Data 56

AWS Big Data

FEBRUARY 6, 2023

AWS Glue is a serverless data integration service that makes it simple to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development. MongoDB Atlas is a developer data service from AWS technology partner MongoDB, Inc.

Metadata 77

Metadata Data Lake Machine Learning Management 77

AWS Big Data

FEBRUARY 1, 2023

This introduces the need for both polling and pushing the data to access and analyze in near-real time. From an operational standpoint, we designed a new shared responsibility model for data ingestion using AWS Glue instead of internal services (REST APIs) designed on Amazon EC2 to extract the data.

Optimization 52

Optimization Forecasting Data Lake Metadata 52

AWS Big Data

NOVEMBER 22, 2023

Hudi provides tables , transactions , efficient upserts and deletes , advanced indexes , streaming ingestion services , data clustering and compaction optimizations, and concurrency control , all while keeping your data in open source file formats. This effectively provides change streams to enable incremental data pipelines.

Data Lake 97

Data Lake Snapshot Metadata Optimization 97

AWS Big Data

SEPTEMBER 7, 2023

We will partition and format the server access logs with Amazon Web Services (AWS) Glue , a serverless data integration service, to generate a catalog for access logs and create dashboards for insights. Both the user data and logs buckets must be in the same AWS Region and owned by the same account.

Metadata 103

Metadata Dashboards Metrics Visualization 103

AWS Big Data

AUGUST 31, 2023

The Iceberg specification allows seamless table evolution such as schema and partition evolution, and its design is optimized for usage on Amazon S3. Iceberg stores the metadata pointer for all the metadata files. In this post, we use the Yellow taxi public dataset from NYC Taxi & Limousine Commission as our source data.

Data Lake 97

Data Lake Data Warehouse Metadata Data Architecture 97

AWS Big Data

MAY 21, 2024

During implementation, the LINQ team worked with OpenSearch Service specialists to optimize the OpenSearch Service cluster configuration to maximize performance and optimize cost of the solution. This results in an optimized record for each product for quick and efficient search in OpenSearch Service.

Manufacturing 100

Manufacturing Sales Optimization Data Processing 100

AWS Big Data

SEPTEMBER 29, 2023

Analyzing XML files can help organizations gain insights into their data, allowing them to make better decisions and improve their operations. Analyzing XML files can also help in data integration, because many applications and systems use XML as a standard data format. This approach optimizes the use of your XML files.

Metadata 67

Metadata Visualization Data-driven Optimization 67

AWS Big Data

NOVEMBER 29, 2023

Modernizing analytics for scale, performance, and reliability “Our migration from legacy on-premises platform to Amazon Redshift allows us to ingest data 88% faster, query data 3x faster, and load daily data to the cloud 6x faster. Here’s a couple of highlights from this week and for the full list, see below.

Data Warehouse 104

Data Warehouse Data Lake Analytics Machine Learning 104

IBM Big Data Hub

OCTOBER 12, 2023

They’re built on machine learning algorithms that create outputs based on an organization’s data or other third-party big data sources. Sometimes, these outputs are biased because the data used to train the model was incomplete or inaccurate in some way.

Cost-Benefit 79

Cost-Benefit Metadata Data Governance Modeling 79

erwin

MAY 27, 2020

What, then, should users look for in a data modeling product to support their governance/intelligence requirements in the data-driven enterprise? Nine Steps to Data Modeling. Provide metadata and schema visualization regardless of where data is stored. naming and database standards, formatting options, and so on.

Modeling 98

Modeling Metadata Data Governance Visualization 98

Alation

JANUARY 25, 2022

As a reminder, here’s Gartner’s definition of data fabric: “A design concept that serves as an integrated layer (fabric) of data and connecting processes. In this blog, we will focus on the “integrated layer” part of this definition by examining each of the key layers of a comprehensive data fabric in more detail.

Metadata 52

Metadata IT Metrics Data-driven 52

AWS Big Data

MARCH 26, 2024

Data ingestion You have to build ingestion pipelines based on factors like types of data sources (on-premises data stores, files, SaaS applications, third-party data), and flow of data (unbounded streams or batch data). Then, you transform this data into a concise format.

Data Strategy 71

Data Strategy Strategy Data Warehouse Prescriptive Analytics 71

AWS Big Data

JULY 20, 2023

The Orca Platform is powered by a state-of-the-art anomaly detection system that uses cutting-edge ML algorithms and big data capabilities to detect potential security threats and alert customers in real time, ensuring maximum security for their cloud environment. Why did Orca choose Apache Iceberg?

Data Lake 89

Data Lake Analytics Snapshot Optimization 89

AWS Big Data

NOVEMBER 29, 2023

Finally, we announced new AI-driven scaling and optimizations for Amazon Redshift Serverless that enable the service to learn from your patterns and proactively scale on multiple dimensions, including concurrent users, data variability, and query complexity. We look forward to seeing what you create!

Data Warehouse 111

Data Warehouse Data Lake Data Integration Dashboards 111

erwin

FEBRUARY 4, 2021

Points of integration. Without an accurate, high-quality, real-time enterprise data pipeline, it will be difficult to uncover the necessary intelligence to make optimal business decisions. So what’s holding organizations back from fully using their data to make better, smarter business decisions? Regulations.

Data Governance 89

Data Governance Metadata Data Quality Risk Management 89

IBM Big Data Hub

JANUARY 17, 2023

IBM Cloud Pak for Data Express solutions offer clients a simple on ramp to start realizing the business value of a modern architecture. Data governance. The data governance capability of a data fabric focuses on the collection, management and automation of an organization’s data. Data integration.

Data-driven 56

Data-driven Enterprise Data Governance Prescriptive Analytics 56

Cloudera

JUNE 6, 2019

When a mix of batch, interactive, and data serving workloads are added to the mix, the problem becomes nearly intractable. While this approach provides isolation, it creates another significant challenge: duplication of data, metadata, and security policies, or ‘split-brain’ data lake. 2) By workload type.

Metadata 85

Metadata Data Lake Optimization Strategy 85

AWS Big Data

FEBRUARY 21, 2023

Loading complex multi-point datasets into a dimensional model, identifying issues, and validating data integrity of the aggregated and merged data points are the biggest challenges that clinical quality management systems face. What is a dimensional data model? It optimizes the database for faster data retrieval.

Data Warehouse 66

Data Warehouse Data Lake Cost-Benefit Modeling 66

AWS Big Data

OCTOBER 11, 2023

The serverless architecture features auto scaling, high availability, and a pay-as-you-go billing model to increase agility and optimize costs. The architecture approach is split into a data intake layer, a data analysis layer, and a data visualization layer.

Visualization 100

Visualization Dashboards Data-driven Gap analysis 100

Ontotext

MAY 30, 2019

Linked Data and Volume. Speaking about data and volume, it seems apt to start this with the famous saying that “most companies think they have “Big Data” problems while they actually have big “data problems””. Ultimately, navigating large amounts of data is not about volume but about structure.

Cost-Benefit 87

Cost-Benefit Technology Big Data Metadata 87

AWS Big Data

FEBRUARY 22, 2023

An AWS Glue ETL job, using the Apache Hudi connector, updates the S3 data lake hourly with incremental data. The AWS Glue job can transform the raw data in Amazon S3 to Parquet format, which is optimized for analytic queries. Data had to be manually processed by data analysts, and data mining took a long time.

Data Lake 52

Data Lake Dashboards Cost-Benefit Metadata 52

Ontotext

MAY 30, 2019

Linked Data and Volume. Speaking about data and volume, it seems apt to start this with the famous saying that “most companies think they have “Big Data” problems while they actually have big “data problems””. Ultimately, navigating large amounts of data is not about volume but about structure.

Cost-Benefit 75

Cost-Benefit Technology Big Data Metadata 75

AWS Big Data

JUNE 12, 2023

Acting as a bridge between producer and consumer apps, it enforces the schema, reduces the data footprint in transit, and safeguards against malformed data. AWS Glue is an ideal solution for running stream consumer applications, discovering, extracting, transforming, loading, and integrating data from multiple sources.

Management 81

Management Metadata Testing Internet of Things 81

Cloudera

JUNE 25, 2019

Intelligent Migration – Providing policy-based controls to automate data movement between on-premises file systems and cloud object stores for one-time migration and ongoing, incremental movement of both data and metadata. Enterprises can auto-scale and optimize to meet the demands of workloads.

Enterprise 87

Enterprise Machine Learning Recreation/Entertainment IoT 87

AWS Big Data

FEBRUARY 2, 2023

Customers of all sizes and industries can store and protect any amount of data for virtually any use case, such as data lakes, cloud-native applications, and mobile apps. For Report path prefix , enter cur-data/account-cur-daily. For Enable report data integration for , select Amazon Athena. Choose Next.

Reporting 89

Reporting Data Lake Management Optimization 89

Smart Data Collective

OCTOBER 13, 2021

There are countless examples of big data transforming many different industries. There is no disputing the fact that the collection and analysis of massive amounts of unstructured data has been a huge breakthrough. We would like to talk about data visualization and its role in the big data movement.

Visualization 135

Visualization Cost-Benefit Big Data Prescriptive Analytics 135

AWS Big Data

DECEMBER 18, 2023

Get a closer look at how scaling for data warehousing works in AWS with the latest introduction of AI driven scaling and optimizations in Amazon Redshift Serverless to enable better price-performance for your workloads. Discover how you can use Amazon Redshift to build a data mesh architecture to analyze your data.

Data Warehouse 102

Data Warehouse Machine Learning Data-driven Data Lake 102

AWS Big Data

MARCH 15, 2023

Infomedia was looking to build a cloud-based data platform to take advantage of highly scalable data storage with flexible and cloud-native processing tools to ingest, transform, and deliver datasets to their SaaS applications. The raw input data is stored in Amazon S3 in JSON format (called the bronze dataset layer).

Cost-Benefit 68

Cost-Benefit Data Processing Optimization Data-driven 68

AWS Big Data

NOVEMBER 29, 2023

The dbt-glue adapter democratized access for dbt users to data lakes, and enabled many users to effortlessly run their transformation workloads on the cloud with the serverless data integration capability of AWS Glue. The team uses dbt-glue to build a transformed gold model optimized for business intelligence (BI).

Data Lake 107

Data Lake Management Metrics Data Warehouse 107

O'Reilly on Data

MAY 21, 2019

Thus, many developers will need to curate data, train models, and analyze the results of models. With that said, we are still in a highly empirical era for ML: we need big data, big models, and big compute. A typical data pipeline for machine learning. Metadata and artifacts needed for audits.

Machine Learning 238

Machine Learning Technology Deep Learning Data Science 238

erwin

AUGUST 31, 2023

It delivers the ability to capture and unify the business and technical perspectives of data assets, enables effective collaboration between a variety of stakeholders, and delivers metadata-driven automation to accelerate the creation and maintenance of data sources on virtually any data management platform.

Data-driven 98

Data-driven Modeling Enterprise Structured Data 98

IBM Big Data Hub

AUGUST 4, 2023

By leveraging data services and APIs, a data fabric can also pull together data from legacy systems, data lakes, data warehouses and SQL databases, providing a holistic view into business performance. It uses knowledge graphs, semantics and AI/ML technology to discover patterns in various types of metadata.

Data Architecture 62

Data Architecture Data Lake Machine Learning Data Governance 62