AWS Big Data

MARCH 2, 2023

Apache Iceberg is an open table format for very large analytic datasets, which captures metadata information on the state of datasets as they evolve and change over time. In early 2022, AWS announced general availability of Athena ACID transactions, powered by Apache Iceberg. Starting with Amazon EMR version 6.5.0,

Data Lake 117

Data Lake Data Processing Metadata Snapshot 117

CIO Business Intelligence

FEBRUARY 28, 2023

In 2022, with the pandemic subsiding, the National Museum of African American History and Culture at the Smithsonian Institution in Washington, DC, once again served more than 1 million visitors. The project was awarded a 2022 US CIO 100 Award for leadership and innovation. And it’s working.

Metadata 98

Metadata Recreation/Entertainment Cost-Benefit Technology 98

DataRobot Blog

SEPTEMBER 1, 2022

This feature will compute some DataRobot monitoring calculations outside of DataRobot and send the summary metadata to MLOps. This strategy allows handling billions of rows per day. 1 IDC, MLOps – Where ML Meets DevOps, doc #US48544922, March 2022. It will let you independently control the scale. Request a Demo.

Metrics 145

Metrics Statistics Modeling Data Science 145

Cloudera

JUNE 7, 2022

The same study also revealed that 89% of IT decision makers agree that organizations that implement a hybrid architecture as part of its data strategy will gain a competitive advantage. Choose a hybrid data-first strategy to deliver value faster. Cloudera is the only company that makes the hybrid data strategy a reality.

IT 112

IT Data Architecture Unstructured Data Big Data 112

CIO Business Intelligence

NOVEMBER 15, 2023

This article was co-authored by Ishan Prakash , a Manager at Metis Strategy. Once relegated to a role of support, the function now can inform entire strategies as critically as it does key processes or core products. By Gartner’s estimations, overall IT spending will grow in 2024 by 8%, more than double that of 2022 and 2023—2.9%

Publishing 115

Publishing Reporting IT Finance 115

Cloudera

JANUARY 5, 2023

Data contains valuable insights for critical business decision-making, and the most innovative and successful organizations recognize data as a strategic resource that demands its own strategy. How this strategy looks depends on the organization’s unique business needs as one affects the other.

Cost-Benefit 52

Cost-Benefit Business Objectives Machine Learning Data Architecture 52

CIO Business Intelligence

APRIL 28, 2023

To ensure data sovereignty, rules may require that all related data, such as metadata, also resides locally. As of May 2022, over 900 fines have been issued for GDPR violations, the largest of which topped $877m (€746m). Two other countries require that data on their citizens remain only within national borders.

Risk 86

Risk Metadata Management Strategy 86

Alation

MAY 11, 2023

We’ve also had the pleasure of being recognised by peer user review site TrustRadius as a primary leader in the data catalog category, as well as in the data collaboration , data governance , and metadata management categories. As CMO, David will oversee our global marketing strategy.

B2B 52

B2B Finance Data Governance Marketing 52

IBM Big Data Hub

APRIL 13, 2023

Here are some of the key tables: FLIGHT_DECTREE_MODEL: this table contains metadata about the model. Examples of metadata include depth of the tree, strategy for handling missing values, and the number of leaf nodes in the tree. Loukides, Mike, AI Adoption in the Enterprise 2022. and Lawrence, N.D., 1–29.

Modeling 81

Modeling Statistics Machine Learning Testing 81

CIO Business Intelligence

JUNE 23, 2022

The same study also revealed that 89% of IT decision makers agree that organizations that implement a hybrid architecture as part of its data strategy will gain a competitive advantage. Choose a hybrid data-first strategy to deliver value faster. Cloudera is the only company that makes the hybrid data strategy a reality.

IT 73

IT Data Architecture Unstructured Data Big Data 73

AWS Big Data

FEBRUARY 24, 2023

In 2022, Amazon Athena announced support of Iceberg , enabling transaction queries on S3 objects. Athena uses the AWS Glue Data Catalog to store and retrieve table metadata for the Amazon S3 data in Iceberg format. This creates an AWS Glue database for metadata storage. Choose Add database.

Data Lake 83

Data Lake Metadata Testing Data Warehouse 83

Alation

MAY 11, 2023

We’ve also had the pleasure of being recognised by peer user review site TrustRadius as a primary leader in the data catalog category, as well as in the data collaboration , data governance , and metadata management categories. As CMO, David will oversee our global marketing strategy.

B2B 40

B2B Finance Data Governance Marketing 40

CIO Business Intelligence

MARCH 23, 2023

All data, including metadata, resides locally, making it easier to comply with residency laws and other local sovereign requirements. A sovereign cloud is compatible with multi-cloud or hybrid cloud strategies and is separate from the underlying infrastructure, preventing vendor lock-in. CLOUD Act, which could result in the U.S.

Metadata 52

Metadata Data Governance Data Collection Strategy 52

IBM Big Data Hub

MAY 9, 2022

The availability of integration strategies including ETL and ELT, data replication, change data capture and data virtualization are key so that the widest possible range of data integration can be enacted. Automated metadata generation is essential in order to turn a manual process into one that is better controlled.

Management 56

Management Data Quality Metadata Data Integration 56

TDAN

AUGUST 2, 2022

This estimate was shared by several speakers at the DGIQ 2022 conference in San Diego. The time wasted spent collecting and validating data can approach 80% of the total effort for managing data. We had previously estimated this “waste tax” to be in the 20-30% range. No matter what the actual numbers bear out, this […].

Management 52

Management Data Quality Metadata Data Governance 52

CIO Business Intelligence

SEPTEMBER 6, 2022

It took about nine weeks to set up the infrastructure, make the connection to the database, and index and understand the metadata. In those situations, he says, you need to tell yourself: “Get back to your strategy. In a first test of the technology, he used Alation to catalog a subset of Very’s data held in an old Teradata database.

IT 77

IT Forecasting Data Lake Enterprise 77

Alation

APRIL 25, 2023

between 2022 and 2029. In this way, data governance has implications for a wide range of data management disciplines, including data architecture, quality, security, metadata, and more. Or how it can elevate your sales and marketing strategies. Using bad data, or the incorrect data can generate devastating results.

Data Governance 52

Data Governance Marketing Machine Learning Sales 52

Andrew White

JANUARY 9, 2022

On Thursday January 6th I hosted Gartner’s 2022 Leadership Vision for Data and Analytics webinar. Which trends do you see for 2022 in AI & ML technology and tools and tool capabilities? We will publish a new Top Trends for D&A for 2022 in a couple of months. – We do have a job description of a CDO.

Analytics 64

Analytics Measurement Modeling Data-driven 64

AWS Big Data

JULY 3, 2023

Backtesting is a process used in quantitative finance to evaluate trading strategies using historical data. This helps traders determine the potential profitability of a strategy and identify any risks associated with it, enabling them to optimize it for better performance.

Snapshot 83

Snapshot Data Lake Testing Strategy 83

Alation

APRIL 25, 2023

between 2022 and 2029. In this way, data governance has implications for a wide range of data management disciplines, including data architecture, quality, security, metadata, and more. Or how it can elevate your sales and marketing strategies. Using bad data, or the incorrect data can generate devastating results.

Data Governance 40

Data Governance Marketing Machine Learning Sales 40

AWS Big Data

APRIL 25, 2024

With this partitioning and bucketing strategy, Athena can first eliminate partitions for irrelevant report types, and then scan only the buckets within the relevant partition that match the specified station ID, significantly reducing the amount of data processed and accelerating query runtimes. There are two folders: data and metadata.

Optimization 65

Optimization Data Lake Cost-Benefit Reporting 65

AWS Big Data

MARCH 26, 2024

In 2022, AWS commissioned a study conducted by the American Productivity and Quality Center (APQC) to quantify the Business Value of Customer 360. In this post, we discuss how you can use purpose-built AWS services to create an end-to-end data strategy for C360 to unify and govern customer data that address these challenges.

Data Strategy 68

Data Strategy Strategy Data Warehouse Prescriptive Analytics 68

IBM Big Data Hub

OCTOBER 4, 2023

It’s no longer a nice-to-have, but an integral part of a successful data strategy. from 2022 to 2026. In fact, according in an IDC DataSphere study, IDC estimated that 10,628 exabytes (EB) of data was determined to be useful if analyzed, while only 5,063 exabytes (EB) of data (47.6%) was analyzed in 2022.

Data Lake 78

Data Lake Metadata Cost-Benefit Data Warehouse 78

CIO Business Intelligence

JUNE 12, 2024

“This does work and is in use today by a growing number of companies,” says Bret Greenstein, partner and leader of the gen AI go-to-market strategy at PwC. For example, gen AI can be used to extract metadata from documents, create indexes of information and knowledge graphs, and to query, summarize, and analyze this data. “And

Modeling 132

Modeling Testing Cost-Benefit IT 132

Alation

AUGUST 11, 2022

I recently had the opportunity to connect with Mohan at Snowflake Summit 2022 in Las Vegas. We chatted about industry trends, why decentralization has become a hot topic in the data world, and how metadata drives many data-centric use cases. What are your thoughts on the centralization of metadata?

Metadata 52

Metadata Data Warehouse Data Quality Data Lake 52

Cloudera

DECEMBER 12, 2022

For a thoughtful and entertaining analysis, I strongly recommend you spend a few minutes watching the keynote session by Pat Moorhead, CEO Moor Insights & Strategy, at the Evolve 2022 Data event in New York. In March 2022, Gartner published A Strategic Roadmap for Migrating Data Management to the Cloud. Hybrid is forever.

Recreation/Entertainment 94

Recreation/Entertainment Insurance Digital Transformation Metadata 94

Alation

AUGUST 16, 2022

To arrive at a score, the writers reviewed product capabilities and strategy, combined with interviews with confirmed vendor customers. We look forward to sitting down with Stewart Bond, the lead analyst on the report, on September 8, 2022 to discuss the report and broader trends he is seeing in the data intelligence market.

Data Quality 52

Data Quality Data Governance Metadata Reporting 52

CIO Business Intelligence

JANUARY 30, 2024

Birgit Fridrich, who joined Allianz as sustainability manager responsible for ESG reporting in late 2022, spends many hours validating data in the company’s Microsoft Sustainability Manager tool. And ESG roadmaps should be built to leverage existing and planned investments aligned to broader IT strategy, EY’s Reppy advises.

Reporting 97

Reporting Data Quality Strategy Data-driven 97

CIO Business Intelligence

SEPTEMBER 26, 2023

Combining these analytics with AIOps health analytics, cybersecurity assessments, and system metadata, gives you insights to make the best sustainability decisions about workload consolidation to reduce your IT footprint and lower emissions and energy costs. and/or its affiliates in the U.S. All rights reserved.

IT 129

IT Key Performance Indicator Software Metadata 129

Cloudera

AUGUST 4, 2021

The data is profiled and enhanced with rich metadata—including operational, social, and business context—creating trusted and reusable data assets and making them discoverable. This all requires a proactive governance strategy. Availability: Make the right data available only to the right users. Companies indeed are taking notice.

Data Governance 86

Data Governance Metadata Data-driven Enterprise 86

AWS Big Data

JUNE 9, 2023

Iceberg doesn’t optimize file sizes or run automatic table services (for example, compaction or clustering) when writing, so streaming ingestion will create many small data and metadata files. Iceberg offers a Merge On Read strategy to enable fast writes. This has impact on write performance, resulting in full table joins.

Data Lake 113

Data Lake Metadata Optimization Statistics 113

BI-Survey

FEBRUARY 23, 2023

“Data culture eats data strategy for breakfast” has become a popular saying among data and analytics managers and executives. Even the best data strategy cannot fulfill its potential if the data culture in the company does not match it. These include tools for metadata management (e.g.,

Metadata 52

Metadata Data-driven Data Strategy Strategy 52

AWS Big Data

JANUARY 25, 2023

We ended 2022 on a high note with the preview release of Amazon OpenSearch Serverless at re:Invent. To adeptly handle the two predominant workloads, OpenSearch Serverless applies different sharding and indexing strategies. Therefore, in the workflow to create a collection, you must define the collection type—time series or search.

Management 108

Management Dashboards Metadata Modeling 108

Alation

MAY 3, 2022

In a couple of weeks (May 17–19) the Alation team joins one of our favorite data events of the year: Tableau Conference 2022. See You at Tableau Conference 2022! Yet there’s still an alarming gap between finding data… and using it. Tableau CEO Mark Nelson agrees.

Data-driven 52

Data-driven Reporting Data Governance Metadata 52

AWS Big Data

MARCH 9, 2023

SCD2 metadata – rec_eff_dt and rec_exp_dt indicate the state of the record. Register source tables in the AWS Glue Data Catalog We use an AWS Glue crawler to infer metadata from delimited data files like the CSV files used in this post. It is also called the surrogate key and has a unique value that is monotonically increasing.

Slice and Dice 81

Slice and Dice Data Warehouse Metrics Metadata 81

Ontotext

MAY 5, 2023

Case Study In their talk: How Knowledge Graphs and Graph Databases Take Your Data Further, presented at Ontotext’s Knowledge Graph Forum 2022, Ragini Okhandiar and Krishna Potluri from JPMorgan Chase & Co. share their expertise in building an in-house knowledge graph.

Data Collection 64

Data Collection Risk Data-driven Interactive 64

AWS Big Data

JULY 21, 2023

See Managing LF-Tags for metadata access control for more details. The following diagram shows the structure of the datasets that we work with in this post and a tagging strategy to provide fine-grained column-level access. We then grant these principals permission to appropriate LF-tags.

Data Lake 86

Data Lake Data Warehouse Marketing Management 86