AWS Big Data

DECEMBER 13, 2023

In collaboration with AWS, BMS identified a business need to migrate and modernize their custom extract, transform, and load (ETL) platform to a native AWS solution to reduce complexities, resources, and investment to upgrade when new Spark, Python, or AWS Glue versions are released.

Metadata 86

Metadata Data Lake Visualization Data Transformation 86

Alation

JULY 19, 2021

What Is Metadata? Metadata is information about data. A clothing catalog or dictionary are both examples of metadata repositories. Indeed, a popular online catalog, like Amazon, offers rich metadata around products to guide shoppers: ratings, reviews, and product details are all examples of metadata.

Metadata 105

Metadata Management Data Governance Machine Learning 105

CIO Business Intelligence

APRIL 25, 2024

At Salesforce World Tour NYC today, Salesforce unveiled a new global ecosystem of technology and solution providers geared to help its customers leverage third-party data via secure, bidirectional zero-copy integrations with Salesforce Data Cloud. CRM Systems, Data Management, Salesforce.com You can do flows and automations on it.

Data Integration 62

Data Integration Data Lake Metadata Data Warehouse 62

AWS Big Data

FEBRUARY 12, 2024

This is the second post in this series; we recommend that you read this first post before diving deep into this solution. AWS Management Console access to launch AWS CloudFormation templates. Up-to-date information and analysis empowered organizations to monitor the rapidly changing situation and direct resources accordingly.

Publishing 89

Publishing Dashboards Metadata Visualization 89

CIO Business Intelligence

MARCH 24, 2023

It encompasses the people, processes, and technologies required to manage and protect data assets. The Data Management Association (DAMA) International defines it as the “planning, oversight, and control over management of data and the use of data and data-related sources.”

Data Governance 142

Data Governance Management Metadata Data Quality 142

Cloudera

APRIL 2, 2024

I am pleased to announce that Cloudera has achieved FedRAMP “In Process”, a significant milestone that underscores our commitment to providing the public sector with secure and reliable data management solutions across on-prem, hybrid and multi-cloud environments.

Metadata 96

Metadata Measurement Management Enterprise 96

AWS Big Data

NOVEMBER 17, 2023

Amazon DynamoDB is a fully managed, serverless, key-value NoSQL database designed to run high-performance applications at any scale. These include internet-scale web and mobile applications, low-latency metadata stores, high-traffic retail websites, Internet of Things (IoT) and time series data, online gaming, and more.

Visualization 81

Visualization Metadata Testing Internet of Things 81

Ontotext

MARCH 19, 2021

way we package information has a lot to do with metadata. The somewhat conventional metaphor about metadata is the one of the library card. This metaphor has it that books are the data and library cards are the metadata helping us find what we need, want to know more about or even what we don’t know we were looking for.

Metadata 124

Metadata Enterprise Publishing Management 124

AWS Big Data

OCTOBER 3, 2023

Open table formats, such as Apache Iceberg , provide a solution to this issue. Apache Iceberg enables transactions on data lakes and can simplify data storage, management, ingestion, and processing. In this method, the metadata are recreated in an isolated environment and colocated with the existing data files.

Data Lake 100

Data Lake Metadata Snapshot Recreation/Entertainment 100

AWS Big Data

FEBRUARY 27, 2024

Customers prefer to let the service manage its capacity automatically rather than having to manually provision capacity. Until now, customers have had to rely on using custom code or third-party solutions to move the data between provisioned OpenSearch Service domains and OpenSearch Serverless.

Metadata 93

Metadata Data Processing Dashboards IoT 93

CIO Business Intelligence

MAY 24, 2022

Just after launching a focused data management platform for retail customers in March, enterprise data management vendor Informatica has now released two more industry-specific versions of its Intelligent Data Management Cloud (IDMC) — one for financial services, and the other for health and life sciences.

Finance 140

Finance Management Metadata Data Quality 140

CIO Business Intelligence

OCTOBER 13, 2023

The data architect is responsible for visualizing and designing an organization’s enterprise data management framework. Solutions data architect: These individuals design and implement data solutions for specific business needs, including data warehouses, data marts, and data lakes.

Data Architecture 131

Data Architecture Data Warehouse Statistics Visualization 131

Octopai

OCTOBER 18, 2021

First, what active metadata management isn’t : “Okay, you metadata! Now, what active metadata management is (well, kind of): “Okay, you metadata! Metadata are the details on those tools: what they are, what to use them for, what to use them with. . That takes active metadata management.

Metadata 83

Metadata Management IT Data Quality 83

Ontotext

AUGUST 22, 2023

Poor data management, data silos, and a lack of a common understanding across systems and/or teams are the root cause that prohibits an organization from scaling the business in a dynamic environment. Enter the Semantic Edge Era: How to Derive Value from Semantic Metadata The problem with Big Data is not the data itself.

Metadata 52

Metadata Knowledge Discovery Big Data Enterprise 52

AWS Big Data

MARCH 2, 2023

Apache Iceberg is an open table format for very large analytic datasets, which captures metadata information on the state of datasets as they evolve and change over time. Apache Iceberg addresses customer needs by capturing rich metadata information about the dataset at the time the individual data files are created. AWS Glue 3.0

Data Lake 116

Data Lake Data Processing Metadata Snapshot 116

Ontotext

JANUARY 29, 2024

As companies in almost every market segment attempt to continuously enhance and modernize data management practices to drive greater business outcomes, organizations will be watching numerous trends emerge this year. I expect to see the following data and knowledge management trends emerge in 2024.

Strategy 52

Strategy Management Metadata Data-driven 52

IBM Big Data Hub

NOVEMBER 28, 2022

Metadata management performs a critical role within the modern data management stack. However, as data volumes continue to grow, manual approaches to metadata management are sub-optimal and can result in missed opportunities. This puts into perspective the role of active metadata management.

Metadata 57

Metadata Data Quality Data-driven Data Governance 57

AWS Big Data

JUNE 2, 2023

Amazon Redshift is a widely used, fully managed, petabyte-scale cloud data warehouse. We use AWS Glue , a fully managed, serverless, ETL (extract, transform, and load) service, and the Google BigQuery Connector for AWS Glue (for more information, refer to Migrating data from Google BigQuery to Amazon S3 using AWS Glue custom connectors ).

Metadata 108

Metadata Data Warehouse Big Data Testing 108

AWS Big Data

OCTOBER 3, 2023

In our previous post Improve operational efficiencies of Apache Iceberg tables built on Amazon S3 data lakes , we discussed how you can implement solutions to improve operational efficiencies of your Amazon Simple Storage Service (Amazon S3) data lake that is using the Apache Iceberg open table format and running on the Amazon EMR big data platform.

Optimization 108

Optimization Snapshot Data Lake Metadata 108

Octopai

AUGUST 8, 2023

The partnership combines Octopai’s ability to automate and centralize business intelligence metadata with Demand Chain AI’s robust forecasting and optimization capabilities. “Rick Davis, Founder and CEO of Demand Chain AI, said, “We are thrilled to combine our strengths with Octopai to bring an innovative solution to the market.

Digital Transformation 52

Digital Transformation Metadata Forecasting Data-driven 52

Alation

MARCH 2, 2023

Analysis, however, requires enterprises to find and collect metadata. In fact, Gartner’s “Market Guide for Active Metadata Management” points to “ active metadata management ” as the key to continuous data analysis – which supports smarter human usage and more valuable insights. What Is Active Metadata Management?

Metadata 52

Metadata Marketing IT Data Quality 52

Octopai

NOVEMBER 10, 2021

Metadata has been defined as the who, what, where, when, why, and how of data. Without the context given by metadata, data is just a bunch of numbers and letters. But going on a rampage to define, categorize, and otherwise metadata-ize your data doesn’t necessarily give you the key to the value in your data. Hold on tight!

Metadata 89

Metadata Management Interactive Strategy 89

erwin

DECEMBER 3, 2020

This data explosion has overwhelmed most organizations, making it nearly impossible for them to manage much less put to smart, strategic use. Time sensitivity in data management and analytics is a massive issue. Data needs to fuel rapid decisions that make your organization more effective, customer-centric and competitive.

Metadata 141

Metadata Data Governance Data-driven Digital Transformation 141

AWS Big Data

APRIL 8, 2024

Overview of the Amazon DataZone integration with Lake Formation hybrid access mode Amazon DataZone is a fully managed data management service to catalog, discover, analyze, share, and govern data between data producers and consumers in your organization. For some of these assets, Amazon DataZone can automatically manage access grants.

Finance 76

Finance Sales Publishing Metadata 76

AWS Big Data

FEBRUARY 29, 2024

The need for an end-to-end strategy for data management and data governance at every step of the journey—from ingesting, storing, and querying data to analyzing, visualizing, and running artificial intelligence (AI) and machine learning (ML) models—continues to be of paramount importance for enterprises.

Data Governance 93

Data Governance Unstructured Data Metadata Data Lake 93

CIO Business Intelligence

MARCH 4, 2024

It can empower individual business units to manage their own data domains and address many of the common challenges facing carrier data environments, all allowing for faster decision-making, and more agile environments that evolve with the data rather than fighting to continuously force ubiquitous harmonization.

Insurance 98

Insurance Strategy Data Governance Data Integration 98

CIO Business Intelligence

FEBRUARY 14, 2024

Fundamentals like security, cost control, identity management, container sprawl, data management, and hardware refreshes remain key strategic areas for CIOs to deal with. In fact, for security, compliance, and efficiency reasons, CIOs will want to carefully manage which data generative AI has access to.

IT 143

IT Metadata Data-driven Management 143

IBM Big Data Hub

OCTOBER 4, 2022

According to Gartner 54% of models are stuck in pre-production because there is not an automated process to manage these pipelines and there is a need to ensure the AI models can be trusted.” Challenges around managing risk. The solution: AI Governance. The three foundational capabilities of the IBM AI Governance solution.

Metadata 102

Metadata Risk Risk Management Experimentation 102

Timo Elliott

JANUARY 6, 2022

Analyze your metadata. If you’ve been properly managing your metadata as part of a broader data governance policy, you can use metadata management explorers to reveal silos of dark data in your landscape. If storage costs are escalating in a particular area, you may have found a good source of dark data.

IT 133

IT Metadata Data-driven Data Governance 133

AWS Big Data

JUNE 15, 2023

In today’s world, customers manage vast amounts of data in their Amazon Simple Storage Service (Amazon S3) data lakes, which requires convoluted data pipelines to continuously understand the changes in the data layout and make them available to consuming systems. Then we compare the query performance improvements using Athena.

Data Lake 86

Data Lake Metadata Cost-Benefit Management 86

Ontotext

APRIL 19, 2024

Recently I had the opportunity to talk with Ivaylo (Ivo for short) Kabakov, Head of Solutions at Ontotext. We mainly talked about the company’s Metadata Studio and the types of features it has that give users the options I’ve listed above. Ontotext is best known for GraphDB, the company’s semantic graph database management system.

Metadata 52

Metadata Statistics Interactive Enterprise 52

AWS Big Data

MARCH 7, 2024

Organizations often need to manage a high volume of data that is growing at an extraordinary rate. UltraWarm nodes use Amazon S3 with related caching solutions to improve performance. An Index State Management (ISM) policy automates the rollover or deletion of indexes that are older than 2 days.

Data Lake 111

Data Lake Analytics Dashboards Metrics 111

AWS Big Data

APRIL 25, 2024

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed orchestration service for Apache Airflow that you can use to set up and operate data pipelines in the cloud at scale. By using multiple AWS accounts, organizations can effectively scale their workloads and manage their complexity as they grow.

Metadata 59

Metadata Data Processing Management Testing 59

Octopai

MARCH 31, 2021

Look for the Metadata. This metadata (read: data about your data) is key to tracking your data. In other words, kind of like Hansel and Gretel in the forest, your data leaves a trail of breadcrumbs – the metadata – to record where it came from and who it really is. Let’s Get Mapping. Data Lineage by Parsing.

Metadata 130

Metadata Data Transformation Business Intelligence Reporting 130

Octopai

JUNE 26, 2020

Steve needed a robust and automated metadata management solution as part of his organization’s data governance strategy. Enterprises, such as Steve’s company, understand that they need a proper data governance strategy in place to successfully manage all the data they process. Metadata in data governance.

Metadata 84

Metadata Data Governance Enterprise Reporting 84

Ontotext

OCTOBER 11, 2023

Krasimira Bozhanova, Ontotext’s Solution Architect, and I were part of the Knowledge Graphs Panel, moderated by Maribel Acosta, TUM School of Computation, Information and Technology. Krasimira showcased how knowledge graphs, combined with semantic metadata , enhance LLMs for better content discovery, understanding, and question-answering.

Metadata 69

Metadata Cost-Benefit Marketing Modeling 69

AWS Big Data

AUGUST 16, 2023

Iceberg captures metadata information on the state of datasets as they evolve and change over time. AWS Glue crawlers will extract schema information and update the location of Iceberg metadata and schema updates in the Data Catalog. For more details, refer to Creating Apache Iceberg tables.

Data Lake 77

Data Lake Metadata Snapshot Management 77