AWS Big Data

MARCH 2, 2023

Apache Iceberg is an open table format for very large analytic datasets, which captures metadata information on the state of datasets as they evolve and change over time. Iceberg has become very popular for its support for ACID transactions in data lakes and features like schema and partition evolution, time travel, and rollback.

Data Lake 117

Data Lake Data Processing Metadata Snapshot 117

Alation

FEBRUARY 13, 2020

By leveraging Google-like smart search to find data assets; using automation and self-learning instead of burdening people with the need to manually update metadata in multiple places; and ensuring that metadata is maintained by the whole data community and is not dependent on a centralized IT team.

Machine Learning 52

Machine Learning Data Analytics Analytics Metadata 52

DataKitchen

AUGUST 3, 2021

Third-generation – more or less like the previous generation but with streaming data, cloud, machine learning and other (fill-in-the-blank) fancy tools. It’s no fun working in data analytics/science when you are the bottleneck in your company’s business processes. The post What is a Data Mesh? See the pattern?

Data Architecture 246

Data Architecture Data Lake Cost-Benefit Data Warehouse 246

AWS Big Data

NOVEMBER 15, 2023

As the organization receives data from multiple external vendors, it often arrives in different formats, typically Excel or CSV files, with each vendor using their own unique data layout and structure. In this blog post, we’ll explore a solution that streamlines this process by leveraging the capabilities of AWS Glue DataBrew.

Metadata 83

Metadata Sales Data Lake Big Data 83

AWS Big Data

SEPTEMBER 7, 2023

Customers want to know how their data is being accessed, when it is being accessed, and who is accessing it. With exponential growth in data volume, centralized monitoring becomes challenging. It is also crucial to audit granular data access for security and compliance needs. Big Data Architect. Zach Mitchell is a Sr.

Metadata 99

Metadata Dashboards Metrics Visualization 99

Ontotext

AUGUST 22, 2023

The age of Big Data inevitably brought computationally intensive problems to the enterprise. Central to today’s efficient business operations are the activities of data capturing and storage, search, sharing, and data analytics. With semantic metadata, enterprise data gets linked to one another and to external sources.

Metadata 52

Metadata Knowledge Discovery Big Data Enterprise 52

DataKitchen

AUGUST 9, 2021

The domain also includes code that acts upon the data, including tools, pipelines, and other artifacts that drive analytics execution. The domain requires a team that creates/updates/runs the domain, and we can’t forget metadata: catalogs, lineage, test results, processing history, etc., ….

Testing 246

Testing Data Lake Metadata Publishing 246

Ontotext

JANUARY 29, 2024

As companies in almost every market segment attempt to continuously enhance and modernize data management practices to drive greater business outcomes, organizations will be watching numerous trends emerge this year. Sometimes, the challenge is that the data itself often raises more questions than it answers.

Strategy 52

Strategy Management Metadata Data-driven 52

Alation

AUGUST 30, 2022

Centralization of metadata. A decade ago, metadata was everywhere. Consequently, useful metadata was unfindable and unusable. We had data but no data intelligence and, as a result, insights remained hidden or hard to come by. This universe of metadata represents a treasure trove of connected information.

Metadata 52

Metadata Data Quality Data Governance Machine Learning 52

AWS Big Data

MAY 28, 2024

Integration with the AWS Glue Data Catalog as a metadata store for Flink applications The AWS Glue Data Catalog is a centralized metadata repository for data assets across various data sources, and provides a unified interface to store and query information about data formats, schemas, and sources.

Data Processing 91

Data Processing Cost-Benefit Metadata Optimization 91

AWS Big Data

OCTOBER 4, 2023

Additionally, data owners and data stewards can make data discovery simpler by adding business context to data while balancing access governance to the data via pre-defined approval workflows in the user interface. The metadata forms types, and asset types can be used as templates for defining your assets.

Metadata 100

Metadata Data Lake Publishing Data Governance 100

AWS Big Data

APRIL 26, 2023

This is a guest blog post co-written with Corey Johnson from Huron. Having an accurate and up-to-date inventory of all technical assets helps an organization ensure it can keep track of all its resources with metadata information such as their assigned oners, last updated date, used by whom, how frequently and more.

Metadata 52

Metadata Dashboards Visualization Consulting 52

DataKitchen

JANUARY 5, 2021

Data Science Workflow – Kubeflow, Python, R. Data Engineering Workflow – Airflow, ETL. Data Visualization, Preparation – Self-service tools sucha as Tableau, Alteryx. Data Governance/Catalog (Metadata management) Workflow – Alation, Collibra, Wikis.

Testing 147

Testing Data Governance Metadata Visualization 147

Cloudera

MARCH 3, 2022

Data users in these enterprises don’t know how data is derived and lack confidence in whether it’s the right source to use. . If data access policies and lineage aren’t consistent across an organization’s private cloud and public clouds, gaps will exist in audit logs. From Bad to Worse.

Data Governance 100

Data Governance Enterprise Data Quality Metadata 100

AWS Big Data

MAY 24, 2024

Apache Spark is a powerful big data engine used for large-scale data analytics. You can use Apache Spark to process streaming data from a variety of streaming sources, including Amazon Kinesis Data Streams for use cases like clickstream analysis, fraud detection, and more.

Metadata 90

Metadata Interactive Business Objectives Management 90

erwin

OCTOBER 16, 2020

And this time sensitivity is a massive issue, as taking a proactive and data-driven approach can literally mean life or death to your business or to your customers. And that’s where data analytics can play a huge role. There’s a common denominator in what they’re all missing, and that is data intelligence.

Internet of Things 145

Internet of Things Data-driven Uncertainty Data Governance 145

AWS Big Data

SEPTEMBER 29, 2023

To analyze XML files stored in Amazon S3 using AWS Glue and Athena, we complete the following high-level steps: Create an AWS Glue crawler to extract XML metadata and create a table in the AWS Glue Data Catalog. We use the AWS Glue crawler to extract XML file metadata. Choose Add a data store. Choose Create.

Metadata 65

Metadata Visualization Data-driven Optimization 65

IBM Big Data Hub

DECEMBER 16, 2022

An enterprise data catalog does all that a library inventory system does – namely streamlining data discovery and access across data sources – and a lot more. For example, data catalogs have evolved to deliver governance capabilities like managing data quality and data privacy and compliance.

Metadata 52

Metadata Data Quality Data-driven Data Governance 52

Cloudera

OCTOBER 11, 2022

Al needs machine learning (ML), ML needs data science. Data science needs analytics. And they all need lots of data. Different data types need different types of analytics – real-time, streaming, operational, data warehouses. And that data is likely in clouds, in data centers and at the edge.

Snapshot 94

Snapshot Data Science Digital Transformation Metadata 94

erwin

JANUARY 30, 2020

The results of our new research show that organizations are still trying to master data governance, including adjusting their strategies to address changing priorities and overcoming challenges related to data discovery, preparation, quality and traceability. And close to 50 percent have deployed data catalogs and business glossaries.

Data Governance 145

Data Governance Metadata Cost-Benefit Digital Transformation 145

Cloudera

JUNE 7, 2022

Want to manage and analyze data of all types including machine, structured, transactional, and unstructured – anywhere? Only Cloudera has the power to span multi-cloud and on-premises with a hybrid data platform. Common security, governance, metadata, replication, and automation enable CDP to operate as an integrated system.

IT 112

IT Data Architecture Unstructured Data Big Data 112

Cloudera

OCTOBER 15, 2021

With FSO, Apache Ozone guarantees atomic directory operations, and renaming or deleting a directory is a simple metadata operation even if the directory has a large set of sub-paths (directories/files) within it. In fact, this gives Apache Ozone a significant performance advantage over other object stores in the data analytics ecosystem.

Testing 87

Testing Measurement Optimization Metadata 87

Ontotext

NOVEMBER 18, 2021

Picture this – you start with the perfect use case for your data analytics product. And all of them are asking hard questions: “Can you integrate my data, with my particular format?”, “How well can you scale?”, “How many visualizations do you offer?”. Nowadays, data analytics doesn’t exist on its own.

Visualization 98

Visualization Reporting Metadata Enterprise 98

DataKitchen

MAY 4, 2023

Chris talks about the idea of a ‘domain’ as a principle of Data Mesh. A domain is a unit that includes integrated or raw data, artifacts created from data, the code that acts upon the data, the team responsible for the data, and metadata such as data catalog, lineage, and processing history.

Measurement 130

Measurement Data-driven Testing Cost-Benefit 130

IBM Big Data Hub

JANUARY 5, 2023

Monitor and identify data quality issues closer to the source to mitigate the potential impact on downstream processes or workloads. Efficiently adopt data platforms and new technologies for effective data management. Apply metadata to contextualize existing and new data to make it searchable and discoverable.

Data Quality 52

Data Quality Data Architecture Strategy Data Lake 52

AWS Big Data

MARCH 18, 2024

In these cases, you may want an integrated query layer to seamlessly run analytical queries across these diverse cloud stores and streamline your data analytics processes. The AWS Glue Data Catalog holds the metadata for Amazon S3 and GCS data.

Data Lake 99

Data Lake Analytics Cost-Benefit Management 99

Cloudera

JANUARY 19, 2024

As described in our recent blog post , an SQL AI Assistant has been integrated into Hue with the capability to leverage the power of large language models (LLMs) for a number of SQL tasks. This is a real game-changer for data analysts on all levels and will make SQL development faster, easier, and less error-prone.

Data Warehouse 72

Data Warehouse Data Processing Optimization Modeling 72

IBM Big Data Hub

JUNE 8, 2022

Lack of a common business vocabulary across your organization’s data and the inability to map those categories to existing data leads to inconsistency of business metrics and data analytics in addition to making it difficult for users to easily find and understand the data.

Data-driven 56

Data-driven Enterprise Data Governance Data Lake 56

Cloudera

FEBRUARY 11, 2019

The January 2019 “Magic Quadrant for Data Management Solutions for Analytics” provides valuable insights into the status, direction, and players in the DMSA market. In this blog, we share our takeaways as they relate to the DMSA market trends. Cloudera’s Positioning.

Management 43

Management Metadata Analytics Machine Learning 43

Ontotext

FEBRUARY 23, 2023

The engines must facilitate the advanced data integration and metadata data management scenarios where an EKG is used for data fabrics or otherwise serves as a data hub between diverse data and content management systems.

Publishing 52

Publishing Metadata Optimization Testing 52

Andrew White

DECEMBER 9, 2019

The client had recently engaged with a well-known consulting company that had recommended a large data catalog effort to collect all enterprise metadata to help identify all data and business issues. Modern data (and analytics) governance does not necessarily need: Wall-to-wall discovery of your data and metadata.

Analytics 66

Analytics Data Lake Data Governance Metadata 66

AWS Big Data

SEPTEMBER 13, 2023

Streaming data has become an indispensable resource for organizations worldwide because it offers real-time insights that are crucial for data analytics. The escalating velocity and magnitude of collected data has created a demand for real-time analytics. This table acts as a metadata layer for the data.

Data Processing 90

Data Processing Management Interactive Metadata 90

Ontotext

JANUARY 26, 2023

The catalog stores the asset’s metadata in RDF. This allows keeping a well-defined representation of the metadata of each asset and enables using a SPARQL endpoint to query it. Towards that end authors introduce a system for integrity checks for building automation applications and using more reliable data for data analytics processes.

Interactive 52

Interactive Metadata Data Integration Data-driven 52

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. The post The Future of the Data Lakehouse – Open appeared first on Cloudera Blog.

Data Lake 94

Data Lake Data Warehouse Machine Learning Cost-Benefit 94

Cloudera

JANUARY 5, 2023

We have been working with APAC organizations to operationalize data analytics and AI solutions to unlock data-driven decision-making and operational efficiency, with them quickly seeing distinct business benefits. These features provide businesses with a common metadata, security, and governance model across all their data.

Cost-Benefit 52

Cost-Benefit Business Objectives Machine Learning Data Architecture 52

Sisense

JULY 24, 2019

In my previous blog , I wrote about Natural Language Query (NLQ, or search analytics for some), as one of the major topics that we, the AI group in Sisense, are working on. NLQ is one of the oldest AI disciplines, but we’ve only recently started hearing about it in conjunction with BI and analytics. Using AI to its Fullest.

Analytics 89

Analytics Experimentation Metadata Big Data 89

Ontotext

FEBRUARY 26, 2024

RDF tends to be more data and metadata -centric. There’s an additional effort around controlled vocabularies, using IRIs to resolve entities, leveraging Linked Open Data to enrich metadata, and managing complex ontologies and taxonomies. You also have controlled vocabularies and better data hygiene with data validation.

Technology 52

Technology Cost-Benefit Data-driven Metadata 52