erwin

SEPTEMBER 10, 2020

Not Documenting End-to-End Data Lineage Is Risky Busines – Understanding your data’s origins is key to successful data governance. Not everyone understands what end-to-end data lineage is or why it is important. Data Lineage Tells an Important Origin Story. Who are the data owners?

Data Governance 131

Data Governance Key Performance Indicator Digital Transformation Metadata 131

Octopai

JANUARY 30, 2023

Amazon , Azure and Google all happily offer cloud data storage cost calculators that will make your head spin with their specificity. How many TiB of data do you need for Streaming Reads on Google BigQuery? Step 1: Investigation One of the simplest ways of reducing data storage costs is to store less data. Obvious, yes.

Optimization 83

Optimization Data Warehouse Reporting IT 83

Octopai

APRIL 23, 2022

If, however, your organization lacks the funds for a time machine (along with the funds for unlimited data intelligence solutions), here are our top four picks for the situations in which you should prioritize initiating a data lineage process. You’re planning a data system migration. The data preparation.

Visualization 89

Visualization Metrics Reporting Software 89

DataKitchen

AUGUST 9, 2021

Below is our third post (3 of 5) on combining data mesh with DataOps to foster greater innovation while addressing the challenges of a decentralized architecture. We’ve talked about data mesh in organizational terms (see our first post, “ What is a Data Mesh? ”) and how team structure supports agility.

Testing 246

Testing Data Lake Metadata Publishing 246

Alation

AUGUST 18, 2022

How do you approach data lineage? We all know that data lineage is a complex and challenging topic. In this blog, I am drilling into something I’ve been thinking about and studying for a long time: fundamental approaches to lineage creation and maintenance. What Is Data Lineage Creation & Maintenance?

Metadata 69

Metadata Visualization Statistics Data Architecture 69

erwin

SEPTEMBER 17, 2020

Replace manual and recurring tasks for fast, reliable data lineage and overall data governance. It’s paramount that organizations understand the benefits of automating end-to-end data lineage. The importance of end-to-end data lineage is widely understood and ignoring it is risky business.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

Octopai

APRIL 20, 2023

Data mesh is an approach to data architecture that is intentionally distributed, where data is owned and governed by domain-specific teams who treat the data as a product to be consumed by other domain-specific teams. What are the principles behind data mesh architecture?

Data-driven 77

Data-driven Data Architecture Sales Modeling 77

DataKitchen

DECEMBER 6, 2022

The document they wrote is exceptionally close to what we see in the market and what our products do ! This document is essential because buyers look to Gartner for advice on what to do and how to buy IT software. The two things we are most excited about are: First, DataOps is distinct from all Data Analytic tools.

Software 130

Software Marketing Data Lake Testing 130

Octopai

OCTOBER 2, 2022

The new coverage of these systems further broadens Octopai’s support of data landscapes in a singular platform. . Octopai’s platform enables data teams by providing them with complete end-to-end lineage across multiple systems regardless of being on-prem or cloud-based.

Business Intelligence 86

Business Intelligence Visualization Enterprise Management 86

erwin

FEBRUARY 28, 2024

At Sparkle, we’re a holistic data partner helping organizations increase their data maturity in a strategic yet pragmatic way. One of the key ingredients to ensure data is really embedded in an organization, and one of the key enablers to increase the strategic impact of data, is the setup of a successful data governance program.

Metadata 52

Metadata Data Governance Data Quality Technology 52

DataKitchen

AUGUST 24, 2023

Since the first edition of the DataOps Cookbook in 2019, we have talked with thousands of companies about their struggles to deliver data-driven insight to their customers. They have built data and analytic systems with great hope of success. As an industry, we have a conceptual hole in how we think about data analytic systems.

Data-driven 130

Data-driven Forecasting Metrics Dashboards 130

CIO Business Intelligence

APRIL 25, 2024

At Salesforce World Tour NYC today, Salesforce unveiled a new global ecosystem of technology and solution providers geared to help its customers leverage third-party data via secure, bidirectional zero-copy integrations with Salesforce Data Cloud. It works in Salesforce just like any other native Salesforce data,” Carlson said.

Data Integration 60

Data Integration Data Lake Metadata Data Warehouse 60

Smart Data Collective

APRIL 20, 2022

Errors in data entry might have serious effects if they are not discovered quickly. Human mistake is the most common cause of data entry errors. Since typical data entry errors may be minimized with the right steps, there are numerous data lineage tool strategies that a corporation can follow.

Data Processing 135

Data Processing Data Collection Technology Optimization 135

Octopai

JULY 17, 2022

will be met with a blank stare – because your conversation partner has no English and therefore no way of processing what you just said. Worse is if your “good morning” sounds like something else in the local language, and your addressee assumes he did process what you said. What monetary standard are you coming from?

Machine Learning 84

Machine Learning Visualization Enterprise Software 84

Cloudera

FEBRUARY 11, 2022

However, to understand what Ethical AI is, we need to have at least a basic understanding of ML, ML models and the data science lifecycle and how they are related. What is Machine Learning. Instead, they are learned by training a model on data. Model performance generally improves with access to more data.

Machine Learning 93

Machine Learning Sales Data Science Modeling 93

Octopai

MARCH 23, 2022

Dirty Meat… and Dirty Data. What caused the atrocious quality issues in the meatpacking industry pre-1906? But even though “dirty meat” is a small concern, “dirty data” is the scourge of any industry that relies heavily on information systems. Cleaning Up Dirty Data. How could we solve those for data?

Data Quality 91

Data Quality Reporting Modeling Interactive 91

Octopai

FEBRUARY 24, 2021

Change is usually incremental: migrating all that data is a slow process, not done all at once so you inevitably have your foot in the old system and the new one at the same time – for a loooooong time. Data and metadata are the glue that connects all these different hybrid system components. The Magic of Data Lineage.

Metadata 84

Metadata Reporting Data Governance Management 84

Octopai

FEBRUARY 20, 2022

That’s the data dream, right? Deciding to migrate your data from legacy, on-prem databases to a cloud-based data warehouse like Snowflake is often a step in the right direction. Let’s take a look at some concrete steps to set you up for a Snowflake data migration that will deliver better insights at a lower cost to you.

Data Warehouse 92

Data Warehouse Optimization Measurement Visualization 92

Octopai

MAY 15, 2023

How can you save your organizational data management and hosting cost using automated data lineage. Do you think you did everything already to save organizational data management costs? What kind of costs organization has that data lineage can help with? Well, you probably haven’t done this yet!

Data Quality 52

Data Quality Data Governance Data Integration Risk 52

Octopai

DECEMBER 28, 2022

For most data analysts and end-users, the vast majority of data activity is spent in the zoomed-in view of their data environment: analytics and reporting. For most data analysts and end-users, the vast majority of data activity is spent in the zoomed-in view of their data environment: analytics and reporting.

Reporting 52

Reporting Data Quality Visualization Data Warehouse 52

Cloudera

FEBRUARY 16, 2022

What if the spacing between blueberries in a muffin is reduced? What if a muffin is well-baked? What if it is an irregular shape? The complete list is shown below: Model Lineage . Machine Learning Model Lineage. classification problem. Figure 01: Is this a chihuahua or a muffin? Model Visibility.

Machine Learning 82

Machine Learning Modeling Metadata Recreation/Entertainment 82

DataKitchen

NOVEMBER 10, 2023

The Perilous State of Today’s Data Environments Data teams often navigate a labyrinth of chaos within their databases. Extrinsic Control Deficit: Many of these changes stem from tools and processes beyond the immediate control of the data team. Identifying Anomalies: Use advanced algorithms to detect anomalies in data patterns.

Data Quality 141

Data Quality Testing Data Lake Data Integration 141

Octopai

FEBRUARY 6, 2023

Do you know where your data is? Instead, more people are lifting their eyes to the heavens and answering, “Um… up there… somewhere…” Estimates of data volume in the cloud range from 60% of all corporate data to over 50% of organizations have more than 40% of their data in the cloud. Do you know what you can leave behind?

Optimization 85

Optimization Reporting IT Management 85

Octopai

MARCH 22, 2023

Data migration can be a daunting task, especially when dealing with large volumes of data. Snowflake is one of the leading cloud-based data warehouse that provides scalability, flexibility, and ease of use. It offers a range of features that make it an ideal platform for storing, managing, and analyzing large volumes of data.

Cost-Benefit 78

Cost-Benefit Data Warehouse Unstructured Data Optimization 78

Alation

FEBRUARY 7, 2023

That was my earliest taste of data lineage. It wouldn’t be until 2013 that the topic of data lineage would surface again – this time while working on a data warehouse project. Data warehouses obfuscate data’s origin In 2013, I was a Business Intelligence Engineer at a financial services company.

Metadata 52

Metadata Data Warehouse Visualization Data Quality 52

O'Reilly on Data

FEBRUARY 4, 2019

In a recent survey , we explored how companies were adjusting to the growing importance of machine learning and analytics, while also preparing for the explosion in the number of data sources. You can find full results from the survey in the free report “Evolving Data Infrastructure”.). (You

Machine Learning 277

Machine Learning Enterprise IoT Big Data 277

Octopai

DECEMBER 11, 2022

Data compliance may be good tidings for consumers, customers and clients, but it’s a big pain in the neck for the organizations that need to comply. Let’s take a look at several regulatory standards and explore automated data lineage’s role in smoothing and improving data compliance. Can you find all that data?

Insurance 52

Insurance Risk Metadata Reporting 52

CIO Business Intelligence

JANUARY 22, 2024

However, the effort to build, train, and evaluate this modeling is only a small fraction of what is needed to reap the vast benefits of generative AI technology. But as we all know, what is below the surface and not visible is significantly larger and more important. In fact, the average hospital produces 50 petabytes of data a year.

Unstructured Data 98

Unstructured Data Digital Transformation Data Strategy Modeling 98

Octopai

DECEMBER 26, 2022

Migrating a data fulfillment center (i.e. Your data warehouse is not too different from an Amazon fulfillment center. Your old data warehouse has become deprecated. Or you predict significant cost and efficiency benefits from transferring to a different data warehousing platform. What was missing from your old platform?

Data Warehouse 85

Data Warehouse Cost-Benefit Recreation/Entertainment Reporting 85

Octopai

OCTOBER 5, 2021

Your CFO finally gave the okay to purchase data catalog software. How will you choose the best data catalog software for your company? Lest the proliferation of data catalog features and options leave you groping for someone’s leftover margarita, here’s a guide with the questions to ask to cut through the overwhelm and reach clarity.

Business Intelligence 84

Business Intelligence Software Cost-Benefit Metadata 84

erwin

OCTOBER 8, 2020

So if you’re going to move from your data from on-premise legacy data stores and warehouse systems to the cloud, you should do it right the first time. And as you make this transition, you need to understand what data you have, know where it is located, and govern it along the way. Then you must bulk load the legacy data.

Data Governance 145

Data Governance Metadata Testing Data Lake 145

O'Reilly on Data

FEBRUARY 11, 2019

In a forthcoming survey, “Evolving Data Infrastructure,” we found strong interest in machine learning (ML) among respondents across geographic regions. Many companies are just beginning to address the interplay between their suite of AI, big data, and cloud technologies. Temporal data and time-series analytics. Deep Learning.

Big Data 209

Big Data Technology Machine Learning Deep Learning 209

Cloudera

AUGUST 10, 2021

Follow your data in object storage on-premises. With Apache Ozone on the Cloudera Data Platform (CDP) , they can implement a scale-out model and build out their next generation storage architecture without sacrificing security, governance and lineage. Ozone stores data as objects which live inside these buckets.

Testing 103

Testing Metadata IT Modeling 103

Cloudera

DECEMBER 1, 2022

Full-stack observability is a critical requirement for effective modern data platforms to deliver the agile, flexible, and cost-effective environment organizations are looking for. RI is a global leader in the design and deployment of large-scale, production-level modern data platforms for the world’s largest enterprises.

Metrics 82

Metrics Data Quality Data Lake Statistics 82

Octopai

MAY 9, 2021

Data lineage is also a tool. Different business intelligence jobs require different kinds of data lineage. Different business intelligence jobs require different kinds of data lineage. Similarly, the lineage type that a Data Architect would use to predict the impact of a process change also differs. .

Visualization 52

Visualization Reporting Business Intelligence Software 52

Alation

JUNE 2, 2022

In part one of this series, I discussed how data management challenges have evolved and how data governance and security have to play in such challenges, with an eye to cloud migration and drift over time. Data management is not yet a solved problem, but modern data management is leagues ahead of prior approaches.

Modeling 52

Modeling Data Governance Statistics Unstructured Data 52

Octopai

DECEMBER 30, 2020

In the financial industry, data lineage (i.e knowing where your numbers came from and what happened to them along the way) is not a nice-to-have. Here Are the Top 5 Reasons Why Financial Institutions Need Automated Data Lineage. A strong data lineage solution is a critical partner in this effort.

Finance 63

Finance Reporting Modeling Risk 63