AWS Big Data

DECEMBER 21, 2023

To speed up the self-service analytics and foster innovation based on data, a solution was needed to provide ways to allow any team to create data products on their own in a decentralized manner. To create and manage the data products, smava uses Amazon Redshift , a cloud data warehouse.

Data Lake 89

Data Lake Data Warehouse Data-driven B2B 89

Ontotext

FEBRUARY 12, 2024

This promotes data autonomy and enables decision-making about data domains without centralized gatekeepers. It also breaks down the code and data monolith and distributes it across the domain teams, which results in better management and scalability. However, data mesh is not about introducing new technologies.

Data-driven 59

Data-driven Data Lake Data Quality Business Objectives 59

Cloudera

DECEMBER 21, 2021

In working with thousands of customers deploying Spark applications, we saw significant challenges with managing Spark as well as automating, delivering, and optimizing secure data pipelines. We wanted to develop a service tailored to the data engineering practitioner built on top of a true enterprise hybrid data service platform.

Snapshot 121

Snapshot Data-driven Optimization Management 121

Ontotext

MARCH 27, 2024

They also don’t have features for enterprise data management such as schema language, data validation capabilities, interoperable serialization formats, or a proper modeling language. RDF is used extensively for data publishing and data interchange and is based on W3C and other industry standards.

Modeling 69

Modeling Metadata Data Quality Enterprise 69

AWS Big Data

JULY 20, 2023

With data becoming the driving force behind many industries today, having a modern data architecture is pivotal for organizations to be successful. Moreover, running advanced analytics and ML on disparate data sources proved challenging. To overcome these issues, Orca decided to build a data lake.

Data Lake 88

Data Lake Analytics Snapshot Optimization 88

AWS Big Data

APRIL 19, 2023

Data mesh is a new approach to data management. Companies across industries are using a data mesh to decentralize data management to improve data agility and get value from data. A modern data architecture is critical in order to become a data-driven organization.

Technology 82

Technology Data-driven Machine Learning Sales 82

AWS Big Data

NOVEMBER 13, 2023

Amazon Redshift is a fully managed data warehousing service that offers both provisioned and serverless options, making it more efficient to run and scale analytics without having to manage your data warehouse. These upstream data sources constitute the data producer components.

Data Warehouse 92

Data Warehouse Data Lake Analytics Data Science 92

IBM Big Data Hub

FEBRUARY 9, 2023

In our last blog , we delved into the seven most prevalent data challenges that can be addressed with effective data governance. Today we will share our approach to developing a data governance program to drive data transformation and fuel a data-driven culture. Don’t try to do everything at once!

Data Governance 76

Data Governance Business Objectives Data Quality Strategy 76

erwin

SEPTEMBER 17, 2020

With complex data architectures and systems within so many organizations, tracking data in motion and data at rest is daunting to say the least. Harvesting the data through automation seamlessly removes ambiguity and speeds up the processing time-to-market capabilities. Improved Customer and Employee Satisfaction.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

AWS Big Data

FEBRUARY 6, 2023

In this post, we dive deep into the tool, walking through all steps from log ingestion, transformation, visualization, and architecture design to calculate TCO. The tool provides a YARN log collector to connect Hadoop Resource Manager to collect YARN logs. George Zhao is a Senior Data Architect at AWS ProServe.

Dashboards 76

Dashboards Optimization Data Lake Cost-Benefit 76

CIO Business Intelligence

OCTOBER 18, 2022

With the new stadium on the horizon, the team needed to update existing IT systems and manual business and IT processes to handle the massive volumes of new data that would soon be at their fingertips. “In Analytics, Data Management Some of our systems were old. They want that information,” she says.

Data Transformation 141

Data Transformation Consulting Data Lake Reporting 141

Sisense

AUGUST 28, 2020

Uncomfortable truth incoming: Most people in your organization don’t think about the quality of their data from intake to production of insights. However, as a data team member, you know how important data integrity (and a whole host of other aspects of data management) is.

Data Integration 101

Data Integration Testing Data Quality Data-driven 101

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x

Enterprise 85

Enterprise Data Warehouse Data Lake Optimization 85

AWS Big Data

FEBRUARY 6, 2023

It enables compute such as EMR instances and storage such as Amazon Simple Storage Service (Amazon S3) data lakes to scale. For various Hadoop jobs, customers have bespoke deployment options of fully managed Amazon EMR, Amazon EMR on Amazon EKS , and EMR Serverless. George Zhao is a Senior Data Architect at AWS ProServe.

Cost-Benefit 52

Cost-Benefit Data Lake Dashboards Big Data 52

DataKitchen

JANUARY 23, 2023

Data lineage and a data catalog are better together because they provide a more complete and accurate view of the data. Data lineage provides information about the origin, history, and movement of data, while runtime operations provide information about the actions performed on data while it is being processed.

Testing 130

Testing Data Governance Data Quality Data-driven 130

Anmut

MARCH 24, 2022

We could give many answers, but they all centre on the same root cause: most data leaders focus on flashy technology and symptomatic fixes instead of approaching data transformation in a way that addresses the root causes of data problems and leads to tangible results and business success. It doesn’t have to be this way.

ROI 52

ROI Measurement Data-driven Data Transformation 52

Peter James Thomas

OCTOBER 30, 2018

Where they have, I have normally found the people holding these roles to be better informed about data matters than their peers. The same goes in general for Marketing Managers. It may well be that one thing that a CDO needs to get going is a data transformation programme. It may be to introduce or expand Data Governance.

Marketing 51

Marketing Strategy Data Architecture Data Strategy 51

AWS Big Data

APRIL 5, 2023

The company also used the opportunity to reimagine its data pipeline and architecture. A key architectural decision that Showpad took during this time was to create a portable data layer by decoupling the data transformation from visualization, ML, or ad hoc querying tools and centralizing its business logic.

Dashboards 52

Dashboards Reporting Cost-Benefit Visualization 52

AWS Big Data

MARCH 3, 2023

You can then apply transformations and store data in Delta format for managing inserts, updates, and deletes. Data ingestion – Steps 1 and 2 use AWS DMS, which connects to the source database and moves full and incremental data (CDC) to Amazon S3 in Parquet format. Let’s refer to this S3 bucket as the raw layer.

Data Lake 83

Data Lake Dashboards Metrics Metadata 83

Data Virtualization

JANUARY 20, 2021

Learn in 12 minutes: What makes a strong use case for data virtualisation How to come up with a solid Proof of Concept How to prepare your organisation for data virtualisation You’ll have read all about data virtualisation and you’ve.

Data Warehouse 59

Data Warehouse Data Architecture Data Transformation IT 59

IBM Big Data Hub

JANUARY 18, 2023

Data lakehouse: A mostly new platform. They defined it as : “ A data lakehouse is a new, open data management architecture that combines the flexibility, cost-efficiency, and scale of data lakes with the data management and ACID transactions of data warehouses, enabling business intelligence (BI) and machine learning (ML) on all data. ”.

Data Lake 52

Data Lake Data Warehouse Data-driven Metadata 52

erwin

APRIL 16, 2024

The data mesh framework In the dynamic landscape of data management, the search for agility, scalability, and efficiency has led organizations to explore new, innovative approaches. One such innovation gaining traction is the data mesh framework. This empowers individual teams to own and manage their data.

Metadata 52

Metadata Data Quality Data Governance Modeling 52

CIO Business Intelligence

NOVEMBER 7, 2023

Given the importance of sharing information among diverse disciplines in the era of digital transformation, this concept is arguably as important as ever. The aim is to normalize, aggregate, and eventually make available to analysts across the organization data that originates in various pockets of the enterprise.

Data Warehouse 140

Data Warehouse Digital Transformation Data-driven Reporting 140

CIO Business Intelligence

AUGUST 3, 2023

It also helps him democratize credit union data so it can be used to improve customer service, automate the maintenance of such data by making various types of data easier to find, and provide chains of custody and audit controls to help meet regulatory needs.

Analytics 93

Analytics Data Lake Metadata Cost-Benefit 93

IBM Big Data Hub

JULY 5, 2023

This was, without a question, a significant departure from traditional analytic environments, which often meant vendor-lock in and the inability to work with data at scale. Another unexpected challenge was the introduction of Spark as a processing framework for big data. What can you do next?

Data Lake 61

Data Lake Metadata Cost-Benefit Data Warehouse 61

Birst BI

FEBRUARY 25, 2020

Managing large-scale data warehouse systems has been known to be very administrative, costly, and lead to analytic silos. The good news is that Snowflake, the cloud data platform, lowers costs and administrative overhead. What value does this joint solution provide?

Data Warehouse 52

Data Warehouse Cost-Benefit Data Architecture Enterprise 52

Jet Global

MAY 1, 2023

Now, Delta managers can get a full understanding of their data for compliance purposes. Additionally, with write-back capabilities, they can clear discrepancies and input data. This also includes IT departments that develop and manage applications used by internal stakeholders and partners.

Analytics 52

Analytics Cost-Benefit Visualization Dashboards 52