Octopai

MARCH 31, 2021

Look for the Metadata. In order to perform accurate data lineage mapping, every process in the system that transforms or touches the data must be recorded. This metadata (read: data about your data) is key to tracking your data. Data Lineage by Tagging or Self-Contained Data Lineage.

Metadata 130

Metadata Data Transformation Business Intelligence Reporting 130

AWS Big Data

MARCH 13, 2024

You can also use the data transformation feature of Data Firehose to invoke a Lambda function to perform data transformation in batches. Athena is used to run geospatial queries on the location data stored in the S3 buckets. The firehose table stores raw, unmodified data from the Amazon Location tracker.

Analytics 94

Analytics IoT Metadata Internet of Things 94

Cloudera

DECEMBER 9, 2022

In 2021 we launched Cloudera DataFlow for the Public Cloud (CDF-PC) , addressing operational challenges that administrators face when running NiFi flows in production environments. Developers need to onboard new data sources, chain multiple data transformation steps together, and explore data as it travels through the flow.

Testing 101

Testing Cost-Benefit Interactive Visualization 101

AWS Big Data

APRIL 19, 2023

The data science algorithm Valentine is an effective tool for this. Valentine is presented in the paper Valentine: Evaluating Matching Techniques for Dataset Discovery (2021, Koutras et al.). This solution solves the interoperability and linkage problem for data products. We focus on the former.

Technology 82

Technology Data-driven Machine Learning Sales 82

AWS Big Data

APRIL 25, 2024

Alternatively, you can use AWS Glue for Apache Spark, which provides built-in support for bucketing configurations during the data transformation process. AWS Glue allows you to define bucketing parameters, such as the number of buckets and the columns to bucket on, providing an optimized data layout for efficient querying with Athena.

Optimization 65

Optimization Data Lake Cost-Benefit Reporting 65

Octopai

NOVEMBER 23, 2020

In fact, the LIBOR transition program marks one of the largest data transformation obstacles ever seen in financial services. Building an inventory of what will be affected is a huge undertaking across all of the data, reports, and structures that must be accounted for. Like any data project, it boils down to the details.

Metadata 54

Metadata Enterprise Business Intelligence Data Governance 54

Octopai

JANUARY 19, 2021

Octopai is the first BI Intelligence Platform in the Industry to Support Azure Data Factory, Providing Full Lineage of Advanced BI Tools. This is done by visualizing the Azure Data Factory pipelines’ full column-level with source-to-target traceability through different data transformations at the most detailed level.

Metadata 52

Metadata ROI Optimization Data Quality 52