Data Lineage Tools: How BI Managers Are Using Them – Straight From the Horse’s Mouth

For the past year, I’ve had the opportunity to speak with scads of BI professionals from multiple verticals, in dozens of countries on a daily basis. From Data Analysts, BI Architects, Developers, and Data Governance Leaders to IT and BI Managers. If you’re in the Business Intelligence world, there is a chance that you and I had a brief chat about metadata. Seriously. 

I’m part of the Business Development team at Octopai whose automated metadata management and analysis platform enables business intelligence and analytics teams to get better visibility of their data flow so they can make better business decisions.  In my many encounters with BI professionals, I always start by asking what use case relating to metadata management and data lineage is most challenging to their team. Here’s what I found: 

Data always tells the truth, even when it lies. Meaning you don’t always get the full picture. The biggest challenge BI groups face today is actually finding their data.  “Where is the data” is literally the question they’re all asking, all the time. Why? Because organizations are finding it nearly impossible to keep up with the massive amounts of data they are dealing with on a daily basis.  This is where metadata management comes into play. 

According to Urban Dictionary metadata is a fancy word for “information” invented by tech folks to make their jobs sound harder than they really are. Simply put, it’s the description of the data. Data is so highly detailed and complex; we need metadata to interpret it. Metadata is fickle and BI depends on it more and more every day. Metadata moves pretty fast, if you don’t stop and look around once in a while, you could miss it. Therefore, more and more BI professionals are looking for automated metadata discovery and lineage tools.

How BI Managers Use Data Lineage Tools – Straight From the Horse’s Mouth

“We are transitioning from one BI system to another.” This is super common, and I hear it all the time. The team is putting in hours upon hours of manual work on a complex project to migrate to a new tool. If you have ever migrated from SSRS to Tableau for example, you know very well how difficult and tedious it is to manually go through thousands upon thousands of endless fields and validating its accuracy. Streamlining this process could save months of wasted resources.

“Inaccuracies in reports.” Ah-ha!. Yet another one I hear constantly. When BI managers get complaints from business users about inaccuracies in their reports, it often ends up taking several data analysts about 3-5 days of manual data mapping to reverse engineer a report to locate the source of an error. These reporting errors often go unnoticed, leaving the organizations to make poor business decisions based on poor data.

“We are implementing new data governance programs.” BI professionals have been struggling with implementing and maintaining data governance programs.  They are manually tracing the data flow to figure out where their data came from, and there are just too many systems to go through.  They simply don’t have the time or resources to focus on this. Metadata management is the foundation for data governance, and without a centralized automated solution, data governance strategies are essentially handicapped.  Octopai is the perfect complementary solution for implementing Data Governance, data catalogs, data dictionaries business glossaries and more. For example, if you are implementing Collibra for data governance, Octopai can seamlessly integrate with them to show you full end-to-end data lineage in seconds. 

“Better communication and sharing of metadata within the team.” I’ve heard this many times, “Oh data discovery, you say? That’s my colleague Lenny’s thing, she’s the expert we go to for this.” If a team wants to be more efficient, they shouldn’t have to depend on that one tech wizard on the team that everyone is constantly bugging. The team can immensely boost efficiency with a centralized user-friendly platform that can be accessed and shared with anyone and everyone on the team. And one of the best parts is that it can be up and running within just a few hours. 

“Discovery and mapping of data for compliance (i.e. GDPR or CCPA).” It’s like searching for a needle in a haystack. Tireless days and nights manually mapping of data and ripping your hair out searching through tables, fields, views, etc. only to find out you’ve been looking in places it doesn’t even exist. Why go in blind? With automated metadata discovery, you can do any search and in seconds see where you need to look and more importantly where you don’t need to look so you’re not wasting your time on a wild goose chase. 

“My team is going to build a business glossary.” Getting a business glossary up and running is a project that will take BI teams months to complete and maintaining it is a project in and of itself. You’ll need to hire a team of professionals to manually input everything one at a time or you can use an Automated Business Glossary to automate the creation and management, keeping it up-to-date with no manual work giving a common business language to both IT and business users. Forget the months-long project, this set up will take just a few hours to implement.  

“Impact analysis.” From my experience speaking with different BI managers, the most shared use case is data lineage for impact analysis. BI groups are constantly building or changing business processes. When someone is changing a view, drops a column, changes a calculation, they don’t know what the impact could be downstream. As usual, they are doing it manually, documenting everything, putting their finger to the wind, not knowing what the impact will be on any reports upstream. It’s the perfect storm, it’s inefficient, a waste of labor, a waste of time and most importantly a big waste of money.

I see Octopai as Waze for BI. You simply type in your search and within seconds, you get a clear view of the map from point A to point B to C and D and back. It takes a single click to discover the exact ETL processes and tables that make up a report and you’ll see where the data is going, how it got there, the transitions it went through, discover new patterns and predict future events. The best part is it’s not a long implementation project, and within 24 hours all the metadata is centralized in one place for unlimited access by everyone in the BI group.