With the exponential growth of data from so many different sources, data silos (or in other words, separate unintegrated data stores) are more prevalent than ever.
However, what is the root cause of data silos? People silos! The more that people view themselves as separated, the more that data is separated.
But aren’t people and various organizations, in truth, actually separate entities? Or are they? If they are actually separate entities, what issues does this cause?
Imagine two groups in an organization: a data science group and a data management group. The data science group is focused on creating and implementing useful models for predictive analytics. The data management group is focused on creating quality integrated information. However, the data science group does not want to talk to the data management group as it does not want to get involved in bureaucratic standards and common vocabulary issues or matters that seem to be less important than their deliverables.
The data management group prioritizes their important task of data integration and sees it as the foundation of successful systems. What I have seen in this scenario is a lack of true cohesive, unified efforts between these two groups. Hence, the data silos between the data management group and the data science group. Research suggests that data scientists are spending about 80% of their data on data quality issues. Often, they will use a data engineer (not from the data management group) to cleanse their data. Since one of the key interests of the data management group is to create data quality, doesn’t it make sense for these groups to collaborate to create a much better outcome? And yet, in my experience and the experience of many others, this does not seem to happen.
There are many other examples of various people and groups not collaborating and acting independently. For example, I have seen the human resource department of an organization say that their data is extremely sensitive (which of course it is), so therefore this type of data cannot be shared or integrated with other types of data and must be kept in a separate system. The irony is that human resource data is one of the most interrelated types of information since so much data is connected with people-oriented information.
Another example is that the financial area of an organization may keep their information in a separate accounting system and be very cautious of data sharing since their information is also extremely sensitive. I have been on data integration efforts where the financial organization will say that they are glad to share and help integrate their information, and yet when we have asked for their data, there are often significant stalls (sometimes indefinite) in getting this data.
Finally, I have been involved in many customer relationship management integration efforts, and when I ask salespeople to share and integrate their customer information so that it can be a useful enterprise resource, this is often a challenge. The underlying thought here (even though not expressed outwardly in this manner) is often, “Really, you want me to share intimidate details that I have collected from my dear customers? First of all, this information is confidential. Secondly, this information really represents power that I have.”
Does data represent power? If you look at the examples of Amazon, Google, and Facebook, it seems that those who have more data have more power. So why would people and organizations share and integrate data and give away their power?
The key operating premises in the above scenarios is that people and organizations are separate. But are they really? Or are all people and organizations already integrated and connected and we just pretend that we are not?
What Does This Have to Do with Zen?
Zen is a Japanese word for ‘awareness’. It also represents a school of Buddhism with philosophical principles. One premise of Zen (and many other philosophical teachings in other traditions) is that the more that we think we are separate, the more difficult things become. Another way to say this is that the greater our ego (our self-referencing), the more suffering we experience.
It seems to me that we can extend this thinking to the premise that the more that we think we are separate within an organization/company, the less integrated our systems are, which makes the organization weaker, less productive, more chaotic, and prone to disintegration (which literally means to ‘separate into parts’ or ‘to crumble’).
So when we think of ourselves or our teams or organizational units as separate, we tend to build separate, disintegrated systems. When we see ourselves or our projects as separate, we tend to create our own vocabulary/language for each system, develop separate hardware and software infrastructure and implementations, develop independent data stores, and this leads to ‘data silos’.
So, if we can think of ourselves as non-separated and connected, then this will lead to systems and data integration.
Although this is not the norm, I have witnessed great examples of this type of thinking in several large organizations that work extremely well as a large integrated team without much ego. For example, in one large financial institution, they adopted a standard business language and standards across the entire organization, were all working towards a common vision that they were all excited about and were able to develop extremely efficient and effective systems to support their investors. The investors themselves were an integrated part of the development process!
Many organizations incent people by pinning them against each other in competitions. In our culture and society, we tend to reward individual successes and compare our accomplishments with each other. In organizations, people are often competing for promotions, raises, and power. Thus, the idea of not acting as if we are separate is difficult to realize.
What Can We Do?
This is a difficult issue and not easy to address, yet if we are to better integrate data, we must address it.
Three possibilities for moving towards more integrated people and thus more integrated systems and data are:
- Think of ourselves and our teams as connected. Continue to ask the question, ‘How am I connected to various people, organizations and things?’ Realize that even though we experience some separation, we are also all connected to everything.
- Establish a shared and compelling purpose for your area of the business and connect it to the overall purpose of the entire organization. When the vision and purpose of an organization is clear, commonly accepted, compelling and concise (I call this the four C’s), then it is more likely that people will see themselves as integrated if they are drawn to the shared purpose and vision.
- Establish a common vocabulary. Quite often, miscommunication leads to separateness. Interestingly, the etymology of the word ‘communicate’, comes directly from the Latin ‘communicationem’ which literally means “a making common”. Luckily, the need for establishing common business glossaries has been recognized more lately. To help these types of efforts, we have developed a tool called the Universal Business Glossary, to help establish common terms and definitions by re-using constructs that have worked well in many organizations (many of these standard terms and definitions are based upon decades of research and based upon many of the concepts in the Data Model Resource Books). If you would like to learn more about this, please see Universal Business Glossary Webinar.
If we were to answer the question, to be separate or not to be separate, let’s consider that both are appropriate. While it seems that to some degree, we are separate, let’s make sure to acknowledge the opposite: that we are also not separate at all. At a deeper level, we are interrelated with all other people, all aspects of our organizations and systems, and for that matter, with everything else!
