In this post, I’ll share some New Year Resolutions that I think Business Intelligence, Data Engineering, Artificial Intelligence, and Data Science teams should start doing from today. I started running teams as an AI Consultant in the early 2000s, and I’ve been adopting leadership roles and being someone who delivers technically. I get to see both sides; the business and the technology. So let’s dive in, and I’d love to hear your thoughts in the comments.
- Adopting the Ops: AIOps, MLOps, DataOps
- Adopt robust Data Engineering Practices
- Continuous transformation through investing in yourself and your teams
- If you can't get a place at the table, build your own table and invite others to pull up a chair
- Consider the ethical and social consequences of your Data Science and AI Applications
1. Adopting the Ops for Artificial Intelligence, Data Science and BI: AIOps, MLOps, DataOps
Back in the day, we didn’t call it ‘*Ops’. It was broken down into smaller processes that could loosely be called ‘productionising’, ‘retraining’, ‘deploying’ and so on. It was usually signed off by the business sponsor and IT, and then the consulting firm got paid. The IT-oriented terminology focused on the fact that Artificial Intelligence models, Business Intelligence assets such as reports, or advanced predictive algorithms were eventually looked after by IT departments. Then, the Ops terminology started to appear.
2. Adopt robust Data Engineering Practices for Business Intelligence, Artificial Intelligence and Data Science
Data Engineering is important for Artificial Intelligence and Data Science, not just Business Intelligence. Data Engineering can mean going from a Big Ball of Mud to a Big Ball of Yarn. In software terms, a Big Ball of Mud can be described as a ‘haphazardly structured, sprawling, sloppy, duct-tape-and-baling-wire, spaghetti-code jungle.’ (Reference). You can see the problems in data estates as well as software estates, where they are characterized by ‘unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated.’ (Reference).
Data Engineering: moving from a Big Ball of Mud to a Big Ball of Yarn
We hear a lot about ETL/ELT patterns, but what about ETL/ELT anti-patterns? When I see these symptoms in a Business Intelligence, Data Science or Artificial Intelligence estate, the data engineering efforts switch from building a Big Ball of Mud to unpicking a Big Ball of Yarn. Here, the engineers try to track back from the reporting front end to the source systems. Let’s assume for a moment that they have access to the source systems; if this is the case, then we often see many views, for example, duplicating effort to retrieve data. All this slows down the organisation from benefitting from their Artificial Intelligence and Data Science efforts.
I once saw that 300 different ETL jobs were retrieving data from one single SQL Server view; there was no need for 300 jobs all pulling the same data, and it took me and my team (at that time) three months to simplify and redo the architecture. In this example, I had only been in the organisation for around two weeks when I found that a developer wanted to add job #301 to the data pipeline quagmire. So, I had to put a stop to it before they caused any more stress on the system. We often blame DBAs for slow database performance. Unfortunately, the reality is that poorly-written queries or poorly thought-through jobs by Artificial Intelligence engineers, Data Scientists or BI engineers can result in an unnecessary load on an already-busy system.
Unpicking a Big Ball of Yarn is never easy; it is hard to work out where the data starts and ends and often only God knows what is happening in between those two points. This can be due to poor documentation, developer turnover, wild expectations over Artificial Intelligence or business pressure – or a combination of these issues with other things.
Getting to the end of the Big Ball of Yarn
Practically speaking, sometimes it is simply too hard to go back and fix everything that was done previously. My recommendation is that you inherit a little bit of data debt when you start your next project or sprint. Over time, you can see the results. Gently, gently, incrementally! Perhaps you can combine this initiative by adopting DataOps as a practice?
3. Continuous transformation through investing in yourself and your teams for Artificial Intelligence, BI and Data Science
When I’m running teams, I do prioritize ongoing training opportunities. Everyone needs to keep growing and adapting as the industry changes. In 2022, I succeeded in passing my Masters of Business Administration (MBA) with my thesis focused on Artificial Intelligence and Ethics. This is my third postgraduate degree.
It took years to achieve the MBA part-time while juggling single parenthood along with a busy consulting life plus a team of employees but I am glad I did it. I invested in myself for the learning and networking opportunities it has brought me. You can do the same for yourself and other people.
4. If you can't get a place at the Diversity, Equality, Equity, Inclusion or Intersectionality table.... build your own table and invite others to pull up a chair
This angers and upsets me so much that I made up Jen Stirrup’s Law: don’t tolerate harassment.
After being bullied out of the Microsoft Data Platform tech community, I built my own table. You need to choose an environment, network and community that works for you.
The sad reality is that sometimes, as a minority member of a group – or even worse, at an intersection of different minority groups – you’re just not safe from some people. If equity is giving people what they need, safety is a key part of meeting that need.
Make yourself safe, and make others safe as well. It’s not hard. Don’t overthink. Remember Jen Stirrup’s Law: don’t tolerate harassment.
Often, conferences ask for speaker and attendee feedback to improve their offering for another future event. For me, the feedback is nothing fancy and simply this: Make people safe. This means focusing on properly enforcing a Code of Conduct and carrying complaints through; don’t ignore or bury them. Suppose there have been repeated complaints about people, such as their behaviour at other events, on social media, or towards others generally. You don’t need to list a vast range of behaviours: Wheaton’s Law should be enough, and if it is not, my Law should be the action you take.
In that case, conferences can reserve the right to exclude people. Just don’t have them there. Simples.
Building your own ED&I table and inviting others
Here is how I built my own table:
It took courage to try to become more social but I overcame my shyness by building a network at charity events, such as DataKind UK. Here, I could contribute while learning new skills PLUS connect with genuinely lovely people.
Developing myself, I also used the MBA as a network opportunity. I contacted people and they took me seriously. I didn’t need a Big Tech name behind me to do that; all it took was courage and following through.
Considering my skill sets, I volunteered with organisations where I would be in contact with leaders from different walks of life. I’m currently a civilian volunteer with the Royal Air Force Air Cadets, working with civilians and serving military members. I am often asked if I’m related to this gentleman. I am not; I doubt that he is ever asked about me!
5. Consider the ethical and social consequences of your Data Science and Artificial Intelligence applications
AI, Business Intelligence and data science have been great for business decisions. Data has helped organizations progress. However, it does have ethical and social implications that need to be considered when implementing algorithms.
As leaders, we need emotional intelligence more than ever. Leaders need to understand the range of human emotions such as fear, pride, and anxiety over the automation of jobs. Automation can make the business run at the speed of the customer, not the speed of IT or Artificial Intelligence departments. Automation facilitates the organization to become more efficient and reliable. By automating the ‘easy’, we can hope to make the most of our human skills, thereby building a human-centred AI system, for example, where we consider the impact of the usage and automation of Artificial Intelligence.
It is crucial to recognize the risks posed by AI in breaching human privacy, data privacy and the ethical concerns associated with such acts. We need to build business models and utilize AI more mindfully.
Over to you!
I’d love to hear what you think? Please let me know in the comments.
If you need help mentoring your Artificial Intelligence, Data Science and Business Intelligence departments, please get in touch.
The article presents insightful and practical New Year’s resolutions for Business Intelligence, Data Science and Artificial Intelligence teams. The author’s background in both business and technology provides a unique perspective on the challenges and opportunities in the field. The discussion on data engineering practices and the importance of continuous transformation is especially valuable for team leaders and practitioners in the industry. Overall, a well-written and informative piece.
Dear Author,
Great post The article discusses new year’s resolutions that businesses dealing with Business Intelligence (BI), Artificial Intelligence (AI), Data Engineering, and Data Science should adopt. The first resolution recommended is adopting robust data engineering practices. Secondly, the article suggests that teams consider the Ops for BI, AI, and Data Science, including MLOps, DataOps, and AIOps. Lastly, the article recommends investing in ongoing training opportunities to promote continuous transformation in the industry. Additionally, the author warns about the ethical and social consequences that AI and Data Science applications may have.Thanks for Sharing
Your blog is very helpful and informative. I read this blog. Thanks For Sharing With Us. I will definitely go ahead and take advantage of this. Cheers for sharing with us your blog.
I appreciate your kind words written in blog and for sharing your thoughts with us.
Your blog is very helpful and informative. Thanks For Sharing With Us.