Data- and people-driven transformation: How the Port of Auckland is navigating the future

The data and insights team at the Port of Auckland is an enterprise-level team that provides information in a systematic and consumable way to different departments in the organization so they can make data-driven decisions, see how they’re progressing toward strategic outcomes and base their decisions on facts…instead of just going by their gut feel. 

We have multiple roles within the team. There is an engineering space where people focus more on the back end, which is more akin to organizing the books in a library so that you can find the information you need when you need it systematically. Analysts and visualization experts on the team focus entirely on making the information more consumable so that when you look at a piece of data, you can immediately get the insight within seconds. Their focus is very much on visualizing things, utilizing UI/UX principles and making information more consumable for people. We also have some data leads on the team, people who take the initiative and find problems that can be solved using data and advanced analytics within the organization. Their role is to bridge the digital and non-digital worlds and work with different business units to find opportunities where data and AI can add value. 

Migrating the Port of Auckland’s data platform to the cloud 

Our legacy platform was quite unstable, and it was supporting some operationally critical pipelines and data products. The Port of Auckland is a 24/7 operation, which means information must be available to the stakeholders 24/7. Decisions are being made about the yard or the vessel visits in almost real-time, and that information has to be available to them to make those decisions. 

The legacy platform was unstable, the cost was ramping up without transparency and we had concerns about the security of the platform. The biggest challenge was that we couldn’t impact business operations, those data assets that were being supported by the previous legacy platform. We couldn’t bring those data assets down while we were undergoing a re-platform. 

What helped me the most was that I was blessed with the best team possible, people who were so passionate about what we were doing. They understood why it needed to be done, and why the lack of stability of the previous platform was an issue for the business stakeholders. They understood the challenges that came with it and the technical complexity that comes with taking an asset that’s being built and supported for one kind of platform and making it fit into a different kind of technology. Instead of it being something daunting, our team was excited about the possibility of having those data assets in a new, stable platform, and what the world would look like once we got through this migration project. Their attitude helped us succeed. My job was to make sure we all kept aligned to the vision we had for the future, for the data platform and for what data as a whole could do for the organization. From time to time, I helped brainstorm technical ideas. And sometimes I’d be there to let them vent when they were having a tough day with some bug in the platforms. But that’s all part of the job, and I quite enjoy those conversations. 

This project was crucial for the organization, but it also allowed me to invest in some younger people on our teams who wanted an opportunity to prove themselves and learn and grow outside of the roles and responsibilities they had at the time. And during this process, we saw some massive growth in people and their career trajectories. I’m most proud about that, proud to have been a coach for them during that journey, and proud to have seen that entire transition happen right in front of me. It is rewarding to see how a project like that could facilitate growth for the people in the organization. 

Why machine learning and AI projects fail 

With all the AI hype that exists today, some of the nuances get lost in conversations about how tricky it can be to have some of these projects land in production where they can add commercial value or improve processes in real-time. There’s a statistic from Gartner that says 85% of machine learning and AI projects fail. And I wanted to shine a light on that when we talk about AI and proof-of-concepts. 

Proof-of-concept is not the end goal. It’s the beginning of a journey to have these products run in production and improve your processes or generate commercial value. To be successful, they have to be built in a way that’s wrapped in processes and people that can support it long-term, at a sustainable cost. 

There are three major reasons why these projects don’t always land the way we want them to. First is that most organizations, when they start the AI journey, don’t have AI-ready data sets. When they’re transitioning from BI reporting analytics to AI, the first bridge is to have a platform and data maturity that can support this transition. When I started working at the Port, we did not have very strong data foundations. It was one of the foundational elements we identified as ripe for transformation three and a half years ago, and it’s been a long journey since then to get the data assets in a form where they can be used to build advanced analytics products on top of. We have one source of truth for critical metrics that matter to us. Without those foundations, you can create models, but models live and die with the data. You can feed them garbage and get some predictions out of it. They just wouldn’t be accurate for the world you inhabit. There’d be some predictions, but they wouldn’t be useful to you. 

Second, a lot of these projects sometimes fail to hit the mark because they’re developed in a silo instead of in cross-functional teams. The data experts need to work alongside the domain experts in the business who understand the business processes that are generating the data. It’s important to involve people who understand the nuances that are wrapped around the data because those nuances need to be built into the design of the AI systems. Only then will they model or represent your world correctly and accurately. 

Third – and this is the one that keeps me up at night as a data and AI professional – is the lack of conversations around AI ethics and governance. Organizations that already have data maturity can move very quickly, but if they don’t have a governance landscape or any responsible AI policy to guide them, it can lead to products that should not have been built in the first place, products that can damage the reputation of the organization in ways that can become irreparable. There needs to be systems and policies in place to ensure that the products they are building are going to be explainable, trustworthy and accountable to their customers, and would respect the privacy of the customers whose data are being used in the products. 

If we want to see success as we move from BI to AI, investing in our teams, our data and our platforms, and having a responsible AI policy in place would be the best place to start and not get distracted by the buzzwords in the industry, because they change every now and then. 

Leading teams in rapidly evolving fields like data science and AI 

When we created the data and insights team, we brought in talent from different places, people with different backgrounds, people who specialized in governance, people who specialized in visualization and people who cared about engineering practices and the rigor that goes into it. There were also a lot of thoughts and discussions about what our vision and strategy was going to be. And until we formed that North Star, there was a lot of confusion. 

Now that we do have a North Star, we evaluate and repurpose our strategy every three years. I’m currently in the phase of redoing the strategy for the next three years. This approach helps cut through the noise a little bit. When you know where the organization is heading and where the data is going to lead the organization, it becomes clear what our priorities are, and the current buzzwords don’t bother us as much. 

The most impactful leaders and mentors I’ve had in my life have been those who practice servant leadership, and I try to follow the same approach of being people oriented. One of my mentors used to say if you don’t care about credit, everything gets done. If you invest in people, if you take care of your people, they take care of everything else. That’s the approach I took during the re-platforming approach I discussed above because it was a long project with time and deadline pressure for everyone involved. It was important to make sure I was available to the people when they were feeling uncertain, when they’re feeling anxious and needed somebody to talk through the challenges they were experiencing – whether those challenges were personal or technical, they knew they could come to me, lean on me, and that has always been important to me.  

I also learned from my mentors that leadership is not a title, it’s a skill. I practiced those principles from the time we were setting this team up in the foundational space, and that is what led me to be in this position today. I’m still very people-oriented: I follow the same path of investing in my people. I follow the principles of servant leadership, and that has helped create a culture in the team, where they know we can collaborate, there’s autonomy to own the tasks and the projects that have been assigned to them. They understand their strengths and weaknesses because we work on them in one-on-one sessions, and from there we build a team that together forms one unicorn. I always say that none of us is a unicorn – a data or AI unicorn – but put us all together and you get a unicorn. That’s the idea and ethos we work with. 

The future of AI 

I’m excited to see more conversation around the responsible use of AI, and the importance of having stronger foundations before you head into it. That wasn’t part of the conversations you’d hear even a year ago when there was a lot of focus on generative AI and proof of concepts that came from that. Thanks to the popularity of ChatGPT, that’s all everyone was focusing on. 

However, it’s important to be aware of the risks and opportunities associated with this technology. Every organization needs to find its line in the sand, in alignment with its own unique priorities. What’s their risk appetite when engaging with this technology for automating or improving processes, or building a commercially viable product? That’s a conversation that I’ve seen pick up. I think the movement in the legislative space will also encourage more conversations around the responsible use of AI within organizations, and that’s a trend I’m excited to see. 

Creating a more inclusive industry 

Representation matters. Seeing someone who looks like you, who thinks like you, who has a similar origin story as yours, makes the STEM space suddenly seem like it’s doable for you as well. When I came to New Zealand, I had my own set of limiting thoughts about whether it could be done or not. Then I saw somebody speak on stage who was a prominent leader in the AI space at the time, and the only woman of color I could find at the time who was leading in that space. Just seeing her do it at the time made such an impact on me, it just unblocked something in my head that it could be done. It can be done. 

To attract more women in STEM careers, representation matters. My mentors have been the guiding light during my journey, to help me navigate things like imposter syndrome, to help me navigate challenges when I wanted to walk on paths that haven’t been charted before. I try to give back to the younger generation the same wisdom I’ve received from the generation before me. 

I consciously invest a lot of time speaking to people who are graduating and who are coming into the industry now, especially women, and being very, very authentic in my conversations about the challenges I experienced back then, and kind of consciously opening the doors for them that weren’t opened that easily for me. I’m very blessed that I’ve met so many women during my career at the Port who have opened those doors for me that I knew would have been a little bit harder to open otherwise without their support or sponsorship. 

The more conversations we have about equality in this space, especially in the data science space – especially when we’re talking about the responsible use of these technologies – it’s important to have those different voices at the table. I’ve fought hard to find my way to some of those tables myself. If we can just inspire the next generation to participate more in those conversations and make sure their voices are heard, we could have a world where AI is a force for good, where data is a force for good.