In a conversation with Con Conlon, Founder of Merit, Gladys explains the challenges that arise in the process of data transformation and extraction, and how AI can play the role of an enabler of efficiency in all three stages.
TRANSCRIPT:
Con
I think we've learned that collecting data is one challenge. It has its complexities at scale for sure, but it’s really the transformation bit, that is pulling data from 400 sources, that is messy and grappling and distilling it down into one source, that's actually quite tricky, isn't it?
Gladys
Absolutely. I think there are three stages or three areas of transformation that we generally face challenges with. The first one is data extraction or entity extraction. For example, we have a client who is in the automotive intelligence business. And for this client, we gather data from a manufacturer or OEM brochures, and vehicle data. So often you find that in the description, you will find the make, model, trim, door, body type, everything all lumped into a single line. Now, extracting that is, you know, quite challenging. I think earlier we used to use rule-based tools like GATE and later like, you know, NLP based tools like CRF. With the advancements that we see in the field of deep learning, let's say with LSTM networks and the transformer models, we are able to efficiently handle such challenges. The second area where I think we find challenges is the data to client, a customer taxonomy or the data schema of the clients. So, I think these are the areas, but as I said, as AI gains traction, we are relying more and more on fine-tuned transformer-based semantic models to overcome such challenges. And it's getting easier.
Con
Yeah, yeah, I guess spotting duplicates and stuff like that. But then you're right, mapping a piece of raw data from a website or from a source to that exact equivalent record buried in the client's system, that mapping can be quite tricky. But yeah, I guess machine learning is... I mean, and just anything else on AI, what's all the rage? I think we'll try to get that balance between being AI skeptics and all the hot air that you have around AI and being slightly cynical on one side. But I think we really are seeing this shift, aren't we? Although we were using it from a long time ago, I think the shift is pretty genuine, isn't it, in terms of what we're able to do now.
Gladys
Yeah. I think we are using AI more and more as an enabler. And if you look at the three stages of transformation that that are the three areas that I mentioned, which is, data extraction, or like no data preparation and data mapping. more to get us to become much more efficient. One thing that we have also seen is that this has now reduced the time that we take to develop a model. Let's say, we have seen that it has come down by 50-60%. So the use of AI has definitely helped us in that aspect. It's also improved the accuracy of the data, you know, to a large extent. And also if you look at generative AI. We have moved from, you know, those old conventional machine learning solutions into much more, you know, complex decision-making tools which mirror a human decision, more and more. So, and the other area that AI has significantly helped us is, and as we collect data, you know, large sets of data, I think is the regulatory security and the data privacy and data governance aspects. So I think definitely AI has helped us and we are seeing that it's just getting better and better.
Информация по комментариям в разработке