Linkdin revealed in a blog post today that it has finally cracked the code to an AI which will provide text descriptions for user photos. From now, once a user updates a photo on the platform, they will be able to see a suggested alternative text recommendation. This has been achieved using LinkedIn’s very own data set in collaboration with Microsoft’s Cognitive Services platform.
Making the announcement through a blog post, LinkedIn said:
Currently, LinkedIn allows members to manually add alternative text description when uploading images via web interface, but not all members choose to take advantage of this feature. To uphold our vision, we must make rich media accessible for all of our members … [That’s why] we are exploring to help us improve content accessibility at LinkedIn.
The authors agreed that there are countless challenges in the path of automatic caption creation, the most prominent one being the subjective nature of these captions. This will require an in depth analysis and know-how of objects present, along with their attributes, as well as the understanding of time and space to accurately identify and then depict the activity.
To overcome these confrontations, the team relied on Cognitive Services Analyze API, which would generate alternative text descriptions from images ranked by Confidence score. Then they hired human evaluators to cross verify the text descriptions, categories and tags with the ones they have prepared themselves.
While Microsoft’s API had no difficulty in capturing real life objects such as newspapers or places like subways, it did have an initial hard time identifying some LinkedIn media, which contained professional contexts such as certificates, charts, posters, slides, projectors, conferences and others. The team overcame this by correcting the existing alternate descriptions of LinkedIn.
Having worked on this, the team developed a meta classifier which would eliminate the text content which might harm LinkedIn customer experience. In addition, they also implanted a description correction module which would dig out and fix, any incorrect description for the images.This created an improved caption generator which works on tags taxonomy, a related dictionary and and texts associated with LinkedIn feed post.
The blog added:
[The] addition of rich media within the LinkedIn feed raises a question: is the feed fully inclusive for all LinkedIn members? For instance, can a member who has a vision disability still enjoy rich media on the feed? Can a member in an area with limited bandwidth, which could stop an image from fully loading, still have the complete feed experience? LinkedIn’s AI teams [continue to build] image description models for rich media content specific to the LinkedIn platform to help improve overall image description accuracy.
LinkedIn is no rookie to AI. The Recommended Candidate feature already employs the tech to identify deserving candidates for a hiring, and prepares a catalog accordingly for display within the dedicated tab. Its AI employed search engine browses through all the profiles and then short lists the best fitting candidates for a particular job.
