Social Media Screening Platform

In this project, we were asked to develop a machine learning system for collecting, analyzing, and reporting data from social media like FB, Instagram, Twitter, and others under 14 criteria.

The main challenge our client faced at the dawn of development concerned the main solution’s functionality, namely the feature of analyzing the candidate’s profile

• by a third-party program (NLP AI in this case) because of most social media privacy policy, which is only allowing to access that information for the logged-in user’s page;
• in bulk, meaning the possibility to analyze several profiles at once with the speed modern technologies can offer.

To put it more technically, our primary task was to develop a social media scrapping solution that would not just be able to access the logged-in user’s personal information but also analyze other people’s content without any obstacles.

PoC

We started by developing a PoC: while we didn’t have any blockers on the way of implementing AI and NLP analysis to such a product, we still had the challenge of scrapping and maintaining the data of different users. 

• We spent one month conducting a PoC on a small volume of posts to prove the possibility of the principle. 
• Establishing the concept meant we could create a scrapping solution that gets publicly available data (gathering posts, images and related information to analyze it further). 
• To tackle the issue of scrapping any given user’s data, we leveraged Selenium scrapping solutions.

NLP Analysis + Cloud

The solution’s principle combines the advanced machine learning module for content and image analysis and the NLP algorithms for marking posts under the risk criteria. This combination created a comprehensive system that processes a large amount of visual and written content from six social networks and provides reliable results in a matter of hours.

The product is designed according to microservices architecture best practices. It provides the client with certain benefits:

• There’s an option to easily extend data sources (social networks).
• A feature toggle allows the platform’s owner to enable or disable certain features for selected user accounts in real time.
• Autoscale Cloud Server Infrastructure (with the help of Kubernetes Cluster), which enables the queue of requests (it’s possible to schedule scrapping or set them in a queue), but most importantly it enables load balancing. It means that the client only pays for the server whenever we actually have a need to conduct the scrapping. To achieve cutting hosting costs and application scalability during peak loads, we leveraged serverless infrastructure. Depending on the amount of data consumed from a particular source, scale-up and scale-down are possible.
• The system encapsulates business logic to integrate with the data source.
• The solution implements an end-to-end DevOps process letting the business concentrate on functionality and on-time feature delivery.

Web-based Solution

We used .NET for back-end development and Angular for the front-end: these frameworks are a proper choice for large-size systems with an extensive workload and a need for quick data processing (back-end) and feature-rich user interface (front-end). We’ve built the front-end as a single-page application (SPA) while also integrating the payment system.

Compliance

We made the platform compliant with the FCRA and EEOC guidelines for social media background checks, meaning that it doesn’t access the information (posts or other content) the user has voluntarily limited, hidden, or deleted, thus only relying on what’s actually available on the UI part.